Generating zstd compressed data to please JSONDecode

[8EXM]

HttpService:JSONDecode and JSONEncode has support for handling data encoded in Zstd/Base64. It’s undocumented, but there are some mentions of it on here.

You can test it out by encoding a repetitive string in the command bar:

> =game.HttpService:JSONEncode(buffer.fromstring(string.rep("HelloFromRoblox",4)))
{"m":null,"t":"buffer","zbase64":"KLUv/SA8rQAAeEhlbGxvRnJvbVJvYmxveAEAijnr"} -- output

> =buffer.tostring(game.HttpService:JSONDecode('{"m": null, "t": "buffer", "zbase64": "KLUv/SA8rQAAeEhlbGxvRnJvbVJvYmxveAEAijnr"}'))
HelloFromRobloxHelloFromRobloxHelloFromRobloxHelloFromRoblox -- output

You can then decode the Base64 data in Linux and see it’s nothing strange happening—just data compressed by zstd, then encoded as base64.

$ echo "KLUv/SA8rQAAeEhlbGxvRnJvbVJvYmxveAEAijnr" | base64 -d | zstd -d | hexdump -C
00000000  48 65 6c 6c 6f 46 72 6f  6d 52 6f 62 6c 6f 78 48  |HelloFromRobloxH|
00000010  65 6c 6c 6f 46 72 6f 6d  52 6f 62 6c 6f 78 48 65  |elloFromRobloxHe|
00000020  6c 6c 6f 46 72 6f 6d 52  6f 62 6c 6f 78 48 65 6c  |lloFromRobloxHel|
00000030  6c 6f 46 72 6f 6d 52 6f  62 6c 6f 78              |loFromRoblox|

However, I can’t get this to work the other way around, and it returns a cryptic error:

-- linux shell
$ echo "HelloFromLinuxHelloFromLinuxHelloFromLinuxHelloFromLinux" | zstd | base64 -w 0
KLUv/QRYrQAAeEhlbGxvRnJvbUxpbnV4CgEAA5pKfyttGg==

-- roblox shell
> =buffer.tostring(game.HttpService:JSONDecode('{"m": null, "t": "buffer", "zbase64": "KLUv/QRYrQAAeEhlbGxvRnJvbUxpbnV4CgEAA5pKfyttGg=="}'))  -  Studio
  16:30:58.579  ZSTD_getFrameContentSize returned error Error (generic)  -  Edit
  16:30:58.579  Stack Begin  -  Studio
  16:30:58.580  Script 'print(buffer.tostring(game.HttpService:JSONDecode('{"m": null, "t": "buffer", "zbase64": "KLUv/QRYrQAAeEhlbGxvRnJvbUxpbnV4CgEAA5pKfyttGg=="}')))', Line 1  -  Studio
  16:30:58.580  Stack End  -  Studio

Has anyone managed to figure out the command-line incantations for making data to please JSONDecode?

The reason I want to do this is for embedding a large binary file into my project (~2.7mb, compresses to ~953k with zstd -19). I have other ideas of how to work around it if this doesn’t work but this looks like a good place to start.

[8EXM]

cc @technology1111 sorry to annoy but you made the only post i could find mentioning doing this, did you ever run into this?

[technology1111 (rust enjoyer)]

i did run into this, and i solved it by “pledging” the stream size; i only used rust, so i’m not entirely sure how this would translate to your use case.

Try:

• using a file instead of stdin, as this may allow the cli tool to read the file size, and set the pledged content size
• using an encoding library which supports pledging the stream’s content size, such as zstd — Rust compression library // Lib.rs (example shown below) or @oneidentity/zstd-js - npm

[8EXM]

using a file instead of stdin, as this may allow the cli tool to read the file size, and set the pledged content size

This did the trick. Thanks a bunch!