Just a few minutes before this post was made, I’ve written a very simple wrapper for this and was surprised to see a notification about this right after I have finished it
local HttpService = game:GetService("HttpService")
local Base64 = require(script.Parent.Base64)
local Zstd = {}
Zstd.Compress = function(String)
local Compressed = buffer.fromstring(String)
local Encoded = HttpService:JSONEncode(Compressed)
local zB64 = Encoded:match('"zbase64":"(.-)"')
local B64 = Encoded:match('"base64":"(.-)"')
if B64 then
return error("String passed is too short to compress.")
end
if zB64 then
return buffer.tostring(Base64.decode(buffer.fromstring(zB64)))
end
return error("Failed to utilize Roblox's internal zstd compression algorithm by JSONEncoding a buffer. If you get this error, this feature may not be removed.")
end
Zstd.Decompress = function(String)
local Encoded = Base64.encode(buffer.fromstring(String))
local ReconstructedJSON = [[{"m": null, "t": "buffer", "zbase64": "]] .. buffer.tostring(Encoded) .. [["}]]
local OriginalBuffer = HttpService:JSONDecode(ReconstructedJSON)
return buffer.tostring(OriginalBuffer)
end
return Zstd
What are the use cases with networking for example sending a string back and forth from the client and server?
I know there are libraries that do their best to try to reduce the packet size with types like strings and also (I assume this uses buffers) uses a buffer system to accomplish this.
FYI - Roblox implicitly compresses buffers using ZSTD behind the scenes when you send them over the wire (remotes, http service, etc.), or at rest (data stores, memory stores). No Luau implementation is going to beat the natively compiled ZSTD operation.
So you can do:
local myStr = "Pretend this is a much longer string..."
local buf = buffer.fromstring(myStr)
-- Examples:
-- "buf" is compressed over the network automatically:
remote:FireAllClients(buf)
-- "buf" is decompressed automatically:
remote.OnServerEvent:Connect(function(buf)
end)
-- "buf" will be compressed in the datastore:
dataStore:SetAsync("key", buf)
-- "buf" will be automatically decompressed:
buf = dataStore:GetAsync("key")
Note: Roblox might compress it, but not always. It uses whatever is best. For small data, often the uncompressed data is smaller.
I use this in very specific and rare cases, typically when I need to send extremely large serialized strings, such as massive Vector3 of data from the server to the client in Roblox. This kind of data transfer can be bandwidth-heavy, and every byte matters. Also the main goal of using this just to replace other old text compression people still use nowadays.
But yea, there are no use cases I found for client to server
While I was aware that Roblox automatically compresses buffer data when saving to DataStores, but I honestly didn’t know that the same applies to network. Thank you for pointing that out, I really appreciate it
I’m not sure what you’re talking about
My version acts mostly like real ZSTD compression by using Roblox APIs that return a base64-encoded ZSTD result, then decode it. It works just like ZSTD, but sadly, you can’t change any settings like compression levels like the real one
I think you’re misunderstanding because most of the code is just base64 encode/decode
Trust me, if you’re thinking of writing ZSTD in pure Lua, it’s not worth the effort
Worth the results in compression ratio but not worth the time to make since you need to port FSE, Huff0,… and it could be very very slow because of luau doesn’t have c built in functions, that need to be reimplemented in luau, and insane amount of cpu usage as same as memory. Even worse, a possible crash.
Luau implementation could be possible but really difficult to do, that would be pretty impressive to be ported by someonez
I can’t see any sane mind creating a non !native port for the sake of adding client support. --!native scripts run on the bare metal. And yes, porting it would be incredibly difficult but i say there are rewards to be reaped for that for sure
--!native doesn’t give a magically huge speed boost. It’s like taking something already slow and making it only slightly less slow. Even with buffer-based code, the main performance downgrade usually comes from algorithm complexity, frequent function calls, memory access overhead, and heavy branching inside hot loops, not from the lack of --!native
