A high-performance, strictly typed Base64 encoder and decoder for Roblox.
Get here on the Roblox marketplace!
Author: @aidan002010
License: MIT
Features:
Why use it?
Roblox will automatically encode buffers using zbase64 which is 1000x faster because the backend is done in c++
--!native
--!optimize 2
local HttpService = game:GetService("HttpService")
local Offset = 34
local function Encode(Text)
local Encoded = HttpService:JSONEncode(Text)
local Buffer = buffer.fromstring(Encoded)
local Length = buffer.len(Buffer)
local Base64Length = Length - Offset - 2
local Base64Buffer = buffer.create(Base64Length)
buffer.copy(Base64Buffer, 0, Buffer, Offset, Base64Length)
return buffer.tostring(Base64Buffer)
end
local TestString = buffer.fromstring(string.rep("h", 1e6))
local Start = os.clock()
for _ = 1, 1e3 do
local _ = Encode(TestString)
end
warn(`Benchmark: ${os.clock() - Start} seconds`) -- around 0.1 seconds
You can just work with the JSONEncode string directly.
local function Encode(Input: buffer): string
return string.sub(HttpService:JSONEncode(Input), 34, -3)
end
local function Decode(Input: string): buffer
return HttpService:JSONDecode("{\"m\":null,\"t\":\"buffer\",\"base64\":\"" .. Input .. "\"}")
end
Granted, a faster implementation is probably feasible with native calls to buffer.readbits/writebits, but this is definitely the simplest one.
Someone did try with read/write bits but was still slower, I don’t have the code so no idea if it could be sped up
Life would be better if roblox just added these directly
You’re right. I ran a benchmark using a 1000-byte string over 1000 iterations:
Base64.encode: 0.0973 s
JSONEncode(buffer): 0.0047 s
JSONEncode(buffer) + slice: 0.0046 s
The native method is a lot faster, since it’s handled in C++. That said, my goal with this module, was something:
It’s mainly for cases where you need reliable Base64 encoding for strings or byte arrays.
Might add an optional buffer-based path later for those who want the speed boost.
Thanks.
Doesn’t work if the string is too long, then it will not be valid base64
Well yes if the string is too long then its a dev problem, it won’t even get past the luau error of long strings with the 1gb limit. You can always process it in chunks
No, I mean
print(Encode(("1234567890"):rep(10)))
Returns
KLUv/SBkjQAAUDEyMzQ1Njc4OTABANepIAE=
And decoded…
(�/� d�P1234567890ש
It doesn’t decode/encode properly.
I wrote that in devfourm as an example, you’d be better off doing something like
local HttpService = game:GetService("HttpService")
local ToString = buffer.tostring
local FromString = buffer.fromstring
local function Compress(Text)
return HttpService:JSONEncode(FromString(Text))
end
local function Decompress(Text)
return ToString(HttpService:JSONDecode(Text))
end
return {
Compress = Compress,
Decompress = Decompress
}
No but if the string is short it will be base64 but if it is long like let’s say 50 chracters, it will be zbase64
I’m looking for it to be base64 for the entire time
But regular implementations are too long or complicated for my use case
Roblox applies zstd compression if the length of the string is more than a 100 or so if it isn’t it defaults to normal base64
So how would I turn that back into regular base64? Or is your method not the way to go for my situation
Unless you want to create a way to process a string in chunks less than a 100, theres no real way to change it back into normal base64 as I believe the internal compression is custom
Made it a little bit better
local HttpService = game:GetService("HttpService")
local Offset = 34
local function Encode(Text : string) : string
local Encoded = HttpService:JSONEncode(buffer.fromstring(Text))
local Buffer = buffer.fromstring(Encoded)
local Length = buffer.len(Buffer)
local Base64Length = Length - Offset - 2
local Base64Buffer = buffer.create(Base64Length)
buffer.copy(Base64Buffer, 0, Buffer, Offset, Base64Length)
return buffer.tostring(Base64Buffer)
end
local function Decode(Text : string) : string
return buffer.tostring(HttpService:JSONDecode(`\{"m":null,"t":"buffer","{Text:match("^KLUv/") and "zbase64" or "base64"}":"{Text}"\}`))
end
0.1 seconds per gigabyte sounds too good to be true 
On random data, that would be 6+ seconds.
--!optimize 2
--!native
local HttpService = game:GetService("HttpService")
local Offset = 34
local function Encode(Text)
local Encoded = HttpService:JSONEncode(Text)
local Buffer = buffer.fromstring(Encoded)
local Length = buffer.len(Buffer)
local Base64Length = Length - Offset - 2
local Base64Buffer = buffer.create(Base64Length)
buffer.copy(Base64Buffer, 0, Buffer, Offset, Base64Length)
return buffer.tostring(Base64Buffer)
end
local function genstr(n: number, ascii: boolean)
local b = buffer.create(n)
local min = ascii and 32 or 0
local max = ascii and 126 or 255
for i=0, n-1 do
buffer.writeu8(b, i, math.random(min, max))
end
return buffer.tostring(b)
end
task.wait()
-- binary
local TestString = buffer.fromstring(genstr(1e6, false))
local Start = os.clock()
for _ = 1, 1e3 do
local _ = Encode(TestString)
end
warn(`Benchmark: ${os.clock() - Start} seconds`) -- Benchmark: $6.74043079999683 seconds
task.wait()
-- ascii
local TestString = buffer.fromstring(genstr(1e6, true))
local Start = os.clock()
for _ = 1, 1e3 do
local _ = Encode(TestString)
end
warn(`Benchmark: ${os.clock() - Start} seconds`) -- Benchmark: $6.314211600001727 seconds
I didn’t know Luau buffers use base64, could you link me a source to this?
1000x is exaggeration, only c++ libraries would be doing it in 0.1 seconds but 6ish seconds isn’t bad when the string is also compressed by zstd which in itself isn’t great for random strings.
Although I managed to get it down to 4.4 seconds using my own base64
Maybe try the luau repository? Just found it out while messing with this: GitHub - Sythivo/RobloxStringCompressor: fastest string compressor for the roblox platform, powered by Roblox API.
If goal is use for DataStores, then it’s better to do base96 encoding. 128 isn’t possible due to some symbols appearing during that, but I did 96 a lot of times.
Sure, it’s very slow, complex, not human readable, but hey - you get a massive 8% space saving!
