Fast & Typed Base64 Encoder/Decoder

Resources Community Resources
1

A high-performance, strictly typed Base64 encoder and decoder for Roblox.

Get here on the Roblox marketplace!

Author: @aidan002010
License: MIT

Features:

  • Encode & decode strings
  • Encode & decode byte arrays
  • Utilities for converting between strings ↔ byte arrays

Why use it?

  • Efficiently store binary data in DataStore
  • Safely send strings over RemoteEvents
3 Likes
2

Roblox will automatically encode buffers using zbase64 which is 1000x faster because the backend is done in c++

--!native
--!optimize 2

local HttpService = game:GetService("HttpService")
local Offset = 34 

local function Encode(Text)
	local Encoded = HttpService:JSONEncode(Text)
	local Buffer = buffer.fromstring(Encoded)
	local Length = buffer.len(Buffer)

	local Base64Length = Length - Offset - 2 
	local Base64Buffer = buffer.create(Base64Length)
	buffer.copy(Base64Buffer, 0, Buffer, Offset, Base64Length)
	
	return buffer.tostring(Base64Buffer)
end

local TestString = buffer.fromstring(string.rep("h", 1e6))
local Start = os.clock()
for _ = 1, 1e3 do
	local _ = Encode(TestString)
end
warn(`Benchmark: ${os.clock() - Start} seconds`) -- around 0.1 seconds
3

You can just work with the JSONEncode string directly.

local function Encode(Input: buffer): string
	return string.sub(HttpService:JSONEncode(Input), 34, -3)
end

local function Decode(Input: string): buffer
	return HttpService:JSONDecode("{\"m\":null,\"t\":\"buffer\",\"base64\":\"" .. Input .. "\"}")
end

Granted, a faster implementation is probably feasible with native calls to buffer.readbits/writebits, but this is definitely the simplest one.

4

Someone did try with read/write bits but was still slower, I don’t have the code so no idea if it could be sped up

Life would be better if roblox just added these directly

5

You’re right. I ran a benchmark using a 1000-byte string over 1000 iterations:

Base64.encode:                0.0973 s  
JSONEncode(buffer):           0.0047 s  
JSONEncode(buffer) + slice:   0.0046 s

The native method is a lot faster, since it’s handled in C++. That said, my goal with this module, was something:

  • that doesn’t rely on the buffer API,
  • and is fully transparent, and easy to modify.

It’s mainly for cases where you need reliable Base64 encoding for strings or byte arrays.
Might add an optional buffer-based path later for those who want the speed boost.
Thanks.

6

Doesn’t work if the string is too long, then it will not be valid base64

7

Well yes if the string is too long then its a dev problem, it won’t even get past the luau error of long strings with the 1gb limit. You can always process it in chunks

8

No, I mean

print(Encode(("1234567890"):rep(10)))

Returns

KLUv/SBkjQAAUDEyMzQ1Njc4OTABANepIAE=

And decoded…

(�/� d�P1234567890ש 

It doesn’t decode/encode properly.

9

I wrote that in devfourm as an example, you’d be better off doing something like

local HttpService = game:GetService("HttpService")

local ToString = buffer.tostring
local FromString = buffer.fromstring

local function Compress(Text)
	return HttpService:JSONEncode(FromString(Text))
end

local function Decompress(Text)
	return ToString(HttpService:JSONDecode(Text))
end

return {
	Compress = Compress,
	Decompress = Decompress
}
10

No but if the string is short it will be base64 but if it is long like let’s say 50 chracters, it will be zbase64

I’m looking for it to be base64 for the entire time

But regular implementations are too long or complicated for my use case

11

Roblox applies zstd compression if the length of the string is more than a 100 or so if it isn’t it defaults to normal base64

12

So how would I turn that back into regular base64? Or is your method not the way to go for my situation

13

Unless you want to create a way to process a string in chunks less than a 100, theres no real way to change it back into normal base64 as I believe the internal compression is custom

1 Like
14

Made it a little bit better

local HttpService = game:GetService("HttpService")
local Offset = 34

local function Encode(Text : string) : string
	local Encoded = HttpService:JSONEncode(buffer.fromstring(Text))
	local Buffer = buffer.fromstring(Encoded)
	local Length = buffer.len(Buffer)

	local Base64Length = Length - Offset - 2 
	local Base64Buffer = buffer.create(Base64Length)
	buffer.copy(Base64Buffer, 0, Buffer, Offset, Base64Length)
	
	return buffer.tostring(Base64Buffer)
end

local function Decode(Text : string) : string
	return buffer.tostring(HttpService:JSONDecode(`\{"m":null,"t":"buffer","{Text:match("^KLUv/") and "zbase64" or "base64"}":"{Text}"\}`))
end