I noticed that I haven’t posted my library here yet and now that it’s even faster - thanks to the buffer library - it’s time to publish it on devforums.
Really fast MessagePack encoder/decoder module
MessagePack is a JSON-like data format. The main difference is that MessagePack is a binary data format which means it takes signifficantly less storage space. It also happens to be much faster to parse than JSON, since data type information is embedded in the file format itself.
For more information on MessagePack look here: https://msgpack.org/
This module sets out(and succeeds!) to encode and decode MessagePack in pure Luau faster than it’s possible to decode JSON using Roblox provided JSONDecode and JSONEnocode methods.
Benchmark comparing decode+encode performance of msgpack-luau module and HttpService JSONDecode+JSONEncode of equivalent dataset:
For more benchmarks and detailed result analysis see the source repository.
API
msgpack.encode(data: any): string
Encodes any pure Luau datatype in MessagePack binary string format. It does not currently handle any Roblox specific datatypes.
msgpack.decode(message: string): any
Decodes MessagePack binary string as pure Luau value.
msgpack.utf8Encode(message: string): string
Wraps binary string in a UTF-8 compatible encoding. Nescessary to save binary strings (like MessagePack serialized data) in DataStore.
msgpack.utf8Decode(blob: string): string
Unwraps binary string from UTF-8 compatible encoding.
msgpack.Extension.new(extensionType: number, blob: buffer): msgpack.ExtensionCreate MessagePack extension type, which is used for custom datatype serialization purposes. First argument extensionType must be an integer.
msgpack.Int64.new(mostSignificantPart: number, leastSignificantPart: number): msgpack.Int64
Represents 64-bit signed integer, which is too large to to represent as Luau integer. Both arguments must be integers.
msgpack.UInt64.new(mostSignificantPart: number, leastSignificantPart: number): msgpack.UInt64
Represents 64-bit unsigned integer, which is too large to to represent as Luau integer. Both arguments must be integers.
It does encode tables. Array like tables will be encoded with smallest valid msgpack array type and dictionaries/mixed tables will be encoded with smallest valid msgpack object type.
If you look at the benchmark setup, you can see that the benchmark is ran against a large, nested JSON dataset(214kB string produced by rojo) and this module can decode and encode it faster than the native JSON methods.
Idea is that you can throw any pure Luau datatype into msgpack.encode and it will handle it without requiring developer to specify datatypes or anything.
Hence the encode method’s description says:
Encodes any pure Luau datatype in MessagePack binary string format. It does not currently handle any Roblox specific datatypes.
Roblox instances and other userdatas can’t be generically serialised and it’s out of scope of this library, so developer has to conver them to/from tables themselves.
More advanced developers can also make use of the msgpack extension data types which allow encoding arbitrary binary data tagged with identifier.
Ah, alright then. I know about the part that instances obviously can’t be serialised because it’s simply too much work and stupid. Thanks for letting me know this can serialise tables though!
First and foremost it’s a Luau implementation for MessagePack format, which is recognized and used by third-party services.
You can use this module for communicating with HTTP APIs that accept MessagePack, store more compact data in DataStore (not sure if DataStores still don’t allow binary data, might need to use the utf8encode/decode for that), and compress in-game network traffic when communicating using remotes(similar to what BridgeNet does).
I don’t have any on hand, but given that I compare this to existing JSON encoding/decoding methods that roblox provides you can look at any MessagePack and JSON comparisions.
From what I noticed online, MessagePack does produce much more compact output than JSON, but when gzipped the difference isn’t that large. Though since in roblox you only get benefits of GZip on communicating through HttpService, then I’d say it’s more practical on roblox.
All I did was simply test it after having read your comment, I would have figured that many others (including the makers of this awesome module) would have done the same. I first tested in the Studio command bar which worked, and then in a live game which also worked.
If you :SetAsync with a buffer/table/string/boolean/number, and then :GetAsync that same key, it will return the data-type which you had stored.
The simple code I used to test this was:
-- Services
local DataStoreService = game:GetService("DataStoreService")
-- Testing
local TestDatastore = DataStoreService:GetDataStore("Test")
print(TestDatastore:GetAsync("test")) -- First session: nil, Second session: buffer
TestDatastore:SetAsync("test", buffer.create(50))
In a live game, I joined a fresh server, it printed nil. Joined a fresh server after giving it enough time to sync, and it returned back a buffer. Note: The reason I did it like this was to make sure Roblox wasn’t using an internal cache or something that would affect the results.
Also another note: Roblox’s documentation doesn’t explicitly state what value it expects for :SetAsync and :UpdateAsync, it just says “Variant”. So the specifics kind of do have to be tested, but I assume given that they don’t explicitly state what the value has to be that any literal/luau-datatype is expected to be supported.
