RBXSerialize : A All-In-One Roblox instance and datatype serializer

RBXSerialize v0.1

A All-In-One Roblox instance and datatype serializer

Supports the majority of ROBLOX instances and datatypes.


Version Notes

  • This is an early version! Some things may not work as expected. I’m releasing it semi-unfinished because of how long its taking me to complete the rewrite.

What is serialization?

RBXSerialize acts as a solution to, “How can i store obejct in a datastore” or “how can i store leaderstats”. Although this modules somewhat encourages bad habits, the point was making this as a easy and all-in-one solution whilst being compact and fast. To learn more about what seralization is i recommend reading this.


Getting started.

It’s extermely easy to get started using this module.

Ah-ha! Step 1, requiring the module and building the api.

local RBXSerialize = require(game.ReplicatedStorage.Modules.RBXSerialize)
RBXSerialize.BuildAPI()

If you dont build the API then the encode methods will fail.


So we got the module. Next step Encoding

local RBXSerialize = require(game.ReplicatedStorage.Modules.RBXSerialize)
RBXSerialize.BuildAPI()

-- Different Example Values 
local Model = Instance.new("Model") 
local Part = Instance.new("Part",Model) 
local CFrame = CFrame.new(Vector3.new(0,10,35)) 

local ModelEncoded = RBXSerialize.Encode(Model) 
local CFrameEncoded = RBXSerialize.Encode(CFrame)


Now our information is stored in a string format. We can use this string to Decode.

local ModelEncoded = RBXSerialize.Encode(Model) 
local CFrameEncoded = RBXSerialize.Encode(CFrame)

-- Theese values are identical to the orginal. You can store the string and recreate it later.
local NewModel = RBXSerialize.Decode(ModelEncoded) 
local NewCFrane = RBXSerialize.Decode(CFrameEncoded) 


That’s it! You should now be a pro at decoding and encoding instances using RBXSerialize. Remember that you should save the encoded string to hold on to the instance data.


Things to Note

You can also modify the DeepValues,PostValues,AlwaysSave,NeverSave tables under RBXSerialize to change how the saving works for certain classes.

You can also serialize custom references or references outside of the instance your encoding by using RBXSerialize.RegisterReferences({})


Limitations
Here some things you cannot do!

  1. Have a instance property value with more than 255 Characters.
  2. Having instance data that exceeds 255 bytes.
  3. Encode children of a unsupported instance.

Features
Some unique features of RBXSerialize.

  1. Saves Instance References, Adornee, Parent, and ObjectValue.Value properties.
  2. Saves Attributes.
  3. No limitation with instance names.

Latest Version!
Here is the latest available version of RBXSerialize.


FAQ

What does encoded data look like?

Summary

RawData Example
This is what raw data looks like for instance data.

^Gv((AIMBt@23pTMHv/EMm_D:y=ARMuWmu*D$AAAAA?DUE#^J$XbCA3FHnHOXflBbjTACAAABtUAMAAAsLAAT|DAuWbS3LH*5N[dEAbLS8XLAAAA*hCtw(z!LRqAgAK`cu|xzW4YlD$AAAAA3DUEAA5FIAAAkBAABthL~x7nJ*o"IGXL8M@BgAAAAA`BKC>{^fUMD";Cy$GH5P$Af<G"AAAAXLFADAAA<CAAd~AAAA@A5!3WED5F.4`AIAAAAAsAYAoyAvi~AtgtN9j){B3(HTvWAAAA6yE"BAAAmBAA8sAA#T@Agz3WED5Fo4`AIAAAAAsAYAuW/uLq8sV!3wc)$BgAAAAA/BKCAA:CEAC"#AAAXL

ex.png

*6 instances described in 350bytes.


Will more DataTypes be supported in the future?

Summary

Im working on supporting more DataTypes that are needed! If there are any that are not listed in the supported instances. Let me know, and i can add them!

What DataTypes are supported?

Summary

Suported DataTypes

  • String
  • BinaryString
  • Content
  • ProtectedString
  • UDim
  • UDim2
  • CFrame
  • CoordianteFrame
  • Boolean
  • Float
  • Number
  • Int
  • Int16
  • Int32
  • Int64
  • ALL Enums!
  • Faces
  • BrickColor
  • Vector3
  • Vector2
  • Color3
  • Rect
  • PhysicalProperties
  • NumberRange
  • Vector2int16
  • Vector3int16
  • ColorSequence
  • ColorSequenceKeypoint
  • NumberSequence
  • NumberSequenceKeypoint

Okay? So how big is this thing?

Summary

Storage Costs

Disclaimer : This references how RAW DATA is stored, data is compressed then put under base92.

Firstly, the default number type is float;
It will only differ if the ROBLOX website specifies something other than “number”

=========================
-RBLXSerializer-
DataType : 1b -- a header, to tell what type of data it is.
-InstanceSeralization-
Default Number<FLOAT> : 4b 
Instance Flat Cost : 1b --[1Byte InstanceName]
Property Flat Cost : 3b  -- [2 Bytes for the PropertyName][1Byte for Value indicator]
Root Flat Cost : 1b 
-NonInstanceSeralization-
Value Flat Cost : 2b -- [1byte for valueName][1Byte for Value indicator]
-Example-  -- (APPROX)
Stings : "My Favorite Folder" ( 10b)  -- Strings arent compressed by default!
Root Name : "HumanoidRootPart/BodyGyro" (30b)  -- working on a better way
[Including the entire stringName of all of its children repeated]
=========================
-DataTypes-
Bool    : 1b 
Faces   : 1b 
Int     : 2b
Enum : 2b
Float   : 4b
UDim    : 8b
Double  : 8b
Vector2 : 8b
Vector3 : 12b 
UDim2   : 16b
CFrame  : 48b 

Small analysis of how data is stored:

Let’s say we have a part with only 2 properties size and position, this is how it would be stored.

(1:DataType)+(1:InstanceName)+(3:Property)+(12:Vector3)(12:Vector3)

It would be stored in as little as 29Bytes.

So I’d say on your average per instance will be 50-100b.


TLDR; Can Serialize pretty much anything! Let’s you store instances in strings to store them in things like datastores !

19 Likes

Omg
I was hoping you made this
ty

If you expect users to… use the module, you may be able to get away with calling BuildAPI at module load time. Calling RBXSerialize.BuildAPI before the final return RBXSerialize should do the trick. Additionally, you could consider renaming the RBXSerialize module to MainModule so users can always reference the latest version with require(11476525100).

You could avoid these limitations by encoding sizes using a variable-length code such as LEB128 if you prefer size, or by using plain 32-bit integers and string.pack/string.unpack if performance matters.


8/10. This is definitely an option to consider for instance serialization. The base92 representation in particular avoids the horrors of non-UTF-8 values stored in DataStores. One of my earlier serialization attempts (for Lua data types in general) works around this encoding limitation by converting raw bytes to valid UTF-8 characters, at the cost of doubling the output size.

--MIT license ^^ [///removed!///]

It looks like you may have missed a few tiny details.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

I am not a lawyer, but there’s no way that outright removing the copyright notice could be justified when the entire code is copied verbatim. Then again, no one really cares about copyright (specifically regarding source code) on this platform. I’d recommend leaving copyright notices untouched, just to be on the safe side of legalese.

1 Like

Oops,

I will include the original copyright notices from the libraries I used. Not quite sure when or why i removed it.

Also to keep things simple i read each byte individually, typically instance data will never exceed 255bytes unless it’s like a text label filled with garbage text. But this is better solution that i will consider.

In my previous versions of my module the .BuildApi() was automatically called when you first attempted to encode something. The reason why i did not when the module was seated is due to the high amount of iterations that could sometimes cause my computer to freeze. Doesn’t seem like a big issue with the dump im using currently, just thought i’d make it a separate method so you could put it onto it’s own thread.

hasn’t this already been made many times in the past? I remember seeing a recourse a few months ago about this.

If your talking about instance serilization than yes~! This however is a module that is meant to perserve as much memory as possible which other instance serilization modules dont do, along with the ability to store reference properties and attribute values.

It’s really depends on your use-case on what you’d use, this module however is a rewrite of another serilization module that removes some of it’s limitations. Also ease is another factor on why i think it’s important to have a serializer with two simple methods. The majority of ROBLOX Scripters wont even be able to figure out how to use something like RBXDOM. (That is importing it from GitHub LOL).

Let me know if you have any feedback on what this is missing as I’d like to make this serializer a go-to option.

2 Likes

On the topic of your computer crashes, most likely because of how big those json files are. What would be much faster is initializing the table in pure code. I made a cool module which you could use for generating the code: VTC - Turn anything into code!.

Now we just need a module that replaces Player:SaveInstance() and Player:LoadInstance() and then we can get the old games to work again.

Never heard of this i think i’m too new :sweat_smile:

They simply save an instance storing it with the player’s UserID.

Additionally, it gave each player a limit, which can be found using Instance.DataCost.

Hello! This looks great! Could this be used to DataStore a table of Instances?
I’m making an inventory system and need a way to DataStore a table of all the items in the player’s inventory.
The only issue is that I use folders and instances to store the inventory data, and as you probably know, you can’t DataStore a table of Instances. But if I encoded the table, would I be able to DataStore the encoded table?

Thanks!

Would this serialization module work for strings? Tested it! It does work with strings! The really annoying thing is if it isn’t a Superclass it warns so I had to remove the function and replaced the line itself with continue so it skips through the loop.

Great resource overall!

1 Like

Didnt see this.

Your module does the same thing as mine, looks like you’ve just generated a Lua table from the same dump. Also, this was no-longer an issue something i just did not include before release.

Cool module, but this wont help anything.

So grateful that you made this serialize module !!
but I met a problem when I tried to serialize a “material”, here is my code:

local SerializeModule = require(game.ReplicatedStorage.modules.common.SerializeModule)

print(SerializeModule.Encode(Enum.Material.Plastic)) 

and the error:

 ReplicatedStorage.modules.common.SerializeModule.Conversion:70: invalid argument #3 to 'pack' (unsigned overflow)

is there anything I did wrong?

My module isn’t quite a serializer. It generates lua tables (like the one you saw in the dump) and I thought since you were decoding a json string that it might be faster if you initialized your dump with a lua table.

Ill take a look at this, looks like there was an encoding error.

1 Like

It would be nice if this was a MainModule so it automatically updated.

is not possible on client side due http_service on get api is not allowed but you can use require(Id_module_library) instead

This doesn’t work on the client, but it works on the server, but I don’t think I’d be using this on the client.

This resource came in clutch as I am going to need to save players towers for a Tower Defense game I am working on.

Please keep updating this since it can become a potential resource!