a note before reading the rest of this
some serializers that advertise themselves as the “best.” this is not made to actually compete with them. i made it bc i found their solutions lacking in features that i couldn’t be asked to hack into their source code. i only consider it “better” for my usecases. my implementation fails to store some hyper specific cases which i know already and will never bother to fix (meaning ill just never make it public). its also unoptimized because i wrote it at 4am in a state of delirium.
also, if you do plan on using this, this is under the MIT License. have fun, beware the const spam, and happy programming!
Milda’s Dream Game File (MDGF) is an data format partially based off of (NOT AN IMPLEMENTATION OF) the Roblox Binary Model Format based on what is detailed on rbx-dom. MDGF implements the vast majority of datatypes, ignoring ones that would seemingly be impossible, and is very lazy in its compression—which is by default set to the max and ignores speed. This is my first functional attempt (depressing) at a serialization format. Instances are written to a buffer, which can be saved using DataStores, or inside of a ModuleScript using a slightly modified version of LuaEncode[1] (originally by @ jitlua/chadhyatt)
The format described here does NOT have support for attributes or tags. Just plain old Instances and Properties.
The format also does NOT fully implement some DataTypes. Please check for yourself if a DataType is written correctly if you plan on using my garbage.
What follows is an imitation of Roblox Binary Model Format, Version 0 | rbx-dom. Modifications were applied to fit the rest of the post. I am so sorry.
Conventions
Conventions
This devforum post assumes an understanding of how the roblox buffer library types its values. An additional numerical type is included in this format, named uvarint.
uvarintis defined as having 1 bit forcontinuanceand 7 bits fornumerical data.If the
continuancebit is set, multiply the existing value by128(lshift 7); addnumerical data; and read the next byte.
If thecontinuancebit is not set, multiply the existing value by 128(lshift 7); addnumerical data; and return the value.
For example:
u32is an unsigned 32-bit integeri16is a signed 16-bit integer
All numerical values are assumed to be little endian and 2’s complement.
The data contained in a chunk may be compressed. The term “chunk data” refers to the decompressed contents.
File Structure
File Structure
MDGF files[2] consist of a short header, followed by a series of chunks. Each chunk has the same format, meaning it is very easy to parse the file.
- File Header
- Chunks
- Zero or more
CLASchunks - Zero or more
NAMEchunks - Zero or more
PROPchunks - Zero or one
PRNTchunks
- Zero or more
File Header
File Header
Every file starts with a 32 byte header.
| Field Name | Format | Value |
|---|---|---|
| Magic Number | 4 bytes | Always MGDF |
| Version | u32 |
Always 0 |
| Instance Count | u32 |
Number of instances in the file |
| Compression Algorithm | u32 |
Enum Value of the Compression Algorithm used for compressed chunks. |
| Reserved | 16 bytes | Always 0 |
Chunks
Chunks
Every chunk starts with a 16 byte header which is followed by the chunk’s data.
Why do I bother including uncompressed? For fun.
| Field Name | Format | Value |
|---|---|---|
| Chunk Name | 4 bytes | The chunk’s name, like CLAS or PROP |
| Uncompressed Length | u32 |
Length of the chunk’s data after decompression |
| Chunk Length | u32 |
Length of the chunk’s data |
| Compressed Flag | u8 |
If the chunk is compressed, equals 1, otherwise, equals 0 |
| Reserved | u24 |
Always 0 |
| Chunk Data | Variable | The data contained in the chunk |
The chunk’s data, if compressed, will always use the compression algorithm defined in the File Header.
CLAS Chunk
The CLAS chunk has this layout:
|Column 1 | Column 2 | Column 3 | Column 4|
| Field Name | Format | Value |
|---|---|---|
| Class ID | u32 |
A number ID referring to a Roblox class |
| Class Name | u32, String |
u32 for the class name length. The class name, like Part or ReflectionMetadataYieldFunctions |
| Instances | Array(<u32>Instance, <u32>Instance) |
The ranges of instances belonging to the class. |
Instances in the chunk are stored with the first Instance of the Class and the last Instance of the Class in a consecutive sequence. If it is only defining one Instance, then it will be the same Instance for the first and last Instance.
Class Name matches the ClassName specified on an instance in Roblox.
Example:
1 - Class A
2 - Class A
3 - Class A
4 - Class A
5 - Class A
6 - Class B
7 - Class A
The Instance array for Class A will be written as:
1 5 7 7
A full chunk for Class A would be:
[Class ID] 7 Class A 1 5 7 7
NAME Chunk
The NAME chunk has this layout:
| Field Name | Format | Value |
|---|---|---|
| Name | u32, String |
u32 for the name length. The name of the instances |
| Instances | Array(<u32>Instance, <u32>Instance) |
The ranges of instances that have this name. |
See CLAS’ example. The chunk types are very similar, but NAME chunk removes the Class ID
PROP Chunk
The PROP Chunk has this layout:
| Field Name | Format | Value |
|---|---|---|
| Class ID | u32 |
The class ID assigned in the CLAS Chunk |
| Property Name | u32, String |
u32 for property name length. The name of the property, like RightClavicleTposeAdjustment |
| Values | Array(<u32>Instance, <u32>Instance, Array(Variable)) |
A list of Instance, Instance, Array(Value) tuples |
The PROP chunk defines a single property for a single instance type. The PROP chunk only includes Instances that have changed properties from their defaults (defined by Instance:IsPropertyModified(property: string))
There should be one PROP chunk per property per instance type.
Instance. Instance works as defined in prior chunks. The Array(Variable) is an array of the values of Instance to Instance.
this is inefficient, yes. it would be better to use a u8 to mark changed and unchanged instead.
Class ID defines the class that this property applies to as defined din a preceding CLAS chunk.
Value
Property Name defines the serializable name of the property. This is the same as the name reflected to Luau, which is sometimes referred to as the canonical name.[3]
PRNT Chunk
The PRNT Chunk has this layout:
| Field Name | Format | Value |
|---|---|---|
| Instances | Array(Instance) |
The Instances’ parent` |
Nothing special. All Instance parents in order according to an Instance`s id.
An id of 0 refers to no parent. (or a parent that wasn’t serialized)
Data Types
Data Types
DataTypes here are from what typeof(value) returns, NOT what is actually present in memory.
DataTypes, when used by other datatypes, do not include the Type ID.
nil
Type ID 0x00
The nil type does not define anything after this.
boolean
Type ID 0x01
The boolean type stores a single byte. If the byte is 0x00, the bool is false. If it is 0x01, it is true.
number
Type ID 0x02
The number type stores a f32 (4 bytes) and is little endian. This is typically referred to as a float.
string
Type ID 0x03
The string type is stored as a length-prefixed sequence of bytes. The length is stored as a uvarlen.
| Field Name | Format | Value |
|---|---|---|
| Length | uvarlen |
The length of the string |
| String | Variable | The data of the string |
string is NOT used when storing names in CLAS, NAME, or PROP.
integer
Type ID 0x04
Reserved for 64-bit integers.
Color3
Type ID 0x05
The Color3 type stores 3 f32, with 1 for each color channel.
| Field Name | Format | Value |
|---|---|---|
| R | f32 |
Red channel |
| G | f32 |
Green channel |
| B | f32 |
Blue channel |
CFrame
Type ID 0x06
The CFrame type stores 2 vectors. The matrix is not stored because the CFrame transmitted is normalized anyways.
| Field Name | Format | Value |
|---|---|---|
| Position | vector |
The Position of the CFrame |
| Rotation | vector |
The Rotation of the CFrame, in YXZEulerAngles |
vector, Vector3
Type ID 0x07
The vector type stores 3 f32s. This is also shared with the Vector3 type.
| Field Name | Format | Value |
|---|---|---|
| x | f32 |
The x component |
| y | f32 |
The y component |
| z | f32 |
The z component |
Vector2
Type ID 0x08
The Vector2 type stores 2 f32s.
| Field Name | Format | Value |
|---|---|---|
| X | f32 |
The X component |
| Y | f32 |
The Y component |
Vector3int16
Type ID 0x09
The Vector3int16 type stores 3 i32s.
| Field Name | Format | Value |
|---|---|---|
| X | i32 |
The X component |
| Y | i32 |
The Y component |
| Z | i32 |
The Z component |
Vector2int16
Type ID 0x0A
The Vector3int16 type stores 2 i32s.
| Field Name | Format | Value |
|---|---|---|
| X | i32 |
The X component |
| Y | i32 |
The Y component |
EnumItem
Type ID 0x0B
The EnumItem type stores 1 uvarlen.
| Field Name | Format | Value |
|---|---|---|
| Value | uvarlen |
The value of the EnumItem from EnumItem.Value |
Content
Type ID 0x0C
The Content type stores a u8 type variable and an optional string.
| Field Name | Format | Value |
|---|---|---|
| Type | u8 |
The Content’s ContentSourceType |
| Value | string |
Only written if the Content’s ContentSourceType is Uri |
The ContentSourceType is written as follows:
| ID | ContentSourceType |
|---|---|
| 0x00 | None |
| 0x01 | Uri |
| 0x02 | Opaque |
| 0x03 | Object |
Object is not supported.
UDim2
Type ID 0x0D
The UDim2 type stores 2 UDims.
| Field Name | Format | Value |
|---|---|---|
| X | UDim |
The X component |
| Y | UDim |
The Y component |
UDim
Type ID 0x0E
The UDim type stores a f32 for Scale and an i32 for Offset.
| Field Name | Format | Value |
|---|---|---|
| Scale | f32 |
The Scale component |
| Offset | i32 |
The Offset component |
Rect
Type ID 0x0F
The Rect type stores 2 Vector2s for Min and Max components
| Field Name | Format | Value |
|---|---|---|
| Min | Vector2 |
The Min component |
| Max | Vector2 |
The Max component |
Ray
Type ID 0x10
The Ray type stores 2 vectors for Origin and Direction components
| Field Name | Format | Value |
|---|---|---|
| Origin | vector |
The Origin component |
| Direction | vector |
The Direction component |
NumberSequence
Type ID 0x11
The NumberSequence type stores a uvarlen and an Array(f32, f32, f32).
| Field Name | Format | Value |
|---|---|---|
| Keypoint Count | uvarlen |
The number of keypoints |
| Keypoints | Array(f32, f32, f32) |
An array of f32 triplets |
The f32 triplets are defined in the order of Time, Value, and Envelope.
No, there is no NumberSequenceKeypoint. Reminder, I wrote a majority of this code in a state of delirium.
NumberRange
Type ID 0x12
The NumberRange type stores 2 f32s for the Min and Max components.
| Field Name | Format | Value |
|---|---|---|
| Min | f32 |
The Min component |
| Max | f32 |
The Max component |
Font
Type ID 0x13
The Font type stores 1 boolean, 2 EnumItems, and 1 string for the Bold, Weight, Style, and Family respectively.
| Field Name | Format | Value |
|---|---|---|
| Bold | boolean |
The Bold option of the Font |
| Weight | EnumItem |
The Weight of the Font |
| Style | EnumItem |
The Style of the Font |
| Family | string |
The Family of the Font |
honestly i dont know if Bold even matters for Font.
Faces
Type 0x14
The Faces type stores 6 bitpacked booleans, which is 1 u8.
The order is Top, Bottom, Left, Right, Front, Back from Least Significant Bit to Most Significant Bit. (0b000001 → 0b100000)
Axes
Type 0x15
The Axes type stores 2 u8s.
| Field Name | Format | Value |
|---|---|---|
| XYZ Bitmap | u8 |
The XYZ Bitmap of the Axes |
| FacesBitmap | u8 |
The Faces of the Axes |
The first u8 is 3 bitpacked booleans. The second u8 is a Faces.
Bits are in XYZ Order from Least Significant Bit to Most Significant Bit (0b001 → 0b100).
Enum
Type 0x016
Not to be confused with EnumItem, the Enum type stores a string.
Instance
Type 0x17
The Instance type stores a uvarlen id which refers to an Instance. A uvarlen of 0 would be equivalent to nil.
BrickColor
Type 0x18
The BrickColor type stores a uvarlen id which refers to the BrickColor Id defined by BrickColor.Number
ColorSequenceKeypoint
Type 0x19
The ColorSequenceKeypoint type stores 1 f32 for Time and a Color3 for Color.
| Field Name | Format | Value |
|---|---|---|
| Time | f32 |
The Time component of the keypoint |
| Color | Color3 |
The Color of the keypoint |
ColorSequence
Type 0x1A
The ColorSequence type stores 1 u32 for the number of keypoints and an Array(ColorSequenceKeypoint) .
| Field Name | Format | Value |
|---|---|---|
| Keypoint Number | u32 |
The number of keypoints |
| Keypoints | Array(ColorSequenceKeypoint) |
The Keypoints of the ColorSequence |
PhysicalProperties
Type 0x1B
The PhysicalProperties type stores 6 f32.
| Field Name | Format | Value |
|---|---|---|
| Density | u32 |
The Density of the PhysicalProperties |
| Friction | u32 |
The Friction of the PhysicalProperties |
| Elasticity | u32 |
The Elasticity of the PhysicalProperties |
| FictionWeight | u32 |
The FictionWeight of the PhysicalProperties |
| ElasticityWeight | u32 |
The ElasticityWeight of the PhysicalProperties |
| AcousticAbsorption | u32 |
The AcousticAbsorption of the PhysicalProperties |
so how does this compare?
Poorly.
Crossroads (4,354 Instances):
MDGF - 108,177 bytes (wow i did bad)
MInstance - 31,914 bytes (im pretty sure this is a different map though, but i dont care, i wont bother actually testing it against them)
SOSDF - 61,542 bytes
The Highly Questionable SaveInstance Copy of a map Havoc.rbxm (30k ish Instances):
MDGF - 715,493 bytes (ehhh closer?)
MInstance - 628,698 bytes
SOSDF - DNE (could not find, i wont bother running it since its prolly better)
Private RBXM that I cannot share (83.5k instances)
MDGF - 1,108,935 bytes
MInstance - DNE (I CANT FIND A SURVIVING COPY???). estimating based off of Havoc.rbxm would place it roughly at 1,687,006 bytes, but thats a pretty high estimate. very reasonable chance it does better.
SOSDF - DNE (i dont have a serializer for SOSDF onhand. im not going to bother.)
SerializationService - Crashed my studio (likely because i pased in {workspace}). but the RBXL of it is roughly 1,572,864 bytes (1.5 MiB)
so it starts winning far beyond reasonable limits. its still cheating by just spamming max compression level. rbxl isnt even a fair comparision.
so why did i waste so much time on a worse solution?
for fun. i found it fun to make a format, write it, and lament over it. even if it does poorly against current solutions, i dont care.
it also is designed to output painful to deserialize things, such as meshes, as cloned instances. this is because its ment to be serialized in studio, and then deserialized only at runtime. if you couldnt tell by the name of this format, “Milda’s DREAM GAME File”, its ment to be used in yume nikki fan games which want to try to use a single-place format that doesnt require sending players with teleport requests.
it also gave me a good excuse to go through how roblox saves its own rbxms and rbxls! very enlightening in these perilous times!
and yes. i did change Format to File for the sole reason to make a joke.[2:1]
download? and a how to?
My implementation of my own format: (bad in every aspect, poorly commented. be warned.)
mdgf-serde.rbxm (20.5 KB) (MIT License included!)
THIS WILL NEVER BE UPDATED!!!
quick rundown of how it works:
serializertakes in an array of instances and an optional config, but the default configuration forced upon you is more than good enough for most cases.
serializeroutputs a buffer (serialized data), and a list of instances that are “difficult” to load. the default instances areMeshParts andLuaSourceContainers. I forgot UnionOperations existed, but you can add those too through the config.
deserializertakes in a buffer, an optional list of instances, and an optional config, but the default configuration works for the default config for serializer. the serializer config and the deserializer config are NOT THE SAME CONFIG!
deserializeroutputs a list of instances (anything that had aParentId of 0 or anything invalid). parent them to your instance of choice and you will have a deserialized MDGF.
No, this does not support parenting things automatically. its easily addable though (and can be easily made into a config parameter).
Yes, I am sorry for mocking the beautiful work done on the unofficial document for the roblox binary format.
conclusion
i wasted my time, i should had modified some existing project, my format only wins out when instance counts become unreasonably large for any game, but these “compression formats” are easily compressible if you just cluster information more efficiently (roblox engineers were… onto something…). hope it was a good read watching a sleep deprived person getting more tired. gn, gm ge, ga. im out.
and uhhh, sorry for the formalities collapsing!
may have missed some things, pls inquire if you have questions (so i didn’t “waste” time)
apparently footnotes are NOT footnotes on devforum. ok.
slightly modified luaencode that writes raw bytes in place of esc sequences when able to. luaencode.rbxm (14.2 KB) (luau allows for 0x80 → 0xFF as just string data lol???) ↩︎
Did you just say MDGF file? The “F” means file, vro! You’re saying “file file!” Would I ask you for a “coffee coffee” with room for “cream cream?”
See here if you don’t get it. ↩︎ ↩︎source for canonical name: Roblox Binary Model Format, Version 0 | rbx-dom ↩︎