Changes to Roblox Product Development Process

I’d hold a normal member up to a standard above “replying to one line out of context with incorrect information,” let alone staff members. The response given is insulting even if it was unintended.

This is a two-way street and that includes respect. Staff members shouldn’t be given a pass to demean people just because it might be an accident. Whatever the intention, Jit deserves an apology because they deserve better than to be treated like a literal child by an engineer in a thread about respecting the community.

1 Like

An apology? Sure. But we as prominent community figures shouldn’t take such an inflammatory and aggressive tone when someone, whom we would rather like to have rapport with, has made a mistake that may be as simple as replying to the wrong person, misunderstanding, or simply writing in a hurry. Let’s be better than that.

19 Likes

I’ve sent a pm going into more detail. Quite a bit to go through I think. Also, thanks for your response.

2 Likes

the best things from roblox themselves

1 Like

Your entire reply was on the basis that @Mr_Purrsalot misunderstood the post, yet you did not even attempt to clarify what was being asked for. Perhaps you should’ve considered doing that.

Anyway, since you didn’t, @Mr_Purrsalot I believe OP wasn’t asking whether a feasible format exists but rather for a format that has been openly documented for the convenience of projects like Rojo.

Neither the XML nor the Binary formats have official documentation and many users have issues with serialising/deserialising to or from those formats (such as the ones OP mentioned).

4 Likes

How would one have issues with serialising to XML.

Also you should probably just be using a custom JSON format as it’s easier to manage in every way with no downsides.

1 Like

Did you even read my entire response? I don’t want to be too critical here, but you have a complete misunderstanding of my request here.

I am very aware of how these formats work and are used, and I am not asking how to use them, I’m not an idiot here…

I’m just requesting the idea of an officially supported rbxm/rbxl binary format spec; I don’t even care for an in-house implementation, just a spec at least!

Thank you to all staff supporting these decisions!

7 Likes

Does this mean just supporting local files but not allowing new ones to be created easily?

I have a feeling that this still means you’re trying to phase out local editing. Please don’t.

Also whats the exact reason that you are pushing cloud only development? It’s a bit suspicious.

2 Likes

…what?

We’re trying to maintain the tooling developers are using for their production work, and that includes supporting the most optimal format supported for builds at a low level. (Such as rbx-dom, used in Rojo, with its spec documented by @Dekkonot)

Also, the JSON format used with Rojo (.model.json files) is just meant for directly writing a model in a fairly human-readable format, and doesn’t even follow the same structure as a normal model/place file. (Doesn’t contain a root, just a base instance for ex)

1 Like

I’m not talking about Rojos JSON format.
What I meant is that serialising of models/games would be much easier to do in JSON with near 0 downsides.

Sure the binary format is slightly more optimal than JSON in regards of data storage. But if you discount the DEFLATE compression of the binary format then the benefits aren’t really that good. (And you can compress the JSON too).

If the tooling were made to use a JSON based format instead maintaining it would becomes miles easier than both the XML format and the binary format. And it would be quite optimal for data storage too (expecially if it’s compressed).
Sure it requires to rewrite code but at the end of the day less code changes would have to be made in the long run to maintain it.

1 Like

It’d be a problem with certain property types (using Base64 or similar for binary data, when you could just have a binary format); having a binary format makes sense for any large project or codebase, and being as efficent as possible to serialize, deserialize, or traverse. This is especially crucial for large production databases, like Roblox has.

You could have a JSON/BSON format in use (if you were actually to do this, please use BSON or similar due to it having datatypes such as BinData and Timestamp), but it’s far more optimal to have a format designed specifically for what it’s storing.

IMO, the binary format could be optimized and improved further aswell.

1 Like

The downside of the binary format that it’s not very conventional to implement nor to maintain.

I’m pretty sure that only applies to union operations and terrainvoxel data

1 Like

(post deleted by author​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​)

3 Likes

Oh thats actually very interesting, I’ve actually been thinking of making a JSON file spec for Roblox myself.

Though implementing it like the binary format isn’t really necessary and makes implementation and updating the spec harder (and is bad for backwards and forwards compabitibilty).
A lot of the data types could just be implemented with:

  • string (UTF8-string, binary data, regex, string, UUID, MD5)
  • number (Double, Int64, Int32, Float64)
  • boolean (Boolean)

And the remaining can be implemented with an array like structure.
Like Color3:

[
255,
123,
77,
]

(Of course the color can be bitpacked or converted to hex to make it use less torage space (if all properties are in the range 0-255))

Vector3:

[
45345.345345345,
-5,
0.243234234
]
2 Likes

All prop and chunk types could be seamlessly integrated with BSON, while still being an efficent and packed format.

I applaud many of the decisions made with the rbxm format when created in 2014, one being decisions on how types were denoted, and how everything was laid out.

One thing many developers don’t take into account when making a format in the first place is that not everything needs to be denoted as a written string.(Like a certain entry being an “Instance” type, or a certain property being a “Color3” type literally) To improve large scale scalability, you could denote types all by individual ‘opcode’ TypeIDs, and in the JSON version of the format, have everything as normal written identifiers for readability.

If the program understands each TypeID, there’s no issue with making the format more efficent. It could always be backwards compatible, and easy to maintain with the proper spec.

2 Likes

Im glad the Roblox team is always working on something new, this gets the whole community hyped cause we’ve been asking for this for some time now. Thank you Roblox!

1 Like

There could just be a reference table (dictionary) in the JSON where theres key value pairs where the name of the key is the name of the property and the value is the name of the data type of the property.

The problem with static TypeIDs is well … they are static. What if Roblox adds or removes them? It’s better to have them completely dynamic for backwards and forwards compatibility.

1 Like

If it’s a community made format, they can’t just remove a certain TypeID out of thin air… If Roblox removes the type itself from Roblox for whatever reason, it could be treated as individual enums, and each ID won’t ever change, and may just be deprecated on Roblox’s end.

1 Like

It’s actually quite a bit better. Even an uncompressed binary file is like night and day when it comes to deserializing and file size.

Consider: a Color3 takes up 24 bytes in binary and can be directly read as a sequence of floats into memory. If you used a human readable format, you’d end up with an indeterminate amount of bytes and you have to parse them first.

The binary format is very much fine to implement because it’s chunk based. I wish some of the property types were more consistently implemented, but I can’t really blame anyone for taking shortcuts when serializing types like Vector3int16 haha :stuck_out_tongue:.

1 Like

For common data types some optimisations could be done. For example for Color3s each compontent could be bitpacked (Or converted to a HEX string, both would take the same amount of bytes in JSON).
Other things to make the JSON format easier to store is instead of using direct property names like:

  • "ClassName": THEDATA
  • "Name": THEDATA
  • "Size": THEDATA

They could instead be HEX encoded indexes to an array table where the elements are property names. Like the indexes for instances would be:

  • "1": THEDATA
  • "5": THEDATA
  • "4F": THEDATA
    And the table where the indexes are stored would be like:
[
"ClassName",
SOME OTHER PROPERTY NAMES HERE
"Name",
SOME OTHER PROPERTY NAMES HERE
"Size"
]

This would make quite a lot of improvements in storage size but still keep the ease of implementation which JSON provides.

Let’s say we wan’t to deserialise or serialise models in Roblox Luau or a web application with javascript. A JSON implementation is by far the easiest to implement and upkeep and it still provides relatively good

A lot of people use XML instead of the binary format due to it’s ease of use, so a JSON format would definetly be in need.


Deserialisation of a JSON based format in Lua code would be as easy as doing

local typeHandlers = {
-- The type handlers
}

local function doStuff(data, parent, indexReferences, typeReferences)
	local object = Instance.new(data[1])

	for k, v in pairs(data[2]) do
		local rawK = indexReferences[k]

		if typeReferences[rawK] then
			object[rawK] = typeHandlers[typeReferences[rawK]](v)
		else
			object[rawK] = v
		end
	end

	for _, v in ipairs(data[3]) do
		doStuff(v, object, indexReferences, typeReferences)
	end
end

local tbl = game:GetService("HttpService"):JSONDecode(THEDATA)

doStuff(tbl[1], nil, tbl[2], tbl[3]).Parent = workspace

Here would be an example JSON serialised model (just an example):

[["Model",{"1":"ModelWithAPart","2":[0,0,0,0,0,0]},[["Part",{"2":[10,0.4523534,5,0,0,0],"3":[10,50,20],"4":10551051},[]]]],["Name","CFrame","Size","Color3"],{"CFrame":"CFrame","Size":"Vector3","Color":"Color3"}]

(The property indexes are HEX instead of numbers so they save some space for models with a lot of properties. Alternatively the indexes could be raw bytes (0-255 meaning all string characters) making them use even less character space)

1 Like