The following is a re-write of a Serialization Spec for CSG that is currently in development, it is not final nor is it yet completely formalized.
It should be helpful for custom importers if anyone still works on them.
It also is subject to change. So do not expect it to work forever.
Research on UnionOperations/IntersectOperations and how they are serialized
brought to you by Superduperdev2
Introduction
This document is based on information gained from:
UnionOperationbehavior in Roblox Studio (saving asrbxm/rbxmx)- Analyzing uploaded assets containing
UnionOperations (gained from a game I own, Insert Wars Remastered) - Observing output from the “Superduperdev2 Insert Webservice”/“Insert Cloud” API (the JSON)
- Analyzing (in a hex editor) specific
rbxmfiles (mainly to confirm/disprove theories)
Research was done over multiple weeks, and this document is a summary of findings.
It is not 100% complete nor final as Roblox could change it at any time,
but should give a good idea of how UnionOperations/IntersectOperations
are currently serialized.
Some context:
UnionOperations are a form of Solid Modeling, in our case, used by Roblox and its AssetDelivery system;
They are what is refered to as Constructive Solid Geometry (CSG), fancy words saying “is made of multiple different parts”.
Each one has MeshData, PhysicsData, and ChildData, all of which are used by the engine to render, simulate, and “Seperate” the object.
For quite a while it was thought they stored a custom format containing all parts,
including an operation tree, physics info, and mesh data, but new research (i.e. this documentation)
suggests its a lot simpler than that.
What was discovered:
UnionOperations are stored in an interesting, but insanely easy to replicate way.
Each UnionOperation if its uploaded, whether as part of a place or model has a hidden AssetId
property that points to a PartOperationAsset, with 2 sets of data:
MeshData(A BinaryString containing XOR encryptedCSGMDLdata {irrelavent to reconstruction})ChildData(URIKA! An RBXM blob containing all of theBaseParts used to make the mesh)
Note: ThePartOperationAssetmay also contain additional properties, but they are not relevant to this spec file
the ChildData property of the PartOperationAsset instance is a RBXM blob, and is parsed
identically to a normal RBXM file.
Therefore, you can reconstruct UnionOperations with relative ease.
IF you fetch the asset which is not stored as the AssetType Model but instead as SolidModel
Do note there are additional properties for the UnionOperation/IntersectOperation
called MeshData2 and ChildData2 that can contain the following:
- weirdly
CSGPHSdata (PhysicsData?, Possibly caused byPROPchunk corruption or a bad parse) - Their respective RBXM/mesh blobs
- An RBXM blob identical to
ChildData - Empty/Null data
When these are present, the other ChildData and MeshData properties of
the UnionOperation will be empty, but ChildData2 is parsed identically to ChildData
unless it contains CSGPHS data. As for the UnionOperation/IntersectOperation itself
it appears to encode a secondary set of data called PhysicsData/PhysicalConfigData
that starts with the bytes CSGPHS and is likely the collision data for the mesh.
Also, from what was observed the PartOperationAsset stores the unscaled mesh.
As such you have to apply the rest of the (UnionOperation/IntersectOperation) properties to
make it work. When not uploaded it appears to store these properties directly.
This is also recursive, so some of these also encode additional UnionOperations/IntersectOperations
that need to be handled in the same way. This is from limited testing and may not be accurate
for all versions of Roblox’s CSG, but its a starting point. There are a few ways this can be implemented,
but the way this spec would suggest doing so is a little complex but not too difficult.
When you encounter a UnionOperation/IntersectOperation in an RBXM parser,
look for the above properties. For AssetId, fetch the PartOperationAsset first, then
parse the ChildData/ChildData2 property (whichever is present) as an RBXM file.
In the case the ChildData/ChildData2 is present directly, just parse it as an RBXM file.
As already mentioned, this is recursive, so you may have to do this multiple times.
In both cases, this will yield the BaseParts used to create the mesh.
You can then use GeometryService calls or BasePart CSG API calls to recreate
the UnionOperation/IntersectOperation.
IMPORTANT:
With this method the pivot will not always be in the correct spot. As such, it will have to be adjusted
manually. This is because the ChildData only contains the raw BaseParts (with their own relative transforms), and not the UnionOperation transform data. You can calculate the correct pivot
by averaging the positions of all the BaseParts used to create it, or make a temporary model and use
the center of its bounding box as shown below.
function centerUnionPivot(union,parent)
local tempModel = Instance.new("Model");
union.Parent = tempModel;
tempModel.PrimaryPart = union;
local boxCFrame,_=tempModel:GetBoundingBox();
local centeredPart=Instance.new("Part",parent);
centeredPart.Size=Vector3.new(0.001,0.001,0.001);
centeredPart.CFrame=boxCFrame;
centeredPart.Anchored=union.Anchored;
centeredPart.Transparency=1;
centeredPart.CanCollide=false;
if union:IsA("UnionOperation") then
local new=centeredPart:UnionAsync({union},options.CollisionFidelity,options.RenderFidelity);
union:SubstituteGeometry(new);
new:Destroy();
elseif union:IsA("PartOperation") then
local new=mod.Services.GeometryService:UnionAsync(centeredPart,{union},options)[1];
union:SubstituteGeometry(new);
new:Destroy();
end;
union.Parent=parent;
end;
This is not perfect, but it will get you close enough for most use cases.
You may need to tweak it further depending on your needs.
An Interesting edge case: AssetData
In some cases, the UnionOperation may not have an AssetId property,
but instead have an AssetData property, which is a BinaryString.
Decoding this string will yield a binary blob that is identical to blob of
the PartOperationAsset mentioned earlier. This is likely used for
smaller UnionOperations that don’t need to be uploaded separately.
as they can be stored directly within the UnionOperation itself.
You can parse this blob in the same way as the PartOperationAsset.
This is not common, nor often, but still something to be aware of
when reconstructing UnionOperations/IntersectOperations.
Reconstruction Guide
Parsing the serialized data
In order to reconstruct UnionOperations/IntersectOperations, you must first make sure they don’t already have the ChildData/ChildData2 stored directly. If they do, parse that and continue from there.
If it doesn’t, Look for the AssetId property, Once you have found an AssetId property (usually they will have at least one of these),
Fetch it from the asset storage and parse it. You will end up with a PartOperationAsset that has both of the required properties.
The PartOperationAsset’s ChildData is an RBXM blob and contains the parts used to construct
the unscaled mesh, this is what you are after.
Parse the blob and you will have 1 of 4 things happen:
- It will just be directly all the constitutent
BaseParts, and can be reconnected viaGeometryService:UnionAsync()
orBasePart:UnionAsync() - It will contain additonal
UnionOperations that you must recurse and parse. - It will contain
NegateOperations that have to be converted (will explain this shortly)
into
BaseParts/UnionOperations and removed from the rootUnionOperationvia
GeometryService:SubtractAsync()
orBasePart:SubtractAsync() - It will contain
UnionOperations/BaseParts marked with therbxNegatedtag
that have to be removed from the rootUnionOperationviaGeometryService:SubtractAsync()
orBasePart:SubtractAsync()
Converting NegateOperations into their subtractables
NegateOperations do not (appear to) store MeshData/MeshData2, only ChildData/ChildData2,
therefore you must parse those properties (whichever is present) to recieve the BasePart
or UnionOperation (this specific case will require recursion) that created the NegateOperation
and then subtract that from the geometry.
Walking the tree
Once you have reached the bottom of the tree, start to climb back up it using various operations.
Most of the time, this includes a bunch of GeometryService:UnionAsync() or BasePart:UnionAsync() calls, and the occasional
GeometryService:SubtractAsync() or BasePart:SubtractAsync() call.
However, IntersectOperations are special,
While they (IntersectOperations) are stored identically to UnionOperations
they are created with GeometryService:IntersectAsync() or BasePart:IntersectAsync() as opposed to the other calls.
Also it seems that Roblox treats IntersectOperations differently internally,
as they have different behavior when it comes to physics and rendering.
ie. they don’t render the same way as UnionOperations, and have different collision behavior
Roblox Studio also creates IntersectOperations in a strange way where it intersects all selected BaseParts first, then intersects the result with the first selected BasePart.
(lines up with observations). As such, simply looking at the classname and checking
if its an IntersectOperation or not, you can reliably determine if you need to adjust
the pipeline to accomodate these differences. Although it will take some trial and error,
you now can reliably create UnionOperations from their RBXM/RBXMX data.
note: the below code is a snippet from my implementation of this, it will be made available as soon as I possibly can.
function mod:applyChildData(childData,isIntersection)
local suc,res=pcall(function()
local response=self.Services.HttpService:RequestAsync({
Url=self.parseUrl,
Method="POST",
Headers={
["Accept"]="application/json",
},
Body=childData,
});
if response.Success then
return self.modules.json.decode(response.Body);
else
return error("Could not fetch. Request Error: "..response.StatusMessage.." ("..tostring(response.StatusCode)..")\nWhat went wrong:\n"..response.Body);
end;
end);
if not suc then
warn("Failed to get data: "..res);
else
print_if_debug(res);
local model=self.modules.modelAssembler:buildAsset({modelData=res},self.Services.ReplicatedStorage);
for _,inst in pairs(model:GetDescendants()) do
if inst:IsA("BaseScript") then
inst:Destroy();
warn("Union ChildData attempted to include a script: ", inst:GetFullName());
end;
end;
local function reconstruct(model)
local partToAttachTo=nil;
local theparts=model:GetChildren();
local negativeParts={};
local parts={};
if #theparts<1 then
warn("Model has no children to union.");
return nil;
end;
partToAttachTo=theparts[1];
if partToAttachTo:GetAttribute("IsNegateOperation") then --fix orientation issues
print_if_debug("found");
table.insert(negativeParts,partToAttachTo);
local old=partToAttachTo;
partToAttachTo=Instance.new("Part");
partToAttachTo.Size=Vector3.new(0.001,0.001,0.001);
partToAttachTo.Position=old.Position
partToAttachTo.Anchored=old.Anchored;
partToAttachTo.Transparency=1;
partToAttachTo.CanCollide=false;
partToAttachTo.Name="UnionBasePart";
partToAttachTo.Parent=model;
end;
for i,v in pairs(model:GetChildren()) do
if v:IsA("BasePart") then
if v~=partToAttachTo then
if v:GetAttribute("IsNegateOperation") then
print_if_debug("found");
table.insert(negativeParts,v);
else
table.insert(parts,v);
end;
end;
end;
end;
local old=partToAttachTo;
local suc,Union=pcall(function()
if isIntersection then
partToAttachTo=partToAttachTo:IntersectAsync(parts,Enum.CollisionFidelity.Default,Enum.RenderFidelity.Precise);
partToAttachTo.Parent=self.Services.ReplicatedStorage;
partToAttachTo=partToAttachTo:IntersectAsync({old},Enum.CollisionFidelity.Default,Enum.RenderFidelity.Precise); -- this is odd but fixes the problems
partToAttachTo.Parent=self.Services.ReplicatedStorage;
partToAttachTo:SetAttribute("IsNegateOperation", old:GetAttribute("IsNegateOperation"));
centerUnionPivot(partToAttachTo,partToAttachTo.Parent);
old:Destroy();
return partToAttachTo;
end;
print_if_debug(parts);
if parts~={} then
partToAttachTo=partToAttachTo:UnionAsync(parts,Enum.CollisionFidelity.Default,Enum.RenderFidelity.Precise);
partToAttachTo.Parent=model;
partToAttachTo:SetAttribute("IsNegateOperation", old:GetAttribute("IsNegateOperation"));
old:Destroy();
old=partToAttachTo;
end;
print_if_debug(negativeParts);
if #negativeParts~=0 then
partToAttachTo=partToAttachTo:SubtractAsync(negativeParts,Enum.CollisionFidelity.Default,Enum.RenderFidelity.Precise);
partToAttachTo.Parent=self.Services.ReplicatedStorage;
partToAttachTo:SetAttribute("IsNegateOperation", old:GetAttribute("IsNegateOperation"));
old:Destroy();
end;
centerUnionPivot(partToAttachTo,partToAttachTo.Parent);
return partToAttachTo;
end);
if not suc then
warn("Union operation failed: "..tostring(Union));
return nil;
end;
for i,v in pairs(parts) do
if v~=partToAttachTo then
v:Destroy();
end;
end;
for i,v in pairs(negativeParts) do
if v~=partToAttachTo then
v:Destroy();
end;
end;
partToAttachTo.Parent=self.Services.ReplicatedStorage;
model:Destroy();
return partToAttachTo;
end;
return reconstruct(model);
end;
end;
And thats pretty much it!
You can now reconstruct UnionOperations/IntersectOperations from their serialized properties!
Happy coding!
Final notes
All findings were gained from studying .rbxm files in multiple ways, at no point was any software disassembled or dumped. However there is a few other notes worthy to mention:
- This spec file is purely observatory, it is not affiliated with or endorsed by Roblox Corporation.
- All findings can be traced to different parts of the document.
- Roblox could change this at any time, when that happens this spec file will be updated with new findings.
Credits for this document are provided below:
- Superduperdev2 (@Superduperbloxer2) {Research, Spec file, and Code}
- SIWeb Network (specifically the Insert Webservice, where uploaded assets were analyzed from)
- Multiple other helpers {Fallen (@josejr0322) [code polish], god (@servertechnology/@thebigreeman) [figured out ze parser issues], vxnquish (@TNA_Cup) [Collaborator] }
Some example use cases:
- A 3D viewer that has the ability to virtually seperate
UnionOperations - A Custom importer that uses a slightly different pipeline than the typical loader
- (what ive used it for) Custom Webservice that has
UnionOperationsupport - And likely even more than I can think of.
