Let me explain the situation first.
So, essentially, I’m trying to bake a lot of data for a game I’m working on with a friend.
To save the baked data, I need to write it somewhere.
I decided to go for setting the source code of a ModuleScript directly into a returnable table which contains the data in proper code format.
Here’s just a screenshot of some of the data in one module script. As you can tell by the scroll bars, it is pretty big.
And there are roughly 50 of these modules filled with data, and most of the map is yet to be baked.
The baking process itself is quite slow, particularly the saving part which actually puts all the data into the ModuleScripts. This is most likely because of the sheer amount text being added again & again to the source string.
As is visible in the image, most of the characters are taken up by floating point numbers that are converted into string.
I figured, that if I turned the floating point numbers into a string with the same bits, then that should save a lot of space.
So, instead of directly writing the floating point numbers to the source, I basically did the following:
local buf = buffer.create(32)
...
loop do
buffer.writef32(buf, 0, 4.399993896484375)
sourceString ..= buffer.tostring(buf)
end
Now, the output of that does give a nice, shortened string: �̌@
However, seems that roblox doesn’t like strings with such special characters?
In fact, as it seems, the string just stops anything after it… Like, when I tried to copy the string from the console, it looked like this in the console:
But after copy-pasting it here, it only came out as START�̌@
See how the “END” is missing? I absolutely selected that part too when copying, but it just doesn’t come.
And this seems to be happening when saving the data too. Writing to the source string and then saving that as the actual source of a ModuleScript only writes this (code for saving below)
return {
{-115.0919189453125,5.326621055603027,'Outside',0,{{2,'�̌@
And this is the code I used to save the data (this is the shortened/simplified version, entire code below):
@native
function mainModule.SaveBakeData(waypoints: {WaypointData})
print("Saving baked data...")
local curSource = [[
return {
]]
local buf: buffer = buffer.create(32)
for i: number, waypoint: WaypointData in waypoints do
curSource ..= "{" .. tostring(waypoint.Position.X) .. "," .. tostring(waypoint.Position.Y) .. ",'" .. waypoint.AreaName .. "'," .. (waypoint.IsWallWaypoint and "1" or "0") .. ",{"
for _, connection: {boolean | number} in waypoint.Connections do
buffer.writef32(buf, 0, connection[2])
curSource ..= "{" .. tostring(connection[1]) .. ",'" .. buffer.tostring(buf) .. "'," .. (connection[3] and "1" or "0") .. "," .. (connection[4] and "1" or "0") .. "},"
end
curSource ..= [[}},
]]
end
curSource ..= [[
}]]
curDataModule.Source = curSource
end
Entire code:
@native
function mainModule.SaveBakeData(waypoints: {WaypointData})
print("Saving baked data...")
local buf: buffer = buffer.create(32)
local curDataModule: ModuleScript? = nil
local curSource: string = ""
local dataModuleIdx: number = 0
for i: number, waypoint: WaypointData in waypoints do
if i % 100 == 0 then
task.wait()
print(`Saved {i} of {#waypoints} waypoints`)
end
if not curDataModule or #curSource > 180000 then
if curDataModule then
curSource ..= [[
}]]
curDataModule.Source = curSource
end
dataModuleIdx += 1
curDataModule = Instance.new("ModuleScript")
curDataModule.Name = "DataModule"..tostring(dataModuleIdx)
curDataModule.Parent = bakedWaypoints
curSource = [[
return {
]]
end
curSource ..= "{" .. tostring(waypoint.Position.X) .. "," .. tostring(waypoint.Position.Y) .. ",'" .. waypoint.AreaName .. "'," .. (waypoint.IsWallWaypoint and "1" or "0") .. ",{"
for _, connection: {boolean | number} in waypoint.Connections do
buffer.writef32(buf, 0, connection[2])
curSource ..= "{" .. tostring(connection[1]) .. ",'" ..buffer.tostring(buf) .. "'," .. (connection[3] and "1" or "0") .. "," .. (connection[4] and "1" or "0") .. "},"
end
curSource ..= [[}},
]]
end
curSource ..= [[
}]]
curDataModule.Source = curSource
end
As you can see on the line
curSource ..= "{" .. tostring(connection[1]) .. ",'" .. buffer.tostring(buf) .. "'," .. (connection[3] and "1" or "0") .. "," .. (connection[4] and "1" or "0") .. "},"
I am writing the buffer to the string .. buffer.tostring(buf) , and then I add another .. "'," after it, and then I add some more data, but as seen in the actual source, anything after the first buffer.tostring(buf) just doesn’t appear.
Oh and also, writing anything into or right after the encoded number in the ModuleScript causes this message and kicks me out of the script:
Kicked from Live Scripting Session: Server received illegal operation
If anyone knows if this is a bug or if there is a way around this (or even a better way to save the numbers), then that’d be much, much appreciated!
