I’m trying to paste my game’s localization CSV into an attribute and it gets cut off. It’s also impossible to copy this data from an attribute as it gets cut off in the same way. I plan to support many languages, and my table is going to grow significantly in size over the next few years.
My game needs to be responsive and extremely scalable (by replicating specific key/language pairs as-needed) so I can’t use the cloud table directly. I have utility functions that I frequently use to combine tables together, and I often need to save/load these tables; I’m okay with doing this by copy/pasting directly in the attribute field, but it needs to actually work.
The limit I get isn’t exactly 131072 and seems to depend on the content. I tried it with string.rep("a", 200000) and it was truncated to 131072, but when I try it with my game’s localization table it gets truncated from 165913 to 157118.
This is what I was doing before attributes. The problem is that there’s a limit of 199,999 bytes when setting source from a script. It works for pasting data, just not if I need to process it or combine tables. I’m adding support for more languages and my CSV is getting pretty close to this limit.
I suppose the root of the problem is that the keys are all out of order when I download the cloud table, but I’d like to be able to copy any large piece of data out of studio easily. Otherwise I would need to save it as a rbxm and write a custom decoder in another IDE (I already have a rbxm encoder for my game’s animation data), or save it as a rbxmx and copy the table there (if attributes are plain text there.)
-- copy-attribute.lua
local model = DataModel.new() -- Container for the model.
local spreadsheet = Instance.new("Folder", model) -- Container for the attribute.
spreadsheet.Name = "Spreadsheet"
local csv = fs.read((...), "bin") -- Read the given file, in raw binary (bin) format.
spreadsheet:SetAttribute("CSV", csv) -- Set the CSV attribute to the csv data.
clipboard.write(model, "rbxm") -- Write the model to the clipboard, ready to be pasted into Studio.
This can be run with the following command:
rbxmk run copy-attribute.lua file.csv
To do the reverse:
-- paste-attribute.lua
local model = clipboard.read("rbxm") -- Read a model copied from Studio.
local spreadsheet = model:Descend("Spreadsheet") -- Get the attribute container (same as `model.Spreadsheet`).
local csv = spreadsheet:GetAttribute("CSV") -- Get raw csv spreadsheet from attribute.
fs.write((...), csv, "bin") -- Write raw csv data to given file in binary (bin) format.
Run with:
rbxmk run paste-attribute.lua file.csv
Large amounts of data appears to copy and paste just fine this way, so the problem is likely with the properties panel.
That’s a great workaround! I wrote an rbxm writer in python last year to streamline adding custom binary animations to my game through blender, but it only implements a few basic types for what I needed. That looks like a really useful tool.
It looks like this applies to string properties in general, and not just attributes. 131072 doesn’t seem super meaningful, so it makes sense for us to look into expanding that to 199,999 to match the API limit for strings.
To clarify, while the API for attributes does not currently limit the size of strings, this was an oversight on our part when shipping the API & we are investigating limits for it now. Your large CSVs may not be supported by attributes in the future. You can make a feature request for us to support your specific use case, but we may not be able to honor extremely specific requests that no other developer needs. My recommendation would be that you store your translation tables in LocalizationTable instances as intended (we even support directly importing them from/exporting them to CSVs on disk) & then you can read the contents with GetEntries to replace the reading you are doing via attributes now.
Undocumented arbitrary limitations like this don’t help anyone. Any system I design that needs to store data is going to need to account for this limitation and save in multiple keys or instances. Even if I don’t expect to need more than 10kb, I won’t be able to sleep at night knowing a larger file will cause the entire system to break when I least expect and cost me time to fix.
This goes far beyond the specific CSV storage use case. I have run into the 199,999 limitation many times and it gives me anxiety whenever I work with anything big:
Processing .bmp data for generated heightmaps in custom terrain.
Processing .obj data for custom terrain.
Using Roblox Studio as a tool for processing/generating files. I’m efficient at writing Lua code, so it’s my go-to for quickly processing strings or files.
Generating long strings in Roblox Studio for other projects. For example: I used it last year to quickly generate a powershell command given a list of input file paths so I can organize stuff. I’m unable to do this easily when the output is too big. This was quicker than mastering powershell.
Processing my entire Roblox sales data (that I scraped) to recover purchases that some players lost. This file was massive even when compressed somewhat.
Storing precise animation data. I rarely need more than 10-20kb, but multidimensional animations (like directional walking/running) can get huge. I know custom animation systems aren’t expected from developers. Regardless, I ended up spending days developing an acceleration-based delta-encoded animation compression format that never sees the game client, mainly to keep my save file from ballooning, but also to stay clear of the 199,999 value limitation. You can see the result of this here: What are you working on currently? (2020) - #5135 by Tomarty
Storing binary map chunk data for my game. Getting multiple 200k-long StringValues to work with undo/redo was a nightmare, but it allows me to create a massively scalable object placement system: What are you working on currently? (2020) - #3568 by Tomarty
Storing player saves for unit testing. The DataStore limit was raised to 4MB, but I am unable to store test saves above 199,999 characters outside of source code, which impacts “Find in all scripts” performance.
Generating source code. Realistically you shouldn’t need more than 200k characters for even a generated script, but I have hit this limitation a few dozen times over the years. Most of the time these large scripts never see the game client and are just used for storing data. I’m closer than ever to hitting this limitation for a few specific game scripts with the introduction of Luau type annotations.
Source code #1: Animation function generation - I have an auto-generated module that returns an optimal function based on what the bone needs to update. While once relatively small, it is now currently 150,000+ characters since I added type annotations.
Source code #2: Viewing logs - I occasionally need to few a log or file that I created. The script editor is a good place to do this but I run into problems if the log is more than 199,999 characters.
Source code #3: Game data - I store my game’s compiled data in ModuleScripts. This is mostly tables. I spent a lot of time making it divide and spread data across multiple ModuleScripts because of this limitation. It was years ago, but it still haunts me.
CSV data not related to localization. I have used spreadsheets for various uses. Most recently I have a spreadsheet for generating starter characters based on a player’s character appearance when they join.
This just scratches the surface. Neural net training data, custom mesh collision data, texture data for custom renderers, etc. Developers are highly creative and will only be held back by this limitation.
Roblox is about empowering imagination, and it’s not hard to imagine needing more than 200 kilobytes. I will waste hundreds of hours working around whatever limitations I end up with so that I can create something great. Please reconsider, for the sake of all future developers and their sanity.
