How to fit a lot of data in a plugin's settings.json file without making rest of studio plugins implode because of "bad allocation" errors?

Help and Feedback Scripting Support
, ,
#1

Hello.

So since yesterday, I was working on plugin that shows what instances are modified, duplicated, added, removed from specific services using HttpService:GenerateGUID(), CollectionService and plugin:GetSetting/SetSetting APIs.

This project was initially going going well until I decided to save a difference logpoint of a DataModel that has 200k+ instances in it and plugin got to unloading process because of me closing the studio instance. That’s where my problems began, because after saving the plugin setting, the settings.json file size jumped to 1.1 gigabytes and when I reopened the studio, my output console filled with “bad allocation” errors and prevented mine and some other plugins from working:

image
image697×95 3.35 KB

image
image1223×1038 123 KB

Now, I could try to get around these problems by trying to compress the data or some other similar solution like that, the problem is that you can only compress the data so much and and you will hit the same problem again in the future when you use the plugin more and more.

Another possible solution is that I can add a GUI to show places that developer has put Difference Logpoints on, the problem with this solution is that large games can easily hit 1.1 gigabytes limit and hit the problems mentioned above anyways (As I already did).

Which is why I’m posting this topic here, I need help on this matter because I’m currently unable to find an optimal solution on this matter, any help is welcome.

#2

Bumping on this post for an update:
I managed to reduce the size of data from 1.1 gigabytes to ~210 megabytes by only saving properties that have Serialization.CanSave set to true (except for a few properties where I had to add manual override on like CanSave on Source properties of LuaSourceContainer objects to false, ect.) and getting rid of unnecessary datatype properties while saving and excluded parts that are a part of package from the checks and saving process. This seemed have an effect but ~210 megabytes is still high for my liking so still, any suggestions are welcome.

I’m planning to add quality of life improvements like creating a window to list places that developer saved a logpoint on and warning developer when they reach high storage usage and stopping them if saving goes over 1 gigabyte limit on top of existing other places data.