i’ve created a neural network, but to be able to train it, i need a way to store the actual network, which i have in tons of tables inside of tons of tables, and transfer it into another model.
i dont know how to take the huge table and put it into a widely accessible area (since the table is a variable inside a script)
Save the training data inside Roblox datastores. You can try compressing it(there’re community made libraries for this) and separate it to 4MB segments if it’s still too large.
Working with a large amount of data will definitely make things more challenging but if you want to share any data across scripts just use a modulescript.
You should use a module and require it from each thing that needs it. As long as it is only needed within that server you should be fine.
@SleepyyRico beat me to it lol
The negative aspect of using modulescripts is that when large tables or strings are pasted in them they create studio lag. This will make place uploading/updating times slower and in general create a bad editing experience.
That lag is only local to studio; data store usage would not only eat up the data budget but also potentially cause server lag, both of these could eventually lead to player data loss depending on code structure. A module would be better.
well i need to store 5 tables at once (the 5 tables being the 5 highest performance neural networks)
They mention training data and they only need to fetch the data once. As long the compressed version of the data isn’t way too large(for example a giga byte) it should not take much to retrieve all the segments of the data. In fact it’s highly possible that after compression the total JSON encoded string size will be less than 4MB.
Data loss can only be created if they eat up the datastores requests per limit and they force-throttle the other players waiting for their data. This can be fixed with retry logic. Basically the people that will load in slow due to that will only be the ones joining first, if the training data itself is way too large.
Basically with good coding, not an issue.
Also why exactly you need the training data, can’t you just put the network/layer data in studio after it has been trained?
Yes, but if data throttling can be avoided, it should. It would probably be less efficient to use a data store and take longer to retrieve the data, not to mention it would be more work to implement. But if @ew_iscool really wants to avoid some studio lag, they could use a data store in studio and a module in servers, using RunService:IsStudio() to differentiate between the two.
It might also be pretty useless if the services are interconnected, more useful to be returned through a module than reading from a data store a bunch of times.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.