The problem of user generated content & full data store keys

Help and Feedback Scripting Support
#1

So this problem has lingered in my mind for almost a year now. I have been constantly thinking of creating a game based on user-generated content that must be saved in data stores. The only problem with this is the limit to each key (around 200,000). With UGC comes a very long list of information that needs to be packed into a data store key (pretty much the simple way won’t cut it).

Basically, what I’m trying to do is have a system that will automatically save information in a new data store key when the previous one is full.

This comes with a task just as difficult which is grabbing information from each key. You have to know which keys exist.

Not sure if anyone can help but if you can that would be great. Not looking for someone to write an entire system for me just give me useful information.

1 Like
#2

So to do this I would do the following:

Saving:

  1. Get the current cache of data
  2. Encode it in JSON (since I assume it’ll be a dict?)
  3. Compress it using something like LZW so we can attempt to save some characters in each key
  4. Split the compressed string into chunks (based on the max key size i.e. the character limit)
  5. Save the chunks to keys
  6. Save the keys to an ordered datastore in sequential order so we know how and in what order to collect the info when we try to load it

Loading:

  1. Have an ordered datastore to store the sequential keys of the real data (preserves the sequence in which we’ll need to decompress the data later)
  2. Get the keys from the ordered datastore on each server instance
  3. Get the packets of data from each of these keys
  4. Once we have all the packets of data, concat them into one string
  5. Decompress it if we’ve used something like LZW
  6. Decode the JSON to get the dict back
  7. Enjoy your data

Edit: I’ve just realised you want this per person. I’d skip the ordered datastore all together and just have 2 seperate datastores (load in Player datastore first to get keys, then load in the UGC datastore from the keys selected):

  1. A player datastore which has a dictionary which would look something like:
PlayerData = {
	-- Whatever other important stuff you want;
	keys = {
		-- 'keys' being a list of all UGC content items they've created e.g. let's say they built some sort of boat
		UGC_Content_1 _BOAT= {
			[1] = 'some_key_reference';
			[2] = 'some_key_reference_2';
			-- and so forth for n keys
		}
	}
}
  1. Then, in another datastore to save all those UGC content to, referenced by the keys ‘some_key_reference’ etc. When a player loads into the server, repeat the methods I mentioned above, but for per person and using the keys in their ‘keys’ array for that specific UGC content they want loaded.
2 Likes
#3

I will definitely be trying something very similar to this! May take a few hours to tweak it and get it running with the UI (that I have totally created already) but I’ll let you know and accept your answer once I get it working.

#4

No worries, good luck! Can confirm the latter method of saving the keys to a seperate player datastore does work as I have previously implemented it for a base building game :slight_smile:

1 Like
#5

I’m just going to mark this as the solution because I never got around to actually finishing it and it was working so far without a problem. I will 100% be revisiting this in the future though.

#6
#7