What's the best way to index large DataStores for info?

Hello,

I’m trying to make a marketplace in a way for my game. Payers can add stuff, and others can search for the things using filters, which will be determined in the data.

The only issue is that the DataStore may get big quickly. My idea was to have an ID system. When data is written, save it as an ID. The thing is, when I want to load a homescreen, this is very inefficient and is highly likely to ratelimit.

I’m not sure what the proper way to do this would be.

I tried using OrderedDataStores, but that will likely not work in my case, and I’m a bit unsure as to how it works.

Thanks for any help.

1 Like

maybe have it cached on the server and whenever a player wants loads in the homescreen, it gets the data from the server maybe in json, the client can store the decoded data for later use

to prevent rate limit, whenever a player loads the homescreen, instead of using GetAsync straight away, you can check when was the last time the data was fetched, if its within your desirable period of time, you can return the cache
you can use UpdateAsync when writing to data, you can put requests in queues and update them in batch every so often, since it also returns the updated data, you can update the cache to that

hope this is some help

1 Like

Thanks, I’ll try this solution and get back.

I’ll be making something just like @pyxfluff in the future. Do you mean to slowly load everything once the player joins, and then just cache it until the player requests some sort of reload?

yeah, the request for reload can be whenever a player wants to load in the homescreen, so they can make one invoke to the server and the server can check how old is the cached data, if its fairly new then you can return it instead of calling GetAsync

you can also cache the returned data on the client and use it until you determine it’s too old before requesting from the server again