Announcing DataStore v2.0 - Automatic Versioning, Data Tagging, & Listing!

Yesyesyesyesyesyesyesyes! I have been waiting for this update for months. I need to know when this update will come out. The feature I want the most, though, is saving instances. That was a feature of the legacy Data Persistence, so I really want that to be added.

2 Likes

This is nice and all, but it seems super wasteful for things like session locking, where only one value in a big table is being changed, which would create thousands of useless versions for active players. Is there anything I can do to avoid straining the systems?

1 Like

Versioning isn’t useless with session-locking :face_with_raised_eyebrow:
ProfileService added features to check older versions of data, and also overwrite the newest version with them, if there’s any little accident on your part, it’s super helpful.

This isn’t a problem, in fact I was able to get a single key to have 7k+ versions while cleaning older versions was disabled, and that didn’t make datastore requests any slower. My :UpdateAsync calls still took around 0.25-0.75~ seconds.

No, older versions already get cleaned up after 30 days, Roblox expects a bunch of versions to be created at all times, and that’s fine, they knew what they were getting themselves into, and they commited to it.
+ This doesn’t affect speed, efficiency, or anything else for you.

3 Likes

It may not do so yet, but at least the feature is there with the intention of automatically handling them in the future (at least, I’d imagine that’s the whole point of it existing in the first place).

2 Likes

Are there any plans to increase datastore limits, mainly for get requests? In my game players can build “machines”, each machine has a GUID key used to save it to datastores, then I have an array of metadata for each machine inside the player’s data

The list below is populated with the metadata from the player’s data file. The metadata only has the name and creator ID of the machine. When a player clicks on one of the buttons, the game makes a get request to download the machine’s content. Current datastore limitations means players can only safely click on 10 buttons per minute at most, and when browsing the library looking for new machines that limit gets hit very fast

3 Likes

Depending on your game, I would say that 6 seconds is a reasonable wait time between different machines. I don’t think someone would just move on to another machine that quickly to require more than 10 requests a minute. But then again, I have no idea how these machines properly work and if it would be better for someone to move on after 1-5 seconds.

Edit: I forgot about there being other players, so my idea probably doesn’t work.

1 Like

Yay! I’ve been waiting for a good change to the DSS for a long time now!

2 Likes

Great feature yet the fact that OrderedDataStores can’t use this kind of make this useless to me since my projects used OrderedDataStores.

OrderedDataStore does not support v2.0 features such as versioning, so calling this function on an OrderedDataStore key will overwrite the current value and make previous versions inaccessible.

It’s great to hear that the Developer Services Team is looking into making it easier for developers to handle GDPR requests!

Here are a couple of ideas regarding developer-facing APIs for automating those requests:

Idea 1: Allow Developers to Enable Deletion of Data Tagged With a Single UserId

  • Given
    • The DataStore v2.0 API now allows developers to tag data with a table of UserIds.
    • There is now a DataStoreOptions instance.
  • Assumptions:
    • Data which is tagged with a table containing a single UserId is most likely associated only with that UserId.
  • Proposed API:
    • Add an additional option to DataStoreOptions which allows developers to enable automatic GDPR deletion of data tagged with a table containing a single UserId.
  • Additional Thoughts:
    • Pros:
      • By allowing developers to opt-in to deletion of data associated with one UserId, Roblox allows developers to maintain ownership of the data while also providing a method to automate a portion of GDPR requests.
      • From a developer perspective, this option is easy to implement.
    • Cons:
      • This option still requires developers to manually handle deletion of data which is associated with multiple users.

Idea 2: Provide a Way to Register a Callback for GDPR Requests

  • Given
    • Developers currently manually handle GDPR requests.
  • Assumptions:
    • In order to handle GDPR requests, developers theoretically know how an experience’s data is organized.
    • Developers likely already have code for handling GDPR requests.
  • Proposed API:
    • Allow developers to register a callback for an experience, which Roblox would automatically call for GDPR requests. A list of UserIds would be provided to the callback, which would handle the logic to delete the data associated with those UserIds from the experience’s DataStores. This is similar to creating an AWS Lambda function. If a callback is not registered for an experience, Roblox can notify the developer to handle the request manually.
  • Additional Thoughts:
    • Pros:
      • Allows developers to fully manage GDPR deletion logic.
    • Cons:
      • Requires more up-front setup by developers in comparison to Idea 1.
4 Likes

Wow, I am overwhelmed! I just programmed my first DataStore (v1?) a few hours ago, discovered DataStore2 module an hour ago when I ran into save limits almost immediately, and now DataStore v2.

I’ll be honest, I have no idea what’s what, but this is only 8 days old, so I guess I’m starting here!

1 Like

finally, better datastores. this is gonna be nice

It doesn’t seem like pageSize works for ListKeysAsync. No matter what number I put for pageSize, I always get just one item per page. Anyone else experience this?

1 Like

Just tested and this is happening for me as well.

How many keys are there in the datastore ? This can happen if there are very few keys in the datastore. Can you try with a datastore with 100+ keys ?

Also the API will always return less than pageSize number of entries, it does not always return the pageSize entries.

Yes, it worked better with more data. What’s the reason for this behavior with small sets of data? If I have just 5 keys, it’s a bit odd having to do 5 different fetches for those keys, rather than just one fetch.

2 Likes

The system is optimized to scale for large datastores. We will look into optimizing for smaller sets of data.

3 Likes

It would be cool if there was a way to fetch what scopes existed for each named DataStore too. As far as I can see, that’s the last remaining “unknown” in terms of finding what exists.

My particular use-case is extending my DataStore visualizer plugin. I can list out the DataStores that exist in a game, but cannot show the user what scopes are included in each store.

3 Likes

We have made Scopes prefix of the object key. AllScopes option was added specifically for the purpose of iterating through all keys in a datastore in all scopes.
In this documentation page
https://developer.roblox.com/en-us/articles/Data-store
AllScopes section contains sample code and sample output.

2 Likes

Can anyone answer this?

No, versions don’t consume the quota. You still have 4MB size limit for the latest version of an object.

1 Like