We’re thrilled to announce the public beta of a major upgrade for the DataStore service – DataStore v2.0! It has been a primary initiative for the Developer Services team in the past few quarters and we’re finally ready to let everyone try it out!
The new features and benefits for the update include:
Automatic Versioning: All of the objects, i.e. key-value pairs, in your datastore will be automatically versioned and stored for 30 days; you can retrieve older versions with the new versioning APIs and restore corrupted data when needed.
Datastore and Key Listing: You can enumerate the data stores in an experience and list all the keys in a data store with the new listing APIs, which will be useful to understand data usage and run migrations. The key listing API also supports filtering through prefix.
Data Tagging: You can now set metadata for each object in a datastore (e.g. color of a house and UserID for GDPR requests). Querying and indexing based on such metadata will be supported in future releases.
All APIs are backward compatible so your existing code won’t break. Please refer to the DevHub reference guide and the API references for more details. Note that you have to opt-in to use the functionality by setting the “v2” flag in GetDataStore() API. This applies to both new datastore and existing ones. As we have finished internal migration, all of your data has been versioned regardless of the v2 flag status. However, you have to turn on the v2 flag in order to use the versioning APIs.
Warning: Once you set the v2 flag for a datastore, you should always keep the flag on. Otherwise, you may lose the metadata you set before.
Given that this is a public beta, the features are pretty stable and APIs are unlikely to be changed unless there are serious issues. We feel confident that you can safely integrate them in production games. Once we officially launch DataStore v2.0 as General Availability, all new features will be available by default. No opt-in will be needed.
FAQ
Update on 8/17/2021
We are really excited about the launch of DataStore v2.0 and happy to see all the positive comments supporting the release. We did also notice some common questions pop-up and so want to take the time to clarify them.
Can I change the 30 day limit for versioning?
We set the 30 day limit because we believe that is enough time for you to identify any issues e.g. bugs, data corruption, etc. and restore the data. Also, we have to balance the infrastructure cost while maximizing the value we provide. In the future, we will consider making the time range configurable so some versions can be retained for longer, similar to backup schedules.
What is so exciting about the native versioning support in DataStore v2.0 compared to other solutions?
Versions of the objects are stored in a data structure similar to an append-only list, which can avoid many issues, such as out of sync time in game servers causing new writes to be stored as older writes.
Compared to other pointer based versioning implementations, the new versioning system have the following advantages:
Objects are versioned in one Set/UpdateAsync call. There is no need to write a separate versioning information and Ordered datastore. It is more performant and better utilizes the quota.
Older versions are automatically deleted by the system. It does not require any action from developers nor does it consume any calls from the DataStore quota.
UpdateAsync of an object will correctly create a new version. It is very hard to correctly implement UpdateAsync behavior in a pointer based scheme, since the pointer and the data needs to be kept in sync.
In the future, other features related to backups could be built on top of the versioning system.
Why should I use metadata instead of storing the data in objects?
Although not fully determined, we have been thinking about two potential use cases for it: 1) indexing and querying based on the metadata so that you can build in-game search features or run ad-hoc analysis, and 2) GDPR tools that make it seamless to list all the RtbF entries to review and remove.
By offering cloud services like DataStore, Roblox is the service provider for developers. Roblox does not have a say for what data is in a datastore because you as developers can use it based on your specific needs. As developers of your experiences, you have the full ownership of the data in DataStore. We don’t want to directly delete it unless authorized by you. This is similar to other cloud vendors, such as using the storage service S3 on AWS from Amazon – Amazon doesn’t unilaterally search through and do anything with your S3 data.
Another issue that arises with DataStore is that Roblox does not know what data is associated with each user. For example, if there is a key called “h_1234567”, Roblox is not aware whether this data is for the player with UserID 1234567, or if “h” is for house and 1234567 is some internal house identifier. If we assume the former and we are trying to auto-delete data for right-to-be-forgotten, we may delete some asset for your game that breaks things. If instead we prompt you that h_1234567 may have data related to user ID 1234567, you can then confirm for us that it actually is, or choose to not delete it if that isn’t your data schema.
We are still investigating the details to make GDPR compliance seamless for you. We’ll reveal more once the detailed feature plan is determined. In the meantime, feel free to let us know if you have any suggestions!
We have also noticed several comments about helpful changes to the documentation and have updated the docs, so please feel free to refer to them for previous issues. Thanks again for all your feedback and we will continue to try to answer questions in this FAQ section.
As always, your feedback is invaluable for us to tweak the features and make them fit your needs. Feel free to post any questions you may have. Our team will read the posts regularly and get back to you.
Almost any DataStore improvement is good news. I was extremely excited to work with this update when I heard the features it was bringing, especially since I’m working on a guild feature and would love to have access to key listing for guild listing and anti-duplication on names. Just waiting to pair this up with memory/ephemeral stores, whenever they’re coming.
It also seems extremely easy to make massive migrations to DataStores. I’ve been wanting to majorly change the structure of a production-level game now but it’s difficult to make that kind of system and I don’t want to cheap out by just hard resetting everyone’s data especially when we’ve promised that player data gathered now is theirs to keep. This should make it easier to create a restructuring framework.
In terms of the v2 flag, there isn’t really an explanation for how to set it here… I took a look at the API though, and I guess it should be easy to set? Does this look about right?
EDIT (08/13/2021): The previous code I had was incorrect. DataStoreOptions is an instance not a datatype. I did not realise this was what was explained in a second follow up post. Code sample below was fixed in case anyone used that. Why are these APIs so inconsistent, lol…?
In the future, thanks to the metadata feature and saving userIDs, will we see an automatic way that’s on by default to deal with GDPR requests using this feature?
This new update is super exciting and very useful! In the future, perhaps adding native support for Color3, CFrame, etc values would be a great addition for datastore as well!
Has the issue with the GetAsync method sometimes returning nil without erroring even when there’s data associated with the key been corrected with the release of this?
Never been an issue for me, that’s a concerning report. Are you sure your code respects GetAsync’s cache? GetAsync caches values for 4 seconds, so any subsequent gets would return nil even if there’s data associated with them if the last GetAsync returned nil.
UpdateAsync does use a get and set request initially but it never caches so it might be worth looking into using that when working with updating data in addition to being aware of this cache.
Is there an API for this that can be accessed over HTTP? There’s an existing one for DataStore v1 but it’s not officially supported and has issues from what I’ve tested with it. This would be extremely useful to me since I’d really like to have Discord integration where players can pull up their stats and view leaderboards.
I’ll be honest—how do any of these new features provide tools that weren’t already possibly to use with the “v1” version of data stores? I already track the version of my data and run migrations as the player joins the game. Is there an actual, feasible way to migrate the data of millions of users while a game is shut down? And is there an advantage to doing this over just doing it as the player joins the game?
As far as listing data stores go… I generally don’t have a use case for this. And data tagging might be useful in the future, but I don’t see any API reference on how to use it, or have any idea for how it is supposed to work.
Key listing is an extremely useful feature for migrations, so that’s cool; but will there ever be implementations of transactions for example, trading, purchasing in-game items with in-game currency - without third-party modules having to implement it?
Even if they do, it isn’t as reliable as most database systems offer natively.
Great update, especially since using OrderedDatastores as a backup is no longer required.*
*As long as 30 days is enough. Would be nice to see a feature to modify this.
Really exited for this API to become to default, hope that it can be done seamlessly unlike Pivot, Content, and other APIs that seemed less thought-out. Thanks for the great update!
Will this allow datastore entries to be automatically deleted upon GPDR requests? Does it need to be exactly UserID? Would be nice to have a bit more explanation on this point, as manually auditing GPDR requests is not fun & takes valuable time. Thanks!
Good point, you probably should be able to do that. Normal GetDataStore calls that don’t specify a scope pass nil as the second argument and internally it just changes that argument to the string “global” (as per the default value listing). I’m just not sure about the live behaviour right now since it just released and wrote it in on a whim. Ideally should return the same DataStores as now.
This is AMAZING and long overdue! Nothing else to say about these changes.
The only other reasonable thing that the DataStoreService is missing is batch/transactional queries. Trading systems aren’t able to be created with 100% certainty that players will end up with the correct items. Networks can fail, DataStore limits can be reached, etc. Having a way to do 2 (or more) Set/UpdateAsync calls at the same time + auto rollback if one fails would completely fix the headaches that developers have implementing trading systems.
I could be wrong, but I thought I saw this feature on the 2020/2021 roadmap. It doesn’t seem to be there now.
Ah okay, makes sense. I think DataStore options refers to the instance though I could be wrong? DataStoreOptions seems to be nil by default, and it isn’t documented on the devhub datatype index. Though, under DataStoreOptions it doesn’t have a v2 property, weird.
