We’re thrilled to announce the public beta of a major upgrade for the DataStore service – DataStore v2.0! It has been a primary initiative for the Developer Services team in the past few quarters and we’re finally ready to let everyone try it out!
The new features and benefits for the update include:
- Automatic Versioning: All of the objects, i.e. key-value pairs, in your datastore will be automatically versioned and stored for 30 days; you can retrieve older versions with the new versioning APIs and restore corrupted data when needed.
- Datastore and Key Listing: You can enumerate the data stores in an experience and list all the keys in a data store with the new listing APIs, which will be useful to understand data usage and run migrations. The key listing API also supports filtering through prefix.
- Data Tagging: You can now set metadata for each object in a datastore (e.g. color of a house and UserID for GDPR requests). Querying and indexing based on such metadata will be supported in future releases.
All APIs are backward compatible so your existing code won’t break. Please refer to the DevHub reference guide and the API references for more details. Note that you have to opt-in to use the functionality by setting the “v2” flag in GetDataStore() API. This applies to both new datastore and existing ones. As we have finished internal migration, all of your data has been versioned regardless of the v2 flag status. However, you have to turn on the v2 flag in order to use the versioning APIs.
Warning: Once you set the v2 flag for a datastore, you should always keep the flag on. Otherwise, you may lose the metadata you set before.
Given that this is a public beta, the features are pretty stable and APIs are unlikely to be changed unless there are serious issues. We feel confident that you can safely integrate them in production games. Once we officially launch DataStore v2.0 as General Availability, all new features will be available by default. No opt-in will be needed.
Update on 8/17/2021
We are really excited about the launch of DataStore v2.0 and happy to see all the positive comments supporting the release. We did also notice some common questions pop-up and so want to take the time to clarify them.
Q1. Can developers change the 30 day limit for versioning?
We set the 30 day limit because we believe that is enough time for you to identify any issues e.g. bugs, data corruption, etc. and restore the data. Also, we have to balance the infrastructure cost while maximizing the value we provide. In the future, we will consider making the time range configurable so some versions can be retained for longer, similar to backup schedules.
Q2. What is so exciting about the native versioning support in DataStore v2.0 compared to other solutions?
Versions of the objects are stored in a data structure similar to an append-only list, which can avoid many issues, such as out of sync time in game servers causing new writes to be stored as older writes.
Compared to other pointer based versioning implementations, the new versioning system have the following advantages:
- Objects are versioned in one Set/UpdateAsync call. There is no need to write a separate versioning information and Ordered datastore. It is more performant and better utilizes the quota.
- Older versions are automatically deleted by the system. It does not require any action from developers nor does it consume any calls from the DataStore quota.
- UpdateAsync of an object will correctly create a new version. It is very hard to correctly implement UpdateAsync behavior in a pointer based scheme, since the pointer and the data needs to be kept in sync.
- In the future, other features related to backups could be built on top of the versioning system.
Q3. Why should I use metadata instead of storing the data in objects?
Although not fully determined, we have been thinking about two potential use cases for it: 1) indexing and querying based on the metadata so that you can build in-game search features or run ad-hoc analysis, and 2) GDPR tools that make it seamless to list all the RtbF entries to review and remove.
Q4. Why doesn’t Roblox automatically handle GDPR requests?
By offering cloud services like DataStore, Roblox is the service provider for developers. Roblox does not have a say for what data is in a datastore because you as developers can use it based on your specific needs. As developers of your experiences, you have the full ownership of the data in DataStore. We don’t want to directly delete it unless authorized by you. This is similar to other cloud vendors, such as using the storage service S3 on AWS from Amazon – Amazon doesn’t unilaterally search through and do anything with your S3 data.
Another issue that arises with DataStore is that Roblox does not know what data is associated with each user. For example, if there is a key called “h_1234567”, Roblox is not aware whether this data is for the player with UserID 1234567, or if “h” is for house and 1234567 is some internal house identifier. If we assume the former and we are trying to auto-delete data for right-to-be-forgotten, we may delete some asset for your game that breaks things. If instead we prompt you that h_1234567 may have data related to user ID 1234567, you can then confirm for us that it actually is, or choose to not delete it if that isn’t your data schema.
We are still investigating the details to make GDPR compliance seamless for you. We’ll reveal more once the detailed feature plan is determined. In the meantime, feel free to let us know if you have any suggestions!
We have also noticed several comments about helpful changes to the documentation and have updated the docs, so please feel free to refer to them for previous issues. Thanks again for all your feedback and we will continue to try to answer questions in this FAQ section.
As always, your feedback is invaluable for us to tweak the features and make them fit your needs. Feel free to post any questions you may have. Our team will read the posts regularly and get back to you.
The Roblox Developer Services Team