DocumentService - A powerful, schematised DataStore library

Resources Community Resources
, , , ,
#1

DocumentService

DocumentService is an open-source Luau library for saving data with Roblox DataStores.
It can be used for sesssion-locked data, such as player data, and for non-session-locked data, like
shared groups or houses. It has several advantages over other available libraries, such as being fully strictly typed.

Links

  • Repository on GitHub for downloading & viewing the source code, reporting bugs and making contributions.
  • Documentation & examples for help with using DocumentService.
  • API reference
  • There is also a thread in the Roblox OSS discord server under “projects”.
  • TypeScript version: coming soon!! (not maintained by myself)

Features

  • Fully strictly typed internals and API. This means you get full intellisense and typechecking on your data and on every API method. Your schema defines the type of your data.
  • Superior Rust-inspired error handling, with a Result type for each method
    that provides unique intellisense on which errors you need to handle.
  • Session-locking: Documents can be session-locked, or not (to allow multi-server editing). Session-locking simply extends the API, so it is easy to use DocumentService for non-player data.
  • Migrations, inspired by Lapis by @nezuo, which allow you to mutate your schema over time.
  • Validate your data against your schema with support for runtime and static typechecking.
  • Immutable cache and autosaves - this prevents bugs caused by updates committing too early and accidental mutations.
  • Run hooks before and after operations. You could use this to implement your own logging or serialisation.
  • Automatic retry with exponential backoff.
  • Checks your data can be stored in DataStores to avoid silent errors.
  • No dependencies (like Promise). Use whatever abstractions you like, and install easily.
  • You can inject a MockDataStore of your choosing. I recommend this one.

Installation

Method 1: Wally

Add DocumentService = "anthony0br/documentservice@LATEST_VERSION" to your wally.toml.

Method 2: Manual

DocumentService has no dependencies so you can just copy and paste the contents of
target/roblox into your project.

Method 3: Roblox Asset Library

You can get it as a model here. Note that this may not be as well-maintained as the other methods!

Frequently Asked Questions

Why use this over ProfileService?

  • DocumentService has far superior error handling, inspired by Rust’s Result type. Intellisense tells you which errors you need to handle for each method - and only the ones relevant to that method. This helps you write better code, faster.
  • It’s fully typed checked. This means you’re less likely to make mistakes and don’t need to spend time cross-referencing the documentation!
  • It’s just as powerful, but with a much simpler API that follows SOLID principles.
  • Validation - ProfileService does not validate your data. This means an accidental mistake could cause data inconsistency or corruption, and if data is manually changed you could get weird errors and side effects. DocumentService lets you validate data before you save it and as you load it, preventing bugs. DocumentService will also error if your data is unsavable, making issues obvious to you before you publish your game - ProfileService will not.
  • Migrations - in ProfileService, if you want to change how your data is formatted, you can’t. You can only add new stuff! This can limit long-term maintenance. DocumentService supports migrations (inspired by Lapis), which allow you to mutate your schema over time and mark changes as not backwards compatible.
  • ProfileService’s API is very player-centric, which makes it difficult to work with data that isn’t session-locked. DocumentService is designed so that session-locking extends the API, rather than defines it. You can turn it off and still use methods that don’t involve caching in the exact same way. DocumentService also has a method designed specifically to work with non-session-locked data efficiently called :OpenAndUpdate.
  • Cache updates are immutable in DocumentService, which helps prevent bugs. In ProfileService, they are mutable.
  • ProfileService is outdated and hasn’t been maintained in several years - for example it still imposes a 7 second wait time between requests, which is no longer needed, and was written in Lua 5.1 rather than Luau.
  • DocumentService’s source code is easier to read and maintain.

But what about MetaTags and GlobalUpdates?

You don’t need MetaTags. Just create a new table in your data. You can implement GlobalUpdates by first checking if the player is in the current server and :IsOpenAvailable. Then,

  1. In the case they are in the same server - just directly update their data. Otherwise:
  2. In the case :IsOpenAvailable returns false, use MessagingService to update their data from the server where their data is open.
  3. In the case :IsOpenAvailable returns true - use :OpenAndUpdate to open and update their data and then close the document.

Give me an example!

Sure, here are some:

How does it compare to other popular alternatives - Lapis, DataKeep, etc?

Lapis and DataKeep are focussed on session-locked data and are not fully strictly typed. They return Promises, whereas DocumentService will yield and return a Result type. DataKeep also does not feature migrations, but has natively implemented GlobalUpdates. I would highly recommend these two projects as alternatives to ProfileService / Suphi’s if you don’t like DocumentService. DocumentSevice also has more powerful hooks, with the ability to run hooks on any main operation and cancel them at any time.

I have an existing game - how do I start using this?

If you try to open a Document on a key which already contains data, but has not been used with DocumentService before, DocumentService will try use the existing data, so it’s easy to migrate - especially if you are not using any library. I’d recommend starting out by following the examples and writing a file for your schema (including your types and check function) based on your existing data. You could also use migrations to tidy up your data!

Disclaimer

This is a new release and, although it has been thoroughly unit tested, it hasn’t yet been used
in a live production game - as with any open source software, use it at your own risk! I am
working on adding it to the games I work on, which currently peak at ~2k CCU, so this won’t be a concern for too long.

Special thanks to @nezuo and @kineticwallet for helping to review my code and API, and for creating a mock DataStoreService which has been incredibly useful.

37 Likes
ProfileStore - Save your player data easy (DataStore Module)
Collection of Useful Community Services for Roblox Developers
Save your player data with ProfileService! (DataStore Module)
Collection of Useful Community Services for Roblox Developers
Collection of Useful Community Services for Roblox Developers
#2

Looks interesting, I’ll give it a try

3 Likes
#3

Looks good, currently using Datakeep.

But can you please elaborate on what these migrations are? Are they just like Global updates or is there any key difference?

2 Likes
#4

Migrations are like the reconcile feature in ProfileService/DataKeep.

You create a function that transforms each version to a new version. These can add fields, change them, or even remove them. You can flag them as not backwards compatible, which prevents them from being loaded by old servers.

If you want to just reconcile, you can create a single migration that does this.

Migrations in DocumentService are based on migrations in Lapis - so this page explains it well:

2 Likes
#5

The Roblox model is not accessible at the moment.

Image
Image1198Ă—976 213 KB

4 Likes
#6

Since this post iv received a few messages so I would just like to respond just to help anyone who has any questions about SDM

Its true that SDM uses the MemoryStore to hold the lock this was done by design but was also designed in a way to be very resilient to MemoryStore failures and the DataStore is not contingent on the success of the MemoryStore in order to save.

In order for the session to close by default the MemoryStore will need to fail 15 times in a row over the time span of 5 minutes during these 5 minutes the DataStore will continue to save and operate as normal.

Once the 15th MemoryStore attempt has failed the DataStore will save 1 last time and the session will gracefully close calling the state changed event informing you that the state of the session has closed, no data during this time will be lost (as long as the DataStore servers are still functional)

Because SDM uses the MemoryStore it only needs to use SetAsync to save data into the datastore this is faster then using UpdateAsync and also has no chance of function re-runs also because SDM only uses SetAsync to save into the DataStore it only consumes 1 set request where UpdateAsync will consume 1 set request + 1 get request causing SDM to consume 50% fewer DataStore requests

SDM is sandboxed preventing you from making any mistakes or breaking the module giving it excellent error handling for example

SDM.LockInterval = "String" -- Attempt to set LockInterval failed: Passed value is not a number
SDM.LockInterval = 3 -- Attempt to set LockInterval failed: Passed value is less then 10
SDM.LockInterval = 10 -- No error
6 Likes
#7

oops, i’ll fix that with v1.1.0 which will release tomorrow

2 Likes
#8

Hey, thanks for taking the time to reply - didn’t expect you to notice this!

I’ll admit I am not too familiar with your library - that’s why the comment is quite brief compared to the comparison with ProfileService. I wasn’t trying to suggest that it is unsafe to use your library, but I think that the use of MemoryStores for locks isn’t a good design choice.

In response to your justifications:

From my experience, UpdateAsync calls are fast - I doubt a combination of a MemoryStores call and a Get/Set DataStore call would be noticeably faster. With correct usage, you shouldn’t be anywhere near the limits. Also, currently UpdateAsync only contributes to the Set limit. The difference, in practice, is almost nothing compared to the limits. Therefore, in my opinion it just adds needless complexity. To run unit tests now you also need a MemoryStore Mock, and there are much more error cases to handle.

UpdateAsyncs are better in other ways too - they will always apply in the correct order and do not use cache. This is really important when dealing with data that isn’t session-locked, which DocumentService is designed to do in addition to just player data. It also ensures the latest data is fetched - e.g. if there is an UpdateAsync in progress closing a Document, the UpdateAsync call responsible for opening the document will yield until the one closing is completed and then run the transform. Of course, the transform can re-run, but it is easy to write pure or pure-ish functions that do not have any problems being re-run. If you’re doing multiple server editing, it is never safe to update data with GetAsync and SetAsync.

Another danger of GetAsync is its cache, which I hope you’ve disabled!

Sandboxing isn’t necessary if you have static typechecking. There actually is no way to change things like autosave period in DocumentService without creating a fork, and I don’t want to add a method for this (there’s no use case for changing it)!

I will remove my comparison with your library until I can give better information. I apologise for any alarm or DM spam I caused!

4 Likes
#9

This has now been fixed. (still on v1.0.0 but v1.1.0 is coming soon)

1 Like
#10

Seems… Pretty cool! might actually switch once V.1.1.0 releases.

2 Likes
#11

It’s out, I’m just having some trouble installing rokit (which is required to publish it to wally now) :sob:

Here are the release notes.

1 Like
#12

Release 1.1.1 is out now

1 Like
#13

New Example: Updating a Player’s data through cache

1 Like
#14

Just tried it out. And honestly, all I can say is that I absolutely love it. Keep it up! For now, I don’t have any suggestions, though.

2 Likes
#15

I hope the docs and module can be updated to be more beginner-friendly! This seems very promising, but as a rookie DataStore programmer this is extremely hard to understand as the docs seem incomplete and/or hard to understand, plus the module itself seems hard to use. I have not used this too much though, and again, I am not too good with DataStores, so this could most likely be an issue on my part.

4 Likes
#16

Will be checking this out! Great release.

1 Like
#17

Would be great to see something like replica service implemented for this.

1 Like
#18

I do agree they aren’t the most comprehensive, but writing docs is time-consuming! Pull requests for improvements are always welcome though :slight_smile:

1 Like
#19

Nezuo has a good one afaik, you could check that out

1 Like
#20

how do I learn how to use documentservice?

1 Like