How to use DataStore2 - Data Store caching and data loss prevention


#1

DataStore2 is an open source module that uses berezaa’s method of saving data that prevents data loss, caches, and verifies data before saving. DataStore2 has been used in a game visited by 15 million players with a constant concurrent player count of around 3,000 - 8,000 players, one time reaching 12,000 concurrent players. To this day, I have yet to receive a single case of data loss.

Where do I get it?

DataStore2 is open source on my GitHub. It is also on Roblox as a free model. You can also require it and make sure it’s always up to date by using require(1936396537). You may not want to do this for security purposes. I highly recommend “watching” the GitHub repository if you are not requiring the ID directly. I update DataStore2 enough times to warrant keeping an eye on it.

API

DataStore2 DataStore2(dataStoreName, player)

Example usage: local coinStore = DataStore2(“coins”, player)

This is what the module returns when you require it. You usually use this on PlayerAdded. Note that the data store name will not be what the name of the data store (because of how the saving method works).

Variant DataStore2:Get(defaultValue=nil, dontAttemptGet=false)

Example usage: coinStore:Get(0)

If there is no cached value, it will attempt to get the value in the data store. Otherwise, it’ll return the cached value. If whatever value it gets is nil, it will return the defaultValue passed. If dontAttemptGet is true, then it will return nil if there is no cached value.

void DataStore2:Set(value)

Example usage: coinStore:Set(100)

Sets the cached value to whatever you passed in.

void DataStore2:Update(updateFunc)

Example usage: coinStore:Update(function(oldValue) return oldValue + 100 end)

Calls the function provided, passing in the current cached value (can be nil). Sets the cached value to the return value.

void DataStore2:Increment(value, defaultValue=nil)

Example usage: coinStore:Increment(100)

Increments the current cached value by value. If there is no cached value, the defaultValue is used before incrementing. To make this simpler to understand, :Increment is implemented simply as…

self:Set(self:Get(defaultValue) + value)

void DataStore2:OnUpdate(callback)

Example usage: coinStore:OnUpdate(print)

Adds a callback function used every time the cached value updates. Not called during :Get().

void DataStore2:Save()

Example usage: coinStore:Save()

Saves the cached result to the data store. Automatically called on PlayerRemoving. You generally shouldn’t call this on your own unless there’s no better reason to. You may want to use this on purchases to make sure they save.

void DataStore2:AfterSave(callback)

Example usage: coinStore:AfterSave(print)

Adds a callback to be ran after the data store is saved. Will yield :Save().

There are other functions but they don’t have much use. Read the GitHub to see the undocumented functions. BeforeInitialGet and BeforeSave are covered in the serializers and deserializers section.

Example Code

This code is an example of what your code may look like if you were making a coins system.

local ReplicatedStorage = game:GetService("ReplicatedStorage")
local Players = game:GetService("Players")

local DataStore2 = require(1936396537) --Or wherever you put the free model if you did.

Players.PlayerAdded:connect(function(player)
	local coinStore = DataStore2("coins", player)

	local function callRemote(value)
		ReplicatedStorage.CoinAmount:FireClient(player, value)
	end

	--Fire a remote event to the player telling them how many coins they have.
	--If this is their first time playing the game, they'll start out with 100.
	callRemote(coinStore:Get(100))

	--Everytime the coin store updates, we'll send the RemoteEvent again.
	coinStore:OnUpdate(callRemote)
end)

--This is a RemoteEvent where a player can purchase a product by its name.
local Products = require(ReplicatedStorage.Products)

ReplicatedStorage.BuyProduct:connect(function(player, productName)
	if not Products[productName] then return end --Make sure the player is buying a real product

	local coinStore = DataStore2("coins", player)
	local productPrice = Products[productName].price
	
	if coinStore:Get(100) >= productPrice then
		print("Buying product", productName)
		coinStore:Increment(-productPrice)
	end
end)

Serializers and Deserializers

DataStore2 has built in support for easy serializing and deserializing with the use of the BeforeInitialGet and BeforeSave functions.

When you call DataStore2:BeforeInitialGet(callback) (must be called before Get), when Get() is called for the first time (and there is a value in the data store), the cached value will be set using the return of the callbacks provided.

When you call DataStore2:BeforeSave(callback) (must be called before Save), when Save() is called, the cached value will first go through the callbacks provided. The value that will be saved in the data store will be the returns of these callbacks.

As an example, let’s say you wanted to save a player’s inventory, but you wanted to optimize the save. For example, if a player has an inventory of {"Item1", "Item2", "Item3"}, and you want to save it as {1, 2, 3}. You can do this easily with these functions. Your code may look like this.

local Players = game:GetService("Players")

local DataStore2 = require(1936396537) --Or wherever you put the free model if you did.

--The list of items. The keys are what you want your code to read, but the values are what you want saved.
local Items = {
	["Item1"] = 1,
	["Item2"] = 2,
	["Item3"] = 3,
}

Players.PlayerAdded:connect(function(player)
	local inventoryStore = DataStore2("inventory", player)

	inventoryStore:BeforeInitialGet(function(inventory)
		--BeforeInitialGet IS NOT CALLED if there is no value in the data store.
		--That means that the value we have now is serialized.
		--The format of this value looks something like {1, 2, 3}.
		--We're going to change this to the string representations to make it much easier to code with.
		local deserialized = {}

		for _,itemId in pairs(inventory) do
			--You could optimize this code, but this is just to make the example easier.
			for key,value in pairs(Items) do
				if value == itemId then --The value in the serialized inventory matches the value in Items.
					table.insert(deserialized, key) --Add the string version to our deserialization.
				end
			end
		end

		return deserialized
	end)

	inventoryStore:BeforeSave(function(inventory)
		--BeforeSave is called with the DESERIALIZED inventory. Whatever this returns is what will be saved.
		--The "inventory" parameter looks something like {"Item1", "Item2", "Item3"}.
		--We're going to save this as {1, 2, 3} as a storage optimization.
		local serialized = {}

		for _,itemName in pairs(inventory) do
			table.insert(serialized, Items[itemName]) --"Item1" -> 1, etc
		end

		return serialized
	end)

	inventoryStore:Get({})
end)

Backups

Data stores will fail through the fault of Roblox and there’s nothing you’ll be able to do about it. DataStore2 now has something to deal with this. You can now use the backup APIs to limit the number of times DataStore2 will try to get the result from the data store. By default, DataStore2 will continue retrying indefinitely. If, however, you set the max number of tries DataStore2 can retry with the :SetBackup API (and DataStore2 reaches that limit), the data store will be marked as a “backup” data store, and :Save() will not truly save, and it won’t call :BeforeSave or :AfterSave.

void DataStore2:SetBackup(retries, value=nil)

Example usage: coinStore:SetBackup(3)

Will set the maximum times DataStore2 will try to retry :GetAsync() when getting values. If it reaches the maximum amount of retries, the data store will be marked as a backup data store, and will set the value of the data store to the value provided. You can choose to omit the value parameter, and whatever default value you used for :Get() will be used instead.

void DataStore2:ClearBackup()

Example usage: coinStore:ClearBackup()

Will unmark the data store as a backup, clear the cached value, and the next time you use :Get(), it will attempt to use GetAsync() again. Usually unnecessary.

bool DataStore2:IsBackup()

Example usage: coinStore:IsBackup()

Returns whether or not the current data store is a backup store.

Gotchas

DataStore2 has some gotchas to users used to the current data store model.

  1. DataStore2 does not save in studio unless explicitly told to.

This was added because with multiple usages of DataStore2 (i.e. for several different keys), the game would take way too long to finish up because of all the BindToCloses. Thus, DataStore2 will not save in studio unless it is told to. For DataStore2 to save in Studio, you must create a BoolValue named “SaveInStudio”, and check it to true. There will be a warning when using DataStore2 if you do not have this object, and a separate warning (to save you from yourself) to tell you when a data store isn’t saved because the option is off.

  1. Set your data stores as they change, not on PlayerRemoving.

Your PlayerRemovings shouldn’t involve DataStore2 at all. DataStore2 will automatically save your player’s data and ensure it saves in the 30 seconds BindToClose provides it. Set the data as it changes (i.e. when a player gets coins, set their coins). This is the whole point of DataStore2.

Please do not hesitate to ask if you have any concerns or questions.


Need help finding how data losses occur with working datastore code
#2

Awesome tutorial. Datastore2 is unequivocally going to help some people, myself included. I’ll definitely try it out myself in some upcoming projects of mine. I also find fascinating that an idea from @berezaa is started to develop into further ideas from other people and thus, I can’t wait to see what other good ideas that come from it and this module. :smiley:


#3

For our new members out there, this is the method berezaa uses:


#4

I updated DataStore2 with a minor typo fix (you’d have to make a value named SaveOnStudio instead of SaveInStudio). This reminds me to inform you that if you’re not using the require ID method to please “Watch” my repository to know when the module updates.


#5

Actually made me tear up a bit. Glad to have been able to help :blush:


#6

Added a “Gotchas” section for common mistakes you may come across when using DataStore2.


#7

This. :revolving_hearts:
Is. :two_hearts:
Beautiful. :sparkling_heart:

EDIT: In the future, making this so that you don’t have to have a player instance, but just a datastore key would also be incredibly helpful!


#8

The problem with just a data store key is that DataStore2 only saves on leave, so it’s not guaranteed the data you’re trying to get will be the same. What’s the use case for non-player instances? I can add it if I feel the use case is valid.


#9

I’m definitely aware that there aren’t as many uses for a regular dafastore key, however, some games may use a global weather systems, notifications, etc. However, with DataStore2, would these systems really require this complexity… :thinking:

Either way, amazing system. Just integrated it into my game and the speed is definitely noticeable!


#10

I think for the use cases you provided the benefits of DataStore2 would not be applicable or noticable, you’re much better suited using standard data stores or HttpService.

Glad to hear DataStore2 is working for you, can you message me a link to your game so I can use it as a success story? :slight_smile:


#11

:AfterSave was added a little bit ago. In order to fix a minor memory leak, I am going to inform all users to remember the rule of not using DataStore2 in PlayerRemoving. This will now be enforced when this memory leak is fixed. Move everything you might put in PlayerRemoving for DataStore2 in AfterSave callbacks.


#12

Suggested by yours truly


#13

Added a backups section and three new APIs for backups in the case of Roblox outages.


#14

Can someone please make a video tutorial? I really didn’t understand much from reading. (English is not my first language, I am sure the tutorial is perfect)


#15

That would help me too, I haven’t really used a Datastore 2 before :wink:


#16

I can look into it @Aorda @Beartikal.


#17

An issue with BeforeInitialGet has been spotted to where it is called on every :Get(), rather than just the first time. If you are currently using BeforeInitialGet, be prepared for this to change.