DataStore help (Serverlocking and Data-loss prevention with versioning)

[spvp1234 (NVIDIAsThug)]

Hello yall!
I’ve been reading more about datastores and I would love to improve on mine:
Current DataStore for main Data:

local DSS = game:GetService("DataStoreService")
local PS = game:GetService("Players")
local SS = game:GetService("ServerStorage")
local RNS = game:GetService("RunService")

local DataStore = DSS:GetDataStore("DimensionsDataDSS")

local PlayersDataFolder = SS:WaitForChild("PlayersData")

local Dimensions = require(game:GetService("ServerScriptService"):WaitForChild("Server").RollingSystem.RollingModuleServer.DimensionsModule)
--//Functions

if success then
	print("Removed Nickname:", nickname)
end

function SetupPlayerData(Player)
	task.wait(1)
	local UserId = Player.UserId
	local key = "Player_"..UserId
	
	local PlayerDataFolder = PlayersDataFolder[Player.Name]
	local DimensionsData =  PlayerDataFolder.DimensionsData
	
	local rolls = Instance.new("NumberValue")
	rolls.Name = "Rolls"
	rolls.Parent = DimensionsData
	rolls.Value = 0
	
	local dimensionequiped = Instance.new("StringValue")
	dimensionequiped.Name = "DimensionEquiped"
	dimensionequiped.Parent = DimensionsData
	
	local titleequiped = Instance.new("StringValue")
	titleequiped.Name = "TitleEquiped"
	titleequiped.Parent = DimensionsData
	
	local luckfactor = Instance.new("NumberValue")
	luckfactor.Name = "LuckFactor"
	luckfactor.Parent = DimensionsData
	luckfactor.Value = 1

	local DimensionsFolder =  Instance.new("Folder")
	DimensionsFolder.Name = "DimensionsFolder"
	DimensionsFolder.Parent = DimensionsData
	
	for Dimension, Value in pairs(Dimensions) do
		local DimensionTypeValue = Instance.new("NumberValue")
		DimensionTypeValue.Name = Value[1]
		DimensionTypeValue.Parent = DimensionsFolder
		DimensionTypeValue = 0
	end
	
	local Success, ReturnValue
	Success, ReturnValue = pcall(DataStore.GetAsync, DataStore, key)

	if Success then
		if ReturnValue == nil then
			ReturnValue = {}
		end

		
		for Index, Value in pairs(ReturnValue) do
			if Index ~= "Rolls" and Index ~= "DimensionEquiped" and Index ~= "TitleEquiped" and Index ~= "LuckFactor" then
				DimensionsFolder[Index].Value = if ReturnValue[Index] ~= nil then ReturnValue[Index] else 0
			elseif Index == "Rolls" then
				rolls.Value = ReturnValue[Index]
			elseif Index == "DimensionEquiped" then
				dimensionequiped.Value = ReturnValue[Index]
			elseif Index == "TitleEquiped" then
				titleequiped.Value = ReturnValue[Index]
			elseif Index == "LuckFactor" then
				luckfactor.Value = ReturnValue[Index]
			end
		end
	else
		Player:Kick("Your data might have been lost during the retrival process. If you have vaulable assets stored, please contact Support Service! [PlayerDimensionsDataDSS]")
	end
end

function SaveData(Player)
	local UserId = Player.UserId
	local key = "Player_"..UserId
	
	local PlayerDataFolder = PlayersDataFolder[Player.Name]
	local DimensionsData =  PlayerDataFolder.DimensionsData
	
	local Data = {
		Rolls = DimensionsData.Rolls.Value,
		DimensionEquiped = DimensionsData.DimensionEquiped.Value,
		TitleEquiped = DimensionsData.TitleEquiped.Value,
		LuckFactor = DimensionsData.LuckFactor.Value
	}
	
	for i, Dimension in pairs(DimensionsData.DimensionsFolder:GetChildren()) do
		Data[Dimension.Name] = Dimension.Value
	end
	
	local Success, ReturnValue
	Success, ReturnValue = pcall(DataStore.UpdateAsync, DataStore, key, function()
		return Data
	end)

	if Success then
		print("Data successfully saved for "..Player.Name..". [DimensionsDataDSS]")
	else
		warn("Error found with DimensionsDataDSS. Reported User is "..Player.Name..". Please clip and report if seen this! [DimensionsDataDSS]")
	end
end

function OnShutdown()
	if RNS:IsStudio() then
		task.wait(2)
	else
		local finished = Instance.new("BindableEvent")
		local AllPlayers = PS:GetPlayers()
		local LeftPlayers = #AllPlayers

		for _, Player in ipairs(AllPlayers) do
			coroutine.wrap(function()
				SaveData(Player)
				LeftPlayers -= 1
				if LeftPlayers == 0 then
					finished:Fire()
				end
			end)
		end
		finished.Event:Wait()
	end
end

--//Connections
PS.PlayerAdded:Connect(function(Player)
	SetupPlayerData(Player)

	while Player ~= nil do
		SaveData(Player)
		wait(20)
	end
end)

PS.PlayerRemoving:Connect(SaveData)
game:BindToClose(OnShutdown)

Since this datastore really doesn’t support data-loss or server locking for multi-server editing, I really need to add parts to achieve this. How can I add these features to my current datastore?
+I did look at ProfileService and its very reliable, but I want to learn how to create something along the lines of that. I’m not really asking for entire script, but just maybe some idea and parts I can add and understand so I can improve how I implement saving and retrieving data better!

I do want to mainly implement server-locking and make sure that the latest of data with verification is edited. I also want to make sure that players can revert to their last version of data if corrupted or lost.

[MayorGnarwhal (Gnarwhal)]

To implement session locking, add a flag to your data that says it is owned by some other session. Then, if you load data and see that flag, you should wait until that flag is removed (or until it is safe to assume the session lock is dead. I believe ProfileService uses 20 minutes for this threshold).

The best way to do this would be with metadata instead of your actual save data. The function you give UpdateAsync can return up to 3 values, the player data, a table of user id’s, and finally the metadata. You should set this metadata value only if this is a save where the playing is not leaving (either PlayerRemoving or BindToClose). For example, here is a snippet from my datastore module:

-- saving data
	local store = DatastoreService:GetDataStore(DATA_KEY, dataScope)
	local success, err = pcall(function()
		store:UpdateAsync(DATA_KEY, function(currentData, keyInfo)
			local metadata = releaseSessionLock and {} or {
				PlaceId = game.PlaceId,
				JobId = game.JobId,
			}

			local userIds = keyInfo and keyInfo:GetUserIds() or {}
			return data, userIds, metadata
		end)
	end)

On data load, check for this metadata, and if the flag is set, then retry a load some time later:

-- loading data
local store = DatastoreService:GetDataStore(dataKey, dataScope)

local data, keyInfo
success, err = pcall(function()
	data, keyInfo = store:GetAsync(dataKey)
end)

local metadata = keyInfo:GetMetadata()
if metadata and metadata.JobId ~= nil then
	-- session lock is active
	
	local updatedTimeSeconds = keyInfo.UpdatedTime / 1000
	local assumeDead = (os.time() - updatedTimeSeconds >= ASSUME_DEAD_SESSION_LOCK)
	if assumeDead then
		-- load data anyways
		-- I use 20 minutes for ASSUME_DEAD_SESSION_LOCK. I believe this number is from ProfileService as well
	end
	
	if retry > 6 then
		-- load data anyways
		-- player has been waiting for data for quite some time (~2 minutes)
		-- we can assume session lock has been resolved
	end
	
	-- otherwise, wait and retry
end

As for preventing data loss, you should be retrying saves for if and when datastores fail. With session locking, you should have two types of saves: intermediate saves and leaving saves. For saves where the player is leaving, you should keep trying until the save goes through (remember, you’re protected by the session lock now!) For intermediate saves I would have the save retry a couple times before giving up, simply because these saves aren’t as important.

I implement retrying with an exponential backoff, so each subsequent retry will wait a bit longer before saving data so we don’t hit a queue limit while also saving very fast in the off chance that datastores experiences a brief hiccup. For example:

	if retry > 0 then
		task.wait(math.min(2 ^ (retry - 1), 64))
	end

As for loading, you should always retry until data is loaded (or the player leaves). Kicking a player for data failure is bad user experience, and with session locking it will be more common for saves to fail since there are more restrictions to the loading process. Remember to have a loading screen to tell the user that their data is loading!

[spvp1234 (NVIDIAsThug)]

Alright thanks for all this feedback!
I’m a very slow and fairly intermediate scripter and trying to yk learn basic things about Datastores. I know for a fact a lot of developers really hate doing it and just straight up use ProfileService or DataStore2.
I did get some ideas from your snips, though how can I recover versions of that data if you know data fails for a really long time? I want to make sure that next time they join the script automatically handles the data loss/corruption. I did read about some Key functions like grabbing the version and applying that type of data from that key version and then recovering past data.
+I’m very confused in your script what DATA_KEY is, as you are using it to grab the GlobalDataStore from the service. Are you using the player’s key as a name for the datastore? Idk if that would take up too much memory of making individual DS for each player.

edit: Do you think ProfileService is alright and ethical of using because of its heavy testing and approval by developers? I see that a lot of games which blew up such as Blox Fruits heavily rely on this single module script.

[MayorGnarwhal (Gnarwhal)]

For the purposes of basic data saving, using versioning is overkill. I recently found this out the hard way, where my datasatore module used versioning, and each load was 3 API calls instead of just the 1. This worked all fine and good for games with 6-10 players per servers, but it quickly blew up with datastore queue errors when it was used in a game with 20 player servers. The whole point of retrying during saves/loads and session locking is to avoid player data from being corrupted/lost in the first place!

DATA_KEY is just a global key used for indexing datastores. Its a defined constant at the top of my script, and it can be useful for having different datastore entries in studio and development places than the live data simply by appending a short string to the end of the key. dataScope is just the player’s UserId so each player has their own data entry.

Having a different datastore for each player will not run into the issue of running out of memory. My datastore module has been used in projects with 30 million+ unique users without issue.

There’s no shame in using ProfileService or any other open source Datastore module! The whole point of these projects are to provide an easy way to add safe and reliable data saving to games for anyone who wants/needs it. Of course, its always fun and rewarding to create a system yourself, but don’t let anything stop you from using ProfileService if it would help you in your development. Though I made my own Datastore module, I read the ProfileService source code to see best practices and their implementation for my own implementation.

[spvp1234 (NVIDIAsThug)]

Thanks for the feedback, now I def. understand!

Thank you for helping me out!