How to prevent data loss? (i dont want to use profileService!)

Help and Feedback Scripting Support
#1

I have a datastore script, But i don’t know if there will be data loss. can someone help me find out, or just tell me how to make it better in general?

any help is greatly appreciated, Thanks!

local DataStoreService = game:GetService("DataStoreService")
local DataStore = DataStoreService:GetDataStore("DataStore")


local function SaveData(player)
	local ItemsToSave = {}
	
	local s,e = pcall(function()
		for i,Stat in player.leaderstats:GetChildren() do
			
			table.insert(ItemsToSave,{Stat.Name,Stat.Value})
			print(ItemsToSave)
		end
		
		DataStore:SetAsync(player.UserId,ItemsToSave)
	end)
	if s then
		print("Player data Saved ✅")
	end
	if e then
		warn("Player data not Saved ⚠️")
	end
end

local function LoadData(Player)
	local data
	
	local s,e = pcall(function()
		data = DataStore:GetAsync(Player.UserId)
		
		if data then
			for i,v in data do
				local StatName = v[1]
				local StatValue = v[2]
				
				Player.leaderstats:FindFirstChild(StatName).Value = StatValue
			end
		end
	end)
	
	if s then
		print("Player data loaded ✅")
	end
	if e then
		warn("Player data not loaded ⚠️")
	end
end

local function CreateLeaderstats(Player)
	local success,errormessage = pcall(function()
		if not Player:FindFirstChild("leaderstats") then
			local leaderstats = script.leaderstats:Clone()
			leaderstats.Parent = Player
			
			
		end
	end)
	
	if success then
		print("leaderstats created ✅")
	end
	if errormessage then
		warn("Leaderstats could not be created ⚠️")
	end
end

game.Players.PlayerAdded:Connect(function(plr)
	CreateLeaderstats(plr)
	LoadData(plr)
end)

game:BindToClose(function()
	for _,Player in game.Players:GetPlayers() do
		SaveData(Player)
	end
end)

game.Players.PlayerRemoving:Connect(function(player)
	SaveData(player)
end)
1 Like
#2

Use UpdateAsync since it allows to manipulate existing data rather than needing to get and set. You can use it on keys that dont exist as well, it just gives you a nil value the first time round.

If you get an error in fetching the data, then do not, under any circumstance save new data.

1 Like
#3

Am i saving new data here? or are you just warning me about this?

#4

There’s many things you can do to prevent data loss. A lot of it comes down to how you read and write data.

Retry logic

Retry logic is retrying a save when it fails. This means if an internal server error causes one attempt to not work, then it tries again without just giving up entirely. This is good because it can prevent issues where data is lost because of internal errors while saving. You should only retry a certain amount of times, because repeated internal server errors could mean Roblox data stores are down and therefore your code will be looping effectively infinitely.

local attempt = 0

repeat
    local success, err = pcall(--[[saving]])
    attempt += 1
    task.wait()
until
    success or attempt == 3

Don’t use retry logic for data loading, it’s not worth it.

Data budgeting

Each data store request type has a certain budget of requests that can be performed in a given period of time. For example, if you call GetAsync(), it takes one from the GetAsync request budget. SetAsync takes from the SetAsync request budget. If you use UpdateAsync and you change the data associated with the key, it takes one from the read and the write budget. If you exceed that budget, requests will begin to be throttled. This means they will take much longer to be processed. The only real way to stop this is if you have an autosave system, and also make sure you are saving at good intervals.

For example, good times to save are probably:

  • When the player leaves
  • When the player performs a Robux purchase
  • Every 10 minutes, 15 minutes if the request budget is low
Session locking

This is hard to intertwine with data budgeting, but when you do, it’s really effective. What this basically is, is where you attach a piece of data to the player’s data store key while the player is in-game. This is the lock. What you can do on other servers is check to see if a lock exists when you are about to write data, and if it does exist, cancel the write operation. You can also check for a lock when they join and kick them if present. This can stop delayed requests from overwriting newer data, which would be disastrous! Just imagine if a player played on a server for an hour and got really far just to find out their data was replaced with what they had on a server they were on 10 seconds prior to that one.

What’s important is that you account for errors that could occur in your session locking system, such as internal errors. To get around this, you can set a lock timeout, which is basically a time limit after a session has been locked where the lock is ignored for the next request. This can prevent an infinite lock loop where data is never saved.

In summary, ignore a lock IF:

  • The server the write operation is on is the same server the lock was set on
  • Exceeded lock validation limit (e.g. 5 minutes)
Data versioning

Sort of similar to session locking. This is basically where you attach a number to the player’s data and increment it any time data is saved. If the version on the old number is greater than or equal to the number on the new version, you know the new version is out of data and you need to not save it.

BindToClose yielding

Basically, when the game server shuts down, you need to wait for a certain amount of time for data store requests to be processed, otherwise they are just completely ditched, which can cause data to not save if a player was the only one on that server and left.

game:BindToClose(function()
    task.wait(game:GetService("RunService"):IsStudio() and 5 or 30) --30 is the limit
end
Custom data queueing

If you have a high volume of data store requests at one time, they can all be put on to the data store queue which is not very long, causing some requests to “fall off the end” and be lost. If you put in your own queueing system and feed requests to the store one by one, you can avoid this.

UpdateAsync and handling internal errors

(expanding on what @metatablecatmaid said)

  • If data fails to load, don’t save it! You probably have a system to add default data to “new” players, but players who have data that failed to load might be mistaken for new players. This is because, in all the things you save, they will be the default value because the player’s data failed to load. So, when they have their data saved, all their old data will be overwritten with the default values. If data fails to load, kick the player and indicate data loading failed with something like an attribute. Then, when saving, check this attribute and don’t save data if loading failed.

UpdateAsync

  • This is a very powerful tool that links in to almost all of my previous points. Instead of reading and then writing when you need to check data at saving, you can put it all into one request, reducing the points for an internal server error.

There’s a common misconception that UpdateAsync provides a lot more security for your code than it actually does. While it does provide some safety features SetAsync does not, most of the security benefits depends on how you implement it.

  • You can basically check a player’s old data entry against their new data entry when saving data. This is very beneficial for things like data versioning, where you would need to check this. You can easily cancel requests by returning nil.

Take this example:

repeat
    local success, result = pcall(dataStore.UpdateAsync, dataStore, key, function(old)
        if (newData.DataVersion <= old.DataVersion) then
            warn("Attempt to write outdated data was ignored.")
            return nil
        end
        --other checks and actual data logic

        return newData --set new data
    end)
    attempt += 1
until
    success or attempt == 3

This is the most bare-bones implementation of UpdateAsync linking in to data versioning. Since UpdateAsync also returns the new data entry, you can also use it to load data and session lock at the same time (since all you’re doing in the session lock is modifying the lock, not their data).

I know I didn’t put in that many examples, but I hope this helps. Any questions, let me know!

1 Like
#5

you should use ProfileStore

1 Like
#6

Why would you use ProfileStore?

#7

Thanks a bunch for this reply! it really helps a lot! and i will definitely be using these methods in my datastore now.

Thanks again!

1 Like
#8

because

  1. its not ProfileService
  2. its better than ProfileService
#9

i dont want to use profile service/profile Store or whatever its called. i want to make my own scripts so that its more customizable. thanks for the suggestion though

2 Likes
#10

No, I mean in general. What’s the benefit of using it over just using DataStore?

#11

i also had this question

more charssssss

1 Like
#12

I mean, none if you’re good enough to make your own, but it’s already made and has almost anything you’d need for player data

the benefits it provides are listed on its topid

2 Likes
#13

Well, from what I can see it mostly just adds session locks. I don’t see how session locks are useful or necessary. But tbf, I don’t do much with DataStores. Maybe I’m not aware of some edge-case which requires this.
The thing you mostly have to worry about is that the server doesn’t shut down before you save, and that’s about it.

2 Likes
#14

The profile community modules are used a lot because they put in a lot of these things like I mentioned above. A lot of people that use them either don’t know how to put in these features themselves or just can’t be bothered. Since you have more control over your code this way, it could be argued that you could make your data store system even more secure than one of these community ones.

1 Like
#15

In my opinion, the greatest benefit of an open-source data module is the peace of mind from its broad usage. Most of them have been battle-tested across thousands of experiences with proven reliability and scalability. These aren’t trivial systems to create, and there’s a lot of room for error involving data loss, corruption, and throttling.

In the case of ProfileService and ProfileStore, you still have the freedom to write your own data handler; it just comes together with fewer concerns. However, it should be mentioned that ProfileStore is the recent successor to ProfileService, so it doesn’t have the same level of production-ready guarantees yet.

1 Like
#16

I will add something that I basically never hear, but there is a simple way to ensure data cannot be overwritten, aka lost, using UpdateAsync()'s underlying functionality

Often, you’ll see UpdateAsync being used like this (With pcalls please, I didn’t include them in my examples):

local function UpdateBobux(BobuxAmount)
	Datastore:UpdateAsync(Key, function(Data) 
		Data = Data or {}

		Data.Bobux = BobuxAmount
			
		return Data
	end)
end

However, while UpdateAsync ensures two UpdateAsync calls cannot happen at the same time, it doesn’t grantee what the order will be, and your BobuxAmount could be outdated, …

So rather than explicitly set what the BobuxAmount is, you can set how it changed. For example, after a transaction, it could have changed by -10 Bobux

local function UpdateBobux(BobuxDiff)
	Datastore:UpdateAsync(Key, function(Data) 
		Data = Data or {}
		Data.Bobux = Data.Bobux or 0

		Data.Bobux += BobuxDiff
			
		return Data
	end)
end

Now, regardless of the order the data is updated in, data will never be lost. Even if you get outdated data, the worst that can happen is that the amount of Bobux the player has is wrong, he is able to do a transaction he shouldn’t be able to do, and his real BobuxAmount goes into the negative. Next time the server retrieves the real amount, the discrepency will be fixed, and will be shown as negative currency

This means, session locking is not needed

One thing to be wary of is manners exploiters or players in general could use to force your datastore to error, and if you got more than 1 datastore, for example, a currency datastore and a inventory datastore, depending on how your game is setup, the currency datastrore could error, but not the inventory datastore, allowing the player to buy an infinite amount of items

This could be achieved if the player can directly save data to the datastore (for example client side settings for something), and he could intentionally save so much data that he fills the datastore and further requests to update the currency fail because they exceed the limit (not sure how currency going down could take up more space than it already took, but perhaps other fields could be vulnerable)

1 Like
#17

In general, Profile Service removes some very harsh work and provides ready to use tool

Profile service is made in similar way to frameworks or OOP, Those allow it to be scalable and organized, but at cost of memory leaks, advanced algorithms and re-factoring in production stage

If you were to make your own module, you’ll need to spend few days perfecting it, as it’s tool used on most basic level, and there is no other way around it

So if you don’t enjoy profiling for days to only find out that everything worked and mistake was only wrong variable name, it’s better to use profile-service

1 Like