Restricting datastores for leaderboards

Hello and thank you for reading.

I have about 6 datastores that I am currently using and I might want to expand that in the future.
I also have 6 billboard style leaderboards to display these values and the top players of each in the lobby.
The problem is that when I added a 6th leaderboard for rebirths, I started getting a warning about datastores reaching the maximum limit. I have been reading through the code on both, watched some videos and read a few posts on here, but I can’t seem to figure out how to restrict datastore requests by:

Combining all values into a datastore table or dictionary
or
Combining the 6 scripts for the leaderboards into one for 1/6th the requests.
or
Make the leaderboards update less frequently

I have a very limited understanding of programming datastores so please try to dumb it down if you can see what I am missing here.

Here is an example of that code that I wrote following an Alvinblox tutorial to get and set and save the datastores (I also have it save somewhere on level up):

local replicatedStorage = game:GetService("ReplicatedStorage")

local currencyName = "Points"
local DataStore1 = game:GetService("DataStoreService"):GetDataStore("PointsDataStore")
local deathsName = "Deaths"
local DataStore2 = game:GetService("DataStoreService"):GetDataStore("DeathsDataStore")

--etc. etc. the names have been changed, idk if it is a security thing or not

        local ID1 = currencyName.."-"..player.UserId
	local ID2 = deathsName.."-"..player.UserId

        local saveData1 = nil
	local saveData2 = nil
	pcall(function()
		savedData1 = DataStore1:GetAsync(ID1)
		savedData2 = DataStore2:GetAsync(ID2)
	end)
if savedData1 ~= nil and data ~= nil then
		points.Value = savedData1
		deaths.Value = savedData2
                print("Data loaded")
else
		-- New Player
		currency.Value = 0
		deaths.Value = 0
print("New player to the game")

end

game.Players.PlayerRemoving:Connect(function(player)

	local ID1 = currencyName.."-"..player.UserId
	DataStore1:SetAsync(ID1,player.leaderstats[currencyName].Value)
	local ID2 = deathsName.."-"..player.UserId
	DataStore2:SetAsync(ID2,player.leaderstats[deathsName].Value)
end)

game:BindToClose(function()
	--when game is ready to shutdown
	for i, player in pairs(game.Players:GetPlayers()) do
		if player then
			player:Kick("This game is shutting down")
		end
	end	
	wait(5)
end)

and here is an example of one of the leaderboards which is a free asset that I edited:

local sg = script.Parent --Surface GUI
local sample = script:WaitForChild("Sample") --Our Sample frame
local sf = sg:WaitForChild("ScrollingFrame") --The scrolling frame
local ui = sf:WaitForChild("UI") --The UI list layout

local dataStoreService = game:GetService("DataStoreService")
--The data store service
local DataStore2 = dataStoreService:GetOrderedDataStore("Deaths")
--Get the data store with key "Leaderboard"
wait(10)
while true do
	for i,plr in pairs(game.Players:GetChildren()) do--Loop through players
		if plr.UserId>0 then--Prevent errors
			local w = plr.leaderstats.Deaths.Value --Get point balance
			if w then
				pcall(function()
				--Wrap in a pcall so if Roblox is down, it won't error and break.
					DataStore2:UpdateAsync(plr.UserId,function(oldVal)
				        --Set new value
						return tonumber(w)
					end)
				end)
			end
		end
		wait(1)
	end    
	local smallestFirst = false--false = 2 before 1, true = 1 before 2
    local numberToShow = 10--Any number between 1-100, how many will be shown
    local minValue = 1--Any numbers lower than this will be excluded
    local maxValue = 10e30--(10^30), any numbers higher than this will be excluded
    local pages = DataStore2:GetSortedAsync(smallestFirst, numberToShow, minValue, maxValue)
    --Get data
    local top = pages:GetCurrentPage()--Get the first page
	local data = {}--Store new data
	for a,b in ipairs(top) do--Loop through data
		local userid = b.key--User id
		local points = b.value--Points
		local username = "[Failed To Load]"--If it fails, we let them know
		local s,e = pcall(function()
		 username = game.Players:GetNameFromUserIdAsync(userid)--Get username
		end)
		if not s then--Something went wrong
		   warn("Error getting name for "..userid..". Error: "..e)
		end
		local image = game.Players:GetUserThumbnailAsync(userid, Enum.ThumbnailType.HeadShot, Enum.ThumbnailSize.Size150x150)
		--Make a image of them
		table.insert(data,{username,points,image})--Put new data in new table
	end
	ui.Parent = script
	sf:ClearAllChildren()--Remove old frames
	ui.Parent = sf
	for number,d in pairs(data) do--Loop through our new data
		local name = d[1]
		local val = d[2]
		local image = d[3]
		local color = Color3.new(1,1,1)--Default color
		if number == 1 then
			color = Color3.new(1,1,0)--1st place color
		elseif number == 2 then
			color = Color3.new(0.9,0.9,0.9)--2nd place color
		elseif number == 3 then
			color = Color3.fromRGB(166, 112, 0)--3rd place color
		end
		local new = sample:Clone()--Make a clone of the sample frame
		new.Name = number--Set name to the number, so UI list layout organizes it
		new.Image.Image = image--Set the image
		new.Image.Place.Text = number--Set the place
		new.Image.Place.TextColor3 = color--Set the place color (Gold = 1st)
		new.PName.Text = name--Set the username
		new.Value.Text = val--Set the amount of points
		new.Value.TextColor3 = color--Set the place color (Gold = 1st)
		new.PName.TextColor3 = color--Set the place color (Gold = 1st)
		new.Parent = sf--Parent to scrolling frame
	end
	wait()
	sf.CanvasSize = UDim2.new(0,0,0,ui.AbsoluteContentSize.Y)
	--Give enough room for the frames to sit in
	wait(120)
end

Thank you again in advance for your help.
I am afraid that I will break everything if I just jump in there and mess with things like how I figure out everything else.

Ah, the spaghetti code. Why did you decide to use a loop to save all player stats synchronously? You should use Players.PlayerAdded which then begins the saving loop for each player. Make sure the loop is long enough.

Players.PlayerRemoving should do the thing about saving if they leave.

So many data stores, why though? The maximum limit is 260k characters. Someone told me the chance is kinda slim to hit that limit, unless you are using ineffective saving methods.


Rewriting code for readability; not after best solution:

-- 2 --
local DataStoreService = game:GetService("DataStoreService")
local Players = game:GetService("Players")

local SurfaceGui = script.Parent
local Sample = script:WaitForChild("Sample")
local ScrollingFrame = SurfaceGui:WaitForChild("ScrollingFrame")
local UILayout = ScrollingFrame:WaitForChild("UI")

local DataStore2 = DataStoreService:GetOrderedDataStore("Deaths")
local ColorIndex = {
	[1] = Color3.new(1,1,0);
	[2] = Color3.new(0.9,0.9,0.9);
	[3] = Color3.fromRGB(166, 112, 0);
}
 
--Get the data store with key "Leaderboard"
wait(10)
while true do
	for i, player in pairs(Players:GetPlayers()) do -- Canonical GetPlayers()
		local deathCount = player.leaderstats.Deaths.Value --Get point balance
		if deathCount then
			pcall(function() -- Wrap in a pcall so if Roblox is down, it won't error and break.
				DataStore2:UpdateAsync(player.UserId, function(oldValue)
					return tonumber(deathCount)
				end)
			end)
		end
	end
	
	local smallestFirst = false -- rising order or descending
    local numberToShow = 10 --Any number between 1-100, how many will be shown
    local minValue, maxValue = 1, 10e30 -- why not math.clamp()????
    local pages = DataStore2:GetSortedAsync(smallestFirst, numberToShow, minValue, maxValue)
    
	--Get data
    local top = pages:GetCurrentPage()--Get the first page
	local data = {}--Store new data
	for _, b in ipairs(top) do --Loop through data
		local userid = b.key --User id
		local points = b.value --Points
		local username = "[Failed To Load]"--If it fails, we let them know
		local success, errorMessage = pcall(function()
			username = Players:GetNameFromUserIdAsync(userid) -- Get username
		end)
		
		if not success then
		   warn("Error getting name for "..userid..". Error: "..e)
		end
		
		local image = Players:GetUserThumbnailAsync(userid, Enum.ThumbnailType.HeadShot, Enum.ThumbnailSize.Size150x150)
		table.insert(data,{username,points,image}) -- Put new data in new table
	end
	
	UILayout.Parent = script
	ScrollingFrame:ClearAllChildren() -- Remove old frames
	UILayout.Parent = ScrollingFrame
	
	for index, entry in pairs(data) do --Loop through our new data
		local name, value, image = entry[1], entry[2], entry[3]
		local color = ColorIndex[index] or Color3.new(1,1,1)
		
		local new = Sample:Clone()--Make a clone of the sample frame
		new.Name = number --Set name to the number, so UI list layout organizes it
		new.Image.Image = image --Set the image
		new.Image.Place.Text = number --Set the place
		new.Image.Place.TextColor3 = color --Set the place color (Gold = 1st)
		new.PName.Text = name --Set the username
		new.Value.Text = value --Set the amount of points
		new.Value.TextColor3 = color --Set the place color (Gold = 1st)
		new.PName.TextColor3 = color --Set the place color (Gold = 1st)
		new.Parent = ScrollingFrame --Parent to scrolling frame
	end

	wait()
	ScrollingFrame.CanvasSize = UDim2.new(0,0,0, UILayout.AbsoluteContentSize.Y)
	-- Give enough room for the frames to sit in
	wait(120) -- lower frequency here
end

I can’t access the wiki at the moment. Make sure to do the math about the limitations.

Edit: Replied to wrong person. Oopsies!

Here is the link with the number of datastore requests per minute allowed and some common error codes: Documentation - Roblox Creator Hub

Okay, so as mentioned above datastores have very large limits so you’re unlikely to ever store too much data within one datastore. They do, however, have limits to how many times you can set/get things in a minute so it’s a good idea to minimize these actions.

I’ll try and do both sections of code separately and make it as easy as possible to understand.

First section: The first thing that stands out to me is that you’re using 2 datastores to store peices of data that don’t need be ordered. Generally you should try and cut down the amount of datastores you are using as you will quickly reach the limit. A good example here is to begin to use tables to store multiple pieces of data instead of 2 seperate datastores.

-- saving
local ID1 = player.UserId
-- save as table {currencyName, deathsName}
DataStore:SetAsync(ID1,{player.leaderstats[currencyName].Value, player.leaderstats[deathsName].Value})


-- getting data
local ID1 = player.UserId
local data = DataStore:GetAsync(ID1)
-- data should look like this:
-- {currencyName, deathsName}
local currency = data[1]
local deaths = data[2]

This way you only need go call getAsync/setAsync once, compared to the previous 2 times.

Second section: Okay, let’s be real here - 6 datastores is a lot. So you can’t be calling these willy-nilly all the time or you’ll soon reach that dreaded limit! In this scenario there isn’t really a way to use less datastores.

But could you call them less? Definitely. In your single leaderboard script you save the deaths for all players and get the highest deaths every 2 minutes, which is causing problems. My first suggestion is to only get and save data when the player joins and leaves the server. If you want maybe have an auto save script that just saves data every 10 minutes as a backup plan, but right now BY FAR the most taxing part of your code is at the start of the while loop where it saves each players currency. Imagine doing this for every player for 6 leaderboards in the space of like 5 seconds!

You need to consider for leaderboards does it matter if they are the most up to date at all times? I would argue that it depends on the amount of players - with less players a slower leaderboard will be really obvious and annoying, but as more people play your game the less likely it is that the values on the leaderboard are constantly changing.

I would recommend getting and saving data at a slower interval such as 10 minutes, and/or only when the player joins and leaves. I would also update the leaderboards slower (once every 5 mins) but spread them across 5 minutes (so each minute the next leaderboard updates).

You seem to have decent knowledge and the ideas you gave of restricting requests seem good. Have a go with some different tactics and see what helps the most! :slight_smile:

Thanks for the help.
I forgot to include that the first script was inside a function.

game.Players.PlayerAdded:Connect(function(player)

I think I am going to slow and offset updating the leaderboard like you said.
That should space out the requests and after reading up on it again as suggested, I think it is running out faster because I am the only player in the server.

When I need to change the datastores again, I will try to put them all in the table now that I know where to start.

Thanks again.