Don't know how to optimize data requests (Making a procedural game)

Hello I’m Ryxku and I’ve been working on a project for 2 months. I’m nearly close to the end of my project right now but there’s a big flaw I didn’t anticipate at the beginning of my project are the data requests that each player need every minute.
I often see datastore requests getting added in a queue and when it happens, the player see empty plains instead of seeing houses.
Here how I organize my datas :
I have a store called Neighborhoods and these are my keys :

image

In each key there are other dictionnaries which store specific informations for different players. The good thing of my game is it synchronizes everything in real time.
image

But the real problem now is when several players want to have a free walk in my game… I gave them an option to get teleported to a specific house. That’s why I have a different store called PlayerData where I store informations to know where is located the house of a specific player because all players are limited to 1 house only!

Here is an important function which eats a lot of ressources for the data requests :

-- The function to get all informations of one neighborhood
local function getNeighborhoodHouses(x, y)
	-- Une seule requête pour obtenir toutes les données du quartier
	local neighborhoodData = neighborhoodDataStore:GetAsync("Neighborhood_" .. x .. "_" .. y)

	if not neighborhoodData then return nil end

	-- Request to get informations of all inhabitants in the neighborhood
	local playerIds = {}
	for houseNumber = 1, 9 do
		local houseKey = "HouseNumber" .. houseNumber
		local houseData = neighborhoodData[houseKey]
		if houseData and houseData.owner ~= 0 then
			table.insert(playerIds, houseData.owner)
		end
	end

	-- Recover datas in one request but idk
	local success, playersData = pcall(function()
		local data = {}
		for _, id in ipairs(playerIds) do
			data[id] = playerDataStore:GetAsync("Player_"..id)
		end
		return data
	end)

	if not success then
		warn("Error I failed recovering data from each user")
		return nil
	end

	-- Building the logic
	local houses = {}
	local index = 1

	for houseNumber = 1, 9 do
		local houseKey = "HouseNumber" .. houseNumber
		local houseData = neighborhoodData[houseKey]

		if houseData then
			local playerData = playersData[houseData.owner]
			local success, ownerName = pcall(function()
				return Players:GetNameFromUserIdAsync(houseData.owner)
			end)

			houses[index] = {
				originalID = houseNumber,
				owner = houseData.owner,
				ownerName = success and ownerName or "Unknown Player",
				houseType = houseData.houseType,
				claimDate = houseData.claimDate,
				Texte = houseData.Texte,
				CustomImage = houseData.CustomImage or "0",
				Rotation = houseData.Rotation or "Front",
				forSale = houseData.forSale,
				saleAssetId = houseData.saleAssetId,
				salePrice = houseData.salePrice,
				Badge1 = playerData and playerData.Badge1 or false,
				Badge2 = playerData and playerData.Badge2 or false,
				Badge3 = playerData and playerData.Badge3 or false,
				TotalVisites = houseData.TotalVisites or 0
			}
			index = index + 1
		end
	end

	return houses
end

Each time I tried something different like using GetSortedAsync() I get http error requests in the output. So I’m really stuck, I would be really happy to receive any kind of help! Thank you!

4 Likes

Are you loading all neighborhoods at once when the server has started? Or are you just loading the specific neighborhood the player’s house is in?

EDIT: That is because I feel like all data should just be loaded at once when the server has started and then when the player once to go to a specific house, requests like these won’t be that as expensive.

4 Likes

Loading one neighborhood costs me around 10 GetAsync so no I can’t loading ALL neighborhoods at the beginning but I load based on player’s position like 0;0 . I have a local script which detects the current coordinates and fireservers when the player goes in a different chunk :

local Players = game:GetService(“Players”)
local ReplicatedStorage = game:GetService(“ReplicatedStorage”)
local RunService = game:GetService(“RunService”)

local player = Players.LocalPlayer
local playerGui = player:WaitForChild(“PlayerGui”)
local coordGui = playerGui:WaitForChild(“ScreenGui”)
local nono = coordGui:WaitForChild(“ImageLabel”)
local coordLabel = nono:WaitForChild(“TextLabel”)

local Constants = require(ReplicatedStorage.Constants)

local lastX, lastY = nil, nil
local lastTriggerTime = 0
local debounceDelay = 10 – Délai de débounce en secondes

local function getNeighborhoodCoordinates(position)
local x = math.floor(position.X / Constants.NEIGHBORHOOD_SIZE)
local y = math.floor(position.Z / Constants.NEIGHBORHOOD_SIZE)
return x, y
end

local function updateCoordinates()
local character = player.Character
if character and character:FindFirstChild(“HumanoidRootPart”) then
local position = character.HumanoidRootPart.Position
local currentX, currentY = getNeighborhoodCoordinates(position)

    coordLabel.Text = string.format("(%d;%d)", currentX, currentY)

    if tick() - lastTriggerTime > debounceDelay then
        if currentX ~= lastX or currentY ~= lastY then
            print("We're changing coordinates atm")
            -- Demander au serveur les maisons du nouveau quartier
            ReplicatedStorage.RequestNeighborhoodHouses:FireServer(currentX, currentY)
            lastX, lastY = currentX, currentY
            lastTriggerTime = tick()  -- Updating in real time
        end
    end
end

end

RunService.RenderStepped:Connect(updateCoordinates)

player.CharacterAdded:Connect(function(newCharacter)
newCharacter:WaitForChild(“HumanoidRootPart”)
updateCoordinates()
end)

2 Likes

If a player has reached a neighborhood, record it immediately on a dictionary that stores all neighborhood data, and when another player steps in the same neighborhood, just check if that neighborhood is a part of that dictionary, but if not, proceed with your GetAsync methods. This way, if all neighborhoods, let’s say 10 neighborhoods were discovered initially, you just have to reference them.

That is if you don’t already keep track of neighborhoods already loaded in the game…

2 Likes

But what if one of the houses get updated? Isn’t there a risk of having obsolete data all the time? Also I don’t have dictionaries out of my data stores.
I’m going to try to create a script module that caches the info relayed by the datastore, but can I store everything in one module? I need a place that’s accessible to all players on the server.

2 Likes

Yes, you can store everything in one module. Also, if you are worried about obsolete data, you would have to unfortunately create some code that detects when a player from this server or from another server changes some properties and just make the adjustments manually. By the way, the dictionary is just a container of all neighborhoods loaded.

3 Likes

Okay and will this module be accessible for all players? I mean it will be the same module for everyone or each player has its own module version?

2 Likes

A universally accessible module for all players to use. But something bothers me, isn’t neighborhoodData containing all the data values for each house of a player living in it? Shouldn’t you just put all the values needed in there and reduce data requests to just one?

2 Likes

You’re definitely not supposed to have that many keys. Can’t you directly set the key to the neighborhood, so you only have one GetAsync call to do for it? And if a neighborhood key is still under the 4MB data limit, then you could regroup multiple neighborhoods on a single key.

2 Likes

Would it not make sense to cache all the data you get on the server (maybe unload old data if it gets too big)?

2 Likes

I realize making caches doesn’t make sense at all since the procedural map should be infinite. So the amount of stockable keys would be 9999x9999

2 Likes

In one GetAsync I can’t manage to get children and grandchildren at the same time in a key.

2 Likes

Does the queue problem arise when playing as a single player or in a server with many people?

2 Likes

Both. Also I’m thinking of regrouping several neighborhoods in one, so it would a chunk regrouping several tiles.

2 Likes