Massive but temporary ping spikes in game with high rate of part generation

Hi there! I’ve been experiencing an issue for a while (since around 2023?) which seems to plague just about every instance of a genre of game I’ve been working on for the last few years.

In short, the games are “mining” focused - it relies on communication from the client (via the mouse) to “mine” blocks and generate new ones in its vicinity. Naturally, and especially in certain specific circumstances, this can cause severe stress on the server when a very large number of these events are occurring per second. However, it is necessary that this update is immediately visible to the client.

In more “tame” cases, this may cause the server to pause for a brief moment before it resumes and the player is able to continue as normal. Occasionally, though, ping-related issues can become extremely detrimental to the experience:

  • In the first case, the client is mostly uninvolved. A block is mined and a cave is generated, which usually contains many more blocks than would normally be found by the player’s direct interactions. Sometimes, caves are so large that they seemingly overload the server and ping spikes massively to numbers as high as 30k (meaning the server will not respond for thirty seconds after generation). Through a lot of trial and error I’ve found a few routes of optimization for this, but it does still happen occasionally for very large caves, which can be frustrating at times as the player must wait this out (and do nothing, usually) until they can mine again.
  • In the second case, the amount of blocks mined per second slowly increases as part of a “tactic” that some players of the game use. I’d like this tactic to be perfectly viable, but due to its continuously high mining rate, this seems to overload the server as well - ping suddenly spikes from its resting point to numbers as high as 5000ms, severely slowing down the client’s ability to interact with the server before dropping back down a handful of seconds after the player releases the mouse and stops mining. This seems to happen even if the received data is nothing too concerning (at the time of this screenshot, it was around 25kbps?)
    image

As I said before, I’ve found a few solutions for the first problem even if they are imperfect - mainly just general optimizations. No matter how hard I optimize it though, this problem seems to stick around. What’s strange is that the extent of it seems to vary from game-to-game - as a specific example, the game where this code was taken from had to be transferred to another place (as the original owner had disabled their account) and the issues became far more severe. I’m not sure if there’s a priority system in place as to how much resource usage Roblox allocates to specific games and if that changed in some way when it was transferred.

In any case, I primarily have two questions - why does this happen in the first place, and what can I do to mitigate it? It hasn’t been the case forever - I noticed it in another game back in 2023 that hadn’t suffered from the same issues previously, and I noticed that it began to affect my own games shortly afterwards. I haven’t seen any posts that mention this issue, at least not directly, so any insight or vague ideas would be extremely helpful. Thanks!!

Here’s a simplified version of a server-sided function that fires whenever the client sends a request through a RemoteEvent to the server:

local function mineOre(plr,obj,abil)
	if obj and not obj:GetAttribute('mining') and not (obj:GetAttribute('placed') and obj:GetAttribute('placed') ~= plr.Name) and not obj.Locked and not (abil and obj:FindFirstChild('Azure')) then
		obj:SetAttribute('mining',true)
		local newVectors = vectors
		if convert(obj.Position.Y) < -4500 and convert(obj.Position.Y) > -5000 then -- rip.
			newVectors = interVectors
		end
		for k,v in pairs(newVectors) do
			local newPos = obj.Position + v
			if convert(newPos.Y) <= 0 then
				generateOre(getOre(newPos):Clone(),newPos)
			end
		end
		_G.take[getConvertedPosition(obj.Position)] = nil
		obj.Parent = nil
		blooker -= 1
	end
end

Here’s another simplified function, in the same script, which is called in the snippet above and handles inserting the newly generated blocks into Workspace:

function generateOre(ore,pos,override,cave)
	if ore and ((not take[getConvertedPosition(pos)]) or override) then
		take[getConvertedPosition(pos)] = true
		_G.take[getConvertedPosition(pos)] = true
		if getCaveNoise(pos) > 0.43 and getCaveNoise(pos) < 0.74 and convert(pos.Y) < 0 and not isPositionNoCave(convert(pos.Y)) then
			local barrier = ore:Clone(); barrier.Parent = workspace
			barrier.Color = Color3.fromRGB(144,144,144); barrier.Material = 'Slate'; barrier.Position = pos
			local ct = getRandomCaveType()
			if convert(pos.Y) <= -4500 and convert(pos.Y) > -5000 then if math.random(1,3) ~= 1 then ct = '' end end
			local cave = caves[initializeCave(ct)]
			cave.caveVectors[1] = pos
			generateCave(cave)
			cave.parent.Parent = workspace.MineFolders
			barrier:Destroy()
		else
			ore.Parent = (cave and cave.parent) or workspace.Mine
			ore.Position = pos; blooker += 1
		end
	end
end
2 Likes

From what I can tell, it seems like your server is just extremely overloaded. I cannot give you an accurate source of the issue without viewing all the scripts.

The only reasonable suggestion I can give you is to ditch the generation on the server all together. You’d probably have to rework most of your systems, but I am very positive that it would fix the issue. Use the server only for storing data, generating seeds and forwarding information between all the clients, but do not generate the terrain/parts themselves on it. Instead just save it to a table (or any other form that suits you) and send it to the clients and make them render/generate the actual instances.

If you’re worried about this worsening the performance for the clients, then be not, as the client is already doing most the work anyways, just behind the scenes as the server handles the replication for you. However it will substantially decrease the amount of work the server has to do, as it basically just works as a data provider and replicator instead of simulating it all as well.

I reworked the system to handle actual instance generation on the client - while this does fix the first issue, the second one still remains (although it takes significantly more to overload the server than it did before). The actual rate of “mining” is limited to about ~60 blocks per second, and while the game can initially handle this fine, its limit decreases as more instances are created - around the 30k mark (about twice as much as it was previously), the same rapid ping increase problems resurface.

I’m assuming the problem is still caused by the server being overloaded due to the extremely large amount of data being requested. I’ve done about all I could think of to optimize the server portion, though. If it helps, here are the new versions of the scripts I sent before:

local function mineOre(plr,obj,packet,abil)
	if typeof(obj) == 'Instance' then
		if obj and not obj:GetAttribute("mining") and not (obj:GetAttribute('placed') and obj:GetAttribute('placed') ~= plr.Name)  then
			obj:SetAttribute('mining',true)
			if obj:GetAttribute('placed') then
				placedOres[obj:GetAttribute('placed')] = false
			end
			obj.Parent = nil	
		end
	elseif obj and GC.oreInfo[obj.Name] and not obj.Mining and not obj.Locked and checkConst(obj) then
		obj.Mining = true
		packet = packet or {}
		local newVectors = vectors
		if convert(obj.Position.Y) < -4500 and convert(obj.Position.Y) > -5000 then -- rip.
			newVectors = interVectors
		end
		for k,v in pairs(newVectors) do
			local newPos = obj.Position + v
			if convert(newPos.Y) <= 0 then
				table.insert(packet,generateOre(getOre(newPos),newPos,nil,nil,obj.Position))
			end
		end
		local tier, rarity = GC.oreInfo[obj.Name].Tier, GC.oreInfo[obj.Name].Rarity
		_G.take[getConvertedPosition(obj.Position)] = nil
		blooker -= 1
		--[[if not abil then
			pickMod.pickAbility(plr)
		end]]
		local del = table.clone(obj)
		if plr:FindFirstChild('Ores') and (repstorage.Poindexter.Value ~= true) then
			if plr.Ores:FindFirstChild(obj.Name) then
				local vinque = 'Normal'
				local vinguevalue = 1
				local main = repstorage.Ores
				
				if obj.Type ~= 'Normal' then
					(plr:FindFirstChild(obj.Type) or plr.Ores)[obj.Name].Value += 1
					vinque = obj.Type
					vinguevalue = (10*(GC.oreInfo[obj.Name].Value+5))/2
				else
					plr.Ores[obj.Name].Value += 1
				end
				if tier then
					if tier < 4 or (vinque ~= 'Normal' and tier < 5) then
						if tier < 2 or (vinque == 'Altered' and tier < 4) or ((vinque == 'Big' or vinque == 'Rainbow') and tier < 5) then
							task.spawn(sendWebhook,plr,obj,vinque)
						end
						sendMessage(plr,obj,false,vinque)
					end
					task.delay(0.01,function()
						local layer = init.getLayerPerPosition(convert(del.Position.Y))
						local mult = 1
						if layer == 'Obsidian' then mult = 3 elseif layer == 'Interblock' then mult = 40 end
						plr.Levels[layer].XP.Value += (((rarity/math.ceil((6-tier)/2))/2*(vinguevalue))*mult)
						local oldLevel = plr.Levels[layer].Level.Value
						xpEvent:FireClient(plr,(((rarity/(6-tier)/2)/2*(vinguevalue))*mult)*2,GC.ores[del.Name].Color)
						if plr.Levels[layer].XP.Value >= repstorage.Levels:FindFirstChild(tostring(oldLevel)).XP.Value then
							plr.Levels[layer].Level.Value += 1
							plr.Levels[layer].XP.Value -= repstorage.Levels:FindFirstChild(tostring(oldLevel)).XP.Value
							xpEvent:FireClient(plr,oldLevel+1,nil,true)
						end
						del = nil
					end)
				end
			end
		end
		return true, packet
	end
	return false
end

game.ServerStorage.ServerMineEvent.Event:Connect(mineOre)

MineOre.OnServerInvoke = function(player, packet)
	local newPacket = {
		['Gen'] = {}
	}
	for k,v in pairs(packet) do
		local success, _ = mineOre(player, v, newPacket.Gen)
		newPacket[v.ID] = success
	end
	return newPacket
end
function generateOre(ore,pos,override,cave,originalPosition)
	if ore and ((not OrePositions[getConvertedPosition(pos)]) or override) then
		OrePositions[getConvertedPosition(pos)] = true
		_G.take[getConvertedPosition(pos)] = true
		ore = initializeOre(ore)
		if not cave and not override and getCaveNoise(pos) > 0.43 and getCaveNoise(pos) < 0.74 and convert(pos.Y) < 0 and not isPositionNoCave(convert(pos.Y)) then
			local ct = getRandomCaveType()
			if convert(pos.Y) <= -4500 and convert(pos.Y) > -5000 then if math.random(1,3) ~= 1 then ct = '' end end
			local cave = caves[initializeCave(ct)]
			cave.caveVectors[1] = pos
			cave.originalPosition = originalPosition or pos
			generator:FireAllClients(cave,"Barrier")
			generateCave(cave)
			generator:FireAllClients(cave,"Cave")
		else
			ore.Position = pos
			blooker += 1
			if GC.oreInfo[ore.Name].Tier and GC.oreInfo[ore.Name].Message then
				sendMessage(false,ore,true)
			end
			ore.Type = GC.makeAlt(ore,nil,true)
			if cave then
				table.insert(cave.ores,ore)
			end
			return ore
		end
	end
	return nil
end

Along with the client-sided script that deals with mining: (packets will be sent once every 1/60s if they contain any data)

local nextPacket, packetTime = {}, os.clock()

function minePacket()
	task.spawn(function()
		local packet = MineOre:InvokeServer(nextPacket)
		for k,v in pairs(packet) do
			if k == 'Gen' then
				for _,j in pairs(v) do
					ClientGenerator.Generate(j)
				end
			else
				if v then ClientGenerator.LocalObjects[tonumber(k)].Reference:Destroy() end
			end
		end
		packet = nil
	end)
	packetTime = os.clock()
	table.clear(nextPacket)
end

function mine(obj)
	if obj and db and giggleCheck(obj) then
		if obj:GetAttribute('ID') and not obj:GetAttribute('mined') then
			if obj:FindFirstChild('Azure') and obj.Parent ~= workspace.PlacedOres then -- yikes.
				local tr = obj.MiningTime.Value
				local vcolor = obj:WaitForChild('VColor',0.1)
				if vcolor then
					progress.BackgroundColor3 = vcolor.Value
					tm.Visible = true
					task.wait()
					for i=1, math.ceil(tr/0.05), 1 do
						if (not down) or (not equipped) or (not obj.Parent) or mouse.Target ~= obj then
							tm.Visible = false; return
						end
						progress.Size = UDim2.new(0,((190/math.ceil(tr/0.05))*(i-1)),0,15)
						task.wait(0.05)
					end
					tm.Visible = false
					box.Adornee = nil
				end
			end
			obj:SetAttribute('mined',true)
			db = false	
			table.insert(nextPacket, ClientGenerator.LocalObjects[obj:GetAttribute("ID")])
		end
	end
	db = true
end