Issue with lag / player timeout when loading models from datastore

I have a game where players can build on the map, and the data saves persistently. No, this is not a plot-based / tycoon / freebuild style server, but the entire map is persistent.

I have an issue where clients are now timing out and disconnecting because I’m trying to load in too many models when the server starts. Using wait() or task.wait() doesn’t help the situation and still causes timeouts.

What’s a good way I could manage the loading of these models in a timely (but working) manner? As of right now, I have around 1783 models to load, and it’s causing persistent connection issues when these buildings are loading in.

If I don’t yield in some way, I get a script execution error because the script times out.

Thoughts?

2 Likes

Can you show some code and images of what the map looks like? How complex models are?

If it’s a large area, streaming could help massively.

Optimizing the models could also help.

The models aren’t overly complex. I think the most complex one I have is about 20-25 parts. Most are around 1-10 though.

The map is very large; 6k x 6k roughly. I don’t have studio open right now but I believe Streaming Enabled is turned on, based on how the map loads around you.

I do not have studio open right now, but I can provide some screenshots and code later.

The main issue seems to be that the server itself freezes up. This has nothing to do with the client loading the models - if a player joins and there’s already a thousand models on the map, they can load in just fine. The issue is loading all of these models when the server initially starts. It seems like the server just completely freezes up and everyone times out.

Server freeze/crash is possible if you have a very high network throughput. You can check the graphs in console and ‘server stats’ if I remember the name correctly.

Is there a way to check that on the website too? There’s a lot of graphs in my experience analytics that just don’t show up for whatever reason that might give me a good idea of what’s going on.

Or is it just in the F9 menu ingame?

F9 in game.
You do need edit permissions to be able to see the server stats though.
I’m not sure what the tab is called but something like ‘server stats’. It’s like the middle tab.

I found it, I think.

During the model loading process (~1 model every 0.2 seconds) it results in a value of up to 11k. When it stops loading as many models, it returns to single-digit values.

Not sure what this means or how to remedy it.

I just experienced another crash, and the only high value was the total data, which spiked up to 9.5k randomly.

Yes, it’s most likely that.
Had a similar issue for our game, and for us it was caused by inefficient code that sent way too much data to everyone. Fixed it by only replicating data to players that actually need it.

Try checking scripts that might have inefficient replication. If you’re sure it’s the model loading, then probably your best bet is to slow down the loading. Use longer waits.

The hole in the graph isn’t a good sign. I think it indicates the server being unresponsive/overwhelmed with the amount of replication data it has to send. Would be handy if roblox provided a baseline, but I couldn’t find it.

Just for reference if it’s of any use; for our game it’s usually below 1k, with occasional peaks that are >10k (pretty rare).

Do note that the server throughput will most likely get worse with each player that is added to the server! So try testing with full servers to be sure it won’t crash.

So, I’ve narrowed it down a little bit.

I don’t think it is actually related to models loading. I do think that was an issue, however, now the server is randomly crashing a few minutes after the models are fully loaded. Now, it seems to be crashing whenever someone joins the game, which is a perplexing issue.

If you’re referring to that 2nd graph, that is what happens when the server crashes when someone joins. There’s just a big random spike in data usage for some reason. I have disabled every script that runs when a player joins, so I don’t know what is going on.

Debugging it is a BIG pain. Roblox doesn’t provide us much tooling to debug what causes the spikes.

I can highly recommend this for debugging your remotes: Packet Profiler - accurately measure remote packet bandwidth!

It’s unlikely that remotes are causing issues though (unless you’re sending loads of data every frame).

What was the biggest issue for our game was attributes. They’re nice and handy, and I used them to store some properties on each character. These properties were updated every frame. This is good, but the issue is that roblox automatically replicates those attribute changes to everyone. This causes massive bandwidth usage for replication.

I suggest thinking about any (server-sided) script that runs very often (every frame or so), and debugging them. It’s trial and error, so not fun…

Fortunately, the attribute thing isn’t the case for me. I don’t use many and they don’t update except once. Aside from that, I have no scripts that constantly run and fire. I think the only exception is a script that runs every 4 seconds, but that is clientsided, and shouldn’t be causing server issues.

Just adding another graph because I tested it again after disabling nearly every last script in the whole game, and it produced the same result:

I will try out the packet profiler, it looks like it will be super useful.

The packet profiler doesn’t seem to work with the in-game UI module. Clientsided stuff just shows blanks. I can see it in the Server window when i run a local server in Studio, but that isn’t giving me the giant data spikes and I can’t simulate a player joining.

if you disabled all scripts, you won’t have remotes either way. What scripts do you still have enabled? Only the loading one?

Make sure there’s no ‘surprise’ script in workspace. Run a GetDescendants loop that prints scripts in console to check this.

I did that. Nothing out of the ordinary.

I ended up reenabling all of the scripts and now Players can join fine, but now I’m experiencing high ping and lag dealing with another aspect of the game that also causes server crashes… an aspect of the game that was working fine a couple days ago.

Alright, probably best to slowly work your way through them then. Optimization is difficult, especially for realistic games (which your game seems to be).
Good luck.

Hey did you manage to fix your issue? If so what was the solution