First reported Dec 3, my users are reporting server crashes in one of the places of my game when the player count is low.
The game is a building sandbox, and this place is the most popular server with a lot of builds in it, so it may be memory related.
They report that the crash happens most frequently when only one player is in the server, infrequently with two players, once with three players, and have not reported it with four or more. Also proportional to player count is the place save period: the time between calls of game:SavePlaceAsync(). The timing of the crash makes it appear that the crash is associated with a call of this API.
The SavePlaceAsync API also appears to have become more expensive lately. It may be a continuation of this issue; see: Studio/ingame freezing on save
Here is an example where three players are booted 14 seconds after the SavePlaceAsync call:
Hey there! It looks like your RCCs are hitting the memory limit, and opting to gracefully shut down and disconnect all players instead of crashing. I don’t have any visibility into how your game is designed/implemented, but I do see a pretty high instance count (over 1 million in some RCC instances) - is this a number you’d expect?
Awesome, I appreciate you checking on that for me.
It looks like the problem server has 931,654 instances. It has been built in a lot; it has 87,795 objects in it, mostly structure objects that players have built. This averages to about 10 instances per object which is about what I would expect, yes.
Is 1,000,000 instances generally where I can expect to hit the memory limit?
Is the problem compounded by the SavePlaceAsync call (serialization doubles memory usage or something like this)?
Are there tools available to get a breakdown of memory usage by lua/instances/other, measured absolutely but also relative to the server limit? I’d like to do some optimization but unsure how to approach it without an idea of where I’m memory heavy; if it’s lua, or specific problematic instance types, etc.
Furthermore, is there ingame API to access total server memory usage and usage relative to the max limit? I’ll need to have or approximate this information in order to design the highest possible ingame build limits.
Sorry, I missed the follow up question on this, thankfully a colleague pointed out to me that I have yet to answer your question!
Unfortunately, I don’t have great answers. I’ve asked around internally, but we don’t actually have great guidelines for you guys on “what’s a reasonable Instance count to support?”.
That being said, it does look like most game servers will be running worlds that contain less than ~1 million at a time, whatever that might imply.
The limit is a separate number, that I believe we have published most recently here: Even More Server Memory
One thing to note might be that you need to have some memory buffer when you’re testing in Studio, so that memory incurred by joining players’ avatars and such won’t push your Server’s memory usage over the edge
