Servers across universe have unexpected OOM crashes while memory snapshot shows much lower usage

Bug Reports Engine Bugs
,
1

Our experience uses a lightweight lobby/start place to teleport players to several places within the same universe.

The destination places are substantially heavier, with large maps, and a high player count per server (e.g. 100+ players). However, they operated without this sudden pattern of crashes, including servers with much higher player counts. Recently, servers across both the lightweight Hub and multiple heavier destination places have begun crashing repeatedly. Most are classified as Out of Memory, while others in the same incident windows are classified as Platform Crashes.

The hub/start game can be found here. (Place ID 130236446350677)
Experience ID: 10111700956

As an example, a hub/start place server running from 11:40 AM to 12:47 PM BST crashed at 18,622MB used, and several replacement servers (around 4 more) also crashed with similarly high memory usage. There was then a crash with around 534 MB usage classified as a Platform Crash. The memory snapshots only show 48-71 MiB and none show the indicated high memory readings from the server management details so there is a large amount of unaccounted memory use in the snapshots.

The server crash logs show no consistency or relevance to the crashes. Initial crash usually occurs with error code 277 for players, then shows 279 when they attempt to rejoin.

Crash log examples, with 47GB memory for a 6 minute server:

image
image2081Ă—963 183 KB

Graph showing sudden spike in platform and OOM crashes:
image
image2113Ă—975 228 KB

We have been running similar games and setups for many years without issue.

Expected behavior

Public servers should normally remain fairly stable and did not start crashing until around ~2 weeks ago.
When an OOM occurs, the snapshot doesn’t account for the indicated memory used, and the server management filter does not categorise any of the crashes as OOM, only as a regular crash event.

A private message is associated with this bug report

16 Likes
2

what error code do u get on server crash?

1 Like
3

Normally error code 277 for initial OOM crash and error code 279 when people try to rejoin

3 Likes
4

show me the memory logs, i may be able to help you

2 Likes
5

or actually show the error reports and send it here,

image
image229Ă—159 2.76 KB

it will help with troubleshooting

2 Likes
6

The logs don’t really reveal anything as they are different each time, sometimes a crash happens with a simple expected log

2 Likes
7

Happening in my game too. All of our servers in past 2-3 hours have been crashing within 5 min with either OOM or Platform Crash as the reason, and I can’t find the culprit yet. No prior crashes. Hopefully someone can give us a clue.

My game also uses teleportservice extensively for custom matchmaking, I wonder if it has something to do with that?

3 Likes
8

Yes - mine is using teleportservice to place as many people into a specific server jobId to fill it as much as possible, that sounds similar to your problem. I don’t get how the lobby start game crashes too despite it using default matchmaking for players to join that

2 Likes
9

Is it still happening to you? It stopped for my game just now after 2 hours of repeated server crashes.

1 Like
10

Still having issues, been this way for around 2 weeks then it stopped and came back in the past few days

2 Likes
11

image
image1415Ă—646 61.3 KB

All my servers were just crashed. Though they say “Crashed” and not “Out of Memory”, the creator hub analytics page indicates 10 out of memory crashes in a very short timesepan, which aligns with the information in the image above.

2 Likes
12

I’ve just run into the same issue with our servers as well.
Game: Bobruisk RP
Screenshots:

image
image423Ă—495 24.9 KB

image
image1341Ă—410 32.5 KB

Typically, our Memory Used is around 3,000 MB - 4,000 MB when the server is fully loaded

1 Like
13

Hello,

We identified several bugs, and rolling a fix in some time.

We will investigate further, if there are additional issues.

Thank you.

3 Likes
14

Our game is also being (probably) affected by the said bugs. Tons of OOM crashes for the past 12H