An increase in server crashing (Around 1-2 minutes for full private servers, can vary for public servers.)
Direct links to games that this occurs in:
This has been happening since around July 14th, at around 8 PM CST.
We have not updated anything with the code for weeks (June 13th was the last time majorly) to the experience and this just popped up on the 14th around that that time.
System Information: Windows 11 Gaming PC with around 200 MBPS download ethernet connection stable.
Expected Behavior
When a full server occurs/server fill happens, server memory goes up way more than usual for us versus before.
This causes a crash to occur.
(We have a higher memory start-out count than most games which can cause crashes earlier/work on code is needed, but we have never had crashes in 1-2 minutes ever in full servers, rather it usually occurs around the 1-2 hour+ mark and not in this short of time.)
Actual Behavior
I’m not sure exactly on what category/what exactly is causing this, as we have not changed anything for a bit and this popped up randomly to the extremity of this server crashing.
(This is near when the server crashed, and how it spikes up fast.)
Issue Area: Engine Issue Type: Connectivity Impact: Very High Frequency: Constantly Date First Experienced: 2022-07-14 16:07:00 (-05:00) Date Last Experienced: 2022-07-15 02:07:00 (-05:00) A private message is associated with this bug report
IKR this has been happening to me a lot of times even on games like CDID, maybe they’re just having an internal server error. restarting ur internet could prob work.
Greenville dev here–when trying to watch what’s spiking on new player joins (what we think it is related to), it’s hard to keep track of all the categories at once, so I took a generalized screenshot of all the memory categories in a game that has had many players join/leave it versus a brand new server.
Frighteningly, it seems the culprit is UntrackedMemory, since the only category that isn’t UntrackedMemory with a large enough difference is Gui (374 MB after ~150 random player rejoins, 341 in a new server. I’ll investigate this too, of course).
UntrackedMemory is 595 in a fresh, new server, and 970 MB in that same server after those 150 or so player rejoins (rough estimate from the amount of kicks I did; 24 per kick as servers fill, I did it 8 times but it wasn’t always full).
Edit: Going to do another test more carefully watching UntrackedMemory as new players join. I should just record each instance, haha.
Edit 2: After kicking players just twice, it does spike up to 900+ MB, but it seems it spikes back down (still higher than before the kicks, though). It does keep spiking higher and higher than previous though; so I’d say I’m pretty confident that UntrackedMemory is still the culprit, but less so I suppose as it does spike down here and there (likely some sort of delayed/time before garbage collection is completed).
I assumed that this change was only available in new servers, so I asked some players to help me test it in a “guaranteed-new” private server, and it seems to still be an issue. It doesn’t take many rejoins for memory to get this high:
EDIT: I should’ve clarified it started around ~450 MB.
I’m doing my best to keep an eye on other categories and nothing seems to be raising even remotely close to how untracked is.
Let me know if there’s anything else I can do to give useful information. Untracked is fluctuating quite a bit, but it always ends up higher and higher with player joins.
I’ll also try to join a new public server to see if there’s any differences.
EDIT 2: Well, I guess I should’ve also checked on player leave, and maybe provide the graph so things are a bit easier to understand… oops
This is right after the players were removed, not actually from join (the spike at the far right, that is):
It seems to end up higher than before the rejoins though, as before.
EDIT 3: milk74l8O is reporting that memory is taking longer to rise, to add. It seems it’s cleaning better (if that makes sense).
I guess my excess of rushed edits can be summarized like this: It’s doing better, yet Untracked is climbing pretty high with leaves/joins. Milk74l8O’s server is up to 1075MB untracked (more players than my private server for testing, though).
Hey there! Thanks for the information! I have some additional questions:
Is your experience still crashing extremely quickly? Or is it running for longer, but you’re still seeing the UntrackedMemory issue?
Is UntrackedMemory spiking a new issue? I’m wondering if it could be true that the experience has always spiked UntrackedMemory, but we just didn’t see it until now because we only check for this kind of stuff when games start crashing.
By your explanation, am I correct in understanding that the UntrackedMemory usage scales with the number of players in the game? As in UntrackedMemory spikes when players join, and go back down when players leave?
It does appear they’re continuing to crash pretty fast. Milk’s testing may have been within error. I wanted to wait until there was actually new servers so I wouldn’t give inaccurate information, and now I can say with confidence they’re still crashing pretty quick (not sure if slower/faster since we’re not tracking server uptime).
UntrackedMemory spiking may not have been a new issue, but I’m actually not positive; servers would crash months ago after a few hours of uptime, so we pinned this on “the game needs work”. However, when we noticed it got very, severely worse without any update or change to the game, we explored the possibility of it being engine-related. It being “untracked” made this more believable to me; even despite the game needing work.
And, while in my explanation I mention UntrackedMemory trickles back down, it never returns even closely to it’s original usage. It just continues to climb if players are repeatedly leaving and rejoining. So it does go down after a bit from the initial spike, but it climbs scarily fast.
As a side note, the game currently stores all vehicle models in the game at all time. We’re actively working to move them to individual Packages (which we should’ve done from the start for a game with this many individual vehicle assets), but we’re mainly doing this to help mitigate the issue as we thought it was a game-specific problem prior.
Let me know if there’s anything else I can help with.
Thanks for the answer!
Yeah, if servers are still crashing quickly, I think that means we still have a problem on our hands.
We’ll continue to look and see if there are any other internal changes that look suspicious. Hope to get back to you with good news soon.
Well I would have said it started earlier, but I’m unsure of if it’s a problem with the specific experience’s optimizations or an engine level issue, as I mentioned. Milk simply listed it as starting yesterday because that’s the first time we noticed it got worse on it’s own/without any experience updates. As we added content to the game, crashing would happen sooner and sooner, which we summed up to us adding more cars to the game, which would increase the “from-start” memory usage.
The “rapid/extremely fast” crashing did just start yesterday, though, where servers would take under 10 minutes to crash. We did observe servers could crash as quickly as in 15-30 minutes before yesterday, likely thanks to the content update we had on 6/6, and a smaller additional content update on 6/11-6/12.
I see. Would you mind double checking to make sure that new servers still crash quickly?
The reason I’m uncertain is because the change we reverted due to your report seemed to bring back Out Of Memory Crashes back down to normal for Greenville and other games, so we think the crash rate should have improved since we reverted that change, and it’s confusing as to why it hasn’t.
We’re wondering if maybe the Untracked Memory spikes are a different problem, related to either an update you guys made to your experience or a change we made to our engine long ago, and we only noticed it because you guys measured just now when it started crashing due to the change we’ve since reverted.
It seems to have improved! I may have been in an older server when testing before in my replies above.
It’s very likely the remaining issue is a problem with the experience itself. We have a good amount of work to do, but yes; it seems things are back to how they were prior to the more severe crashing that appeared yesterday. Untracked is still fluctuating and rising when players leave and join, so we’ll have to investigate on that front ourselves as I’m not doubting the idea that it may be related to Greenville (once again, the game could definitely use some work optimization-wise, haha).
