How to fix random server crashing/Error 277/Error 267 on my game?

CoolJohnnyboy · April 22, 2024, 9:47pm

My game has been having a pretty notable issue of servers randomly crashing, with the Error Code 277 popping up. I wouldn’t say it is very frequent, but when it happens, it brings down the entire server of players, and obviously is bad for the CCU when the CCU isn’t very high to begin with. I’ve noticed that when these crashes happen to me, it’s after about an hour or more of playing the game - making the issue even worse: imagine playing for an hour, the game crashes, and none of that progress you made saves… I’d be pissed.

Keep in mind, the game is two places in one game - a two-player place and single-player place. Below are the client usage and crash rate stats for the overall game:

The total crash rate sits under 10%, but some devices, like console, have insane spikes. Console and Tablet seem to be the most affected by crashes, but all the people who report server crashes (and any issues that result from them) are PC players, which tends to have the lowest crash rate.

For the two-player place:

For the single-player place:

The single-player place seems to have some of the worst spikes, which doesn’t make sense - single-player servers are half the size (12 players max) the size of two-player servers (24 players max).

Now, for the server-side of things, here are the stats for the overall game (server-sided stats do not vary wildly between places, unlike client-sided stats):

One thing to note is, we do not use StreamingEnabled on the game at this time. The map is entirely terrain with some 3D meshed buildings here and there, as well as player bases, and the map size is roughly 4700x7200 studs. I have more info here.

This glitch also causes bigger issues, as in that when a server is taken down, it screws up the way ProfileService works. At best, players are able to rejoin, but don’t their stats from the crashed session don’t save/update, at worst, they join a new server, but their stats don’t load. I have a separate topic about that issue here. I am very confident that fixing the server crashing issue will (mostly) fix the ProfileService issue.

I have been hesitant to add StreamingEnabled to the game, since tests with it showed it would break some things, but I think it might be the only way to solve this server-crashing issue - unless I am totally wrong. Is the solution to this as simple as making StreamingEnabled in the game? Or is this going to be a much more involved process to fix?

CoolJohnnyboy · April 22, 2024, 10:05pm

Worth noting is, a developer with a game that is at 4,000+ CCU now had a similar issue, but had a fix described in this post:
Why do players keep encountering “Error 277” on my game? - Help and Feedback / Game Design Support - Developer Forum | Roblox

I do not think that would be applicable to this, but I could be wrong.

ChocolateAnts_33 · April 23, 2024, 2:26am

I’m sorry, there was something I forgot to mention in my response in the post you provided.

The script I sent is for the server side, but since the Character’s Network Ownership is held by the client, it might be a good idea to also place a similar script on the client side to achieve the same functionality.
However, considering that your game’s Character is likely using the default Character compared to my game’s completely physics-based Character, it may not have much effect.

If there are physical joints (especially BallsocketConstraints) attached in any parts in your game, and the anchor is turned off, applying that script to those parts may be beneficial.

Regarding the issue with ProfileService, I also encountered the same problem, so I ended up rebuilding the database system from scratch. However, as you mentioned, I believe resolving the issue of the server crashing could also resolve the issue with ProfileService.

I hope this helps resolve the issue.

Tomi1231 · April 23, 2024, 3:33am

If the servers themselves are crashing, then client framerate and client crash rate are not useful metrics, and streaming enabled wont fix it as streaming enabled only reduces memory usage on clients (though it could reduce the crash rate of some clients, if memory is an issue)

CoolJohnnyboy · April 23, 2024, 1:39pm

The only constraint parts we have in the game are welds for holding things together, hinges and springs and such for vehicle movement, and AlignOrientation and LinearVelocity for plane movement.

As for ProfileService, I’m sorry to hear you had a similar issue. I worked on this game before ownership was transferred, and I used the same setup/format/etc for ProfileService there, and the only issues I have had there with it were when servers crashed, which was such an incredibly rare instance.

I also had no checks for constraint parts moving too fast there and did not have any issue.

CoolJohnnyboy · April 23, 2024, 1:41pm

What could be causing these crashes then? And what would be the solution?

From what I have gathered from my research on the problem, all the players in the server get the error that says “please check your internet connection,” so the server goes down. From what I’ve gathered, that is a client-sided error - but why would they all be getting it at the same time?

Tomi1231 · April 23, 2024, 1:45pm

Hmmm that is odd

Maybe it could be something replicating that is just too big, causing the clients to lose connection???
I really don’t know what causes this error when the internet connection is fine, so I really can’t say

When the network trafic is too high, the ping slowly increases towards infinity, so that can’t be it. Idk what a single remote or replication that is massive would do, if it is even possible for one to happen

CoolJohnnyboy · April 23, 2024, 2:03pm

Yeah, it makes no sense. Only thing I can think of is implement StreamingEnabled, try looking for memory leaks, and hope and pray for the best

ChocolateAnts_33 · April 24, 2024, 1:58am

If there are no issues with the parts that physically move, there doesn’t seem to be any problems that I can think of…
I played your game for a little while, and except for one time the network received exceeded 300mb, there didn’t seem to be any major issues.

It might take some effort, but I think it would be a good idea to copy the game to a different place and gradually remove each function one by one to determine which system is causing the problem. (such as vehicles or weapons)

I’m sorry I couldn’t be of more help.

I hope you will find a solution.

CoolJohnnyboy · April 24, 2024, 1:59am

Thanks for playing! Means a lot to see successful developers playing my games.

IIRC, turning StreamingEnabled should help with the ping issues. Right now, the 7200x4200 terrain map and all its glory is being rendered all at once for players…