Hello developers!
Over the last week, I’ve had issues with 2 games related to a group (The Robloxian Army TRA) I develop for.
These games are both ‘forts’, in war clan terms this means that other clans raid this place in an attempt to gain a victory for their clan. Both these places are very popular amongst other clans and our own members, and gain more traffic than our other games.
-Outpost Frostbyte
-Castle Beaumont
NOTE: I assume other games that share a similar feature set to these 2 games likely have the same issue, however I didn’t receive any reports from these games (they’re not as popular and therefore less likely to have a server that crashes, as there’s only a handfull short lived servers on a daily basis at those places).
Ever since I’ve published a new menu system (on the 26th of June) I’ve been receiving reports of servers crashing at these games.
note: I’ll be censoring the names of some members to respect their privacy.
As soon as this crash occurs, all clients receive error code 277 and are unable to rejoin this server (error code 277 keeps appearing when attempting to rejoin).
Since there’s a large American audience (and I’m from Europe, thus unable to instantly jump into studio at night to solve critical bugs), other co-developers for TRA (abbreviation of our group) kept an eye on how the situation evolved at these places.
On the 28th of June, we kept receiving reports that servers were crashing.
One of my co-developers for TRA, @Flowtonio, reported this in one of our chat rooms;
He also received messages from another user, explaining that VIP servers at these games are also affected by this crashing issue.
note: the image says ‘Today’, but was taken on the 28th
We decided to revert our menu update shortly after (about half an hour after the last report, still on the 28th of June).
Ever since we’ve reverted this update, we haven’t received any reports of server crashes whatsoever across all our games.
Since we’ve reverted the menu, we’ve had more than 40 servers, both VIP and public servers, run for over an hour without any reports of crashes.
On the 30th of June, we decided to publish the menu update we made to live servers again.
Once again, we’re receiving reports of servers crashing.
Our games are connected to GameAnalytic, for debugging and analytic purpose. We didn’t find any errors that could explain why a server would crash (this is a quality overview of the 30th of June).
Of course, we also took a look at previous days that experienced server issues - there were no errors that could explain why the server would crash.
This gives an indication on the timeframe we’ve been experiencing issues, and bug reports we’ve collected as a development team.
Now, I would like to explain the update we’ve rolled out that might be causing these issues (we’re not sure if this issue is caused by us (we don’t have any trace to errors), an exploiter (seems unlikely, considering the timing with our update and since VIP servers are also affected) or a roblox bug.
This menu update is essentially a visual overhaul of a menu we’ve been using for months. We haven’t made any changes towards server components whatsoever, we’ve only updated the structure of a GUI and the localscript (thus, only changes on the CLIENT, not the SERVER).
In this visual overhaul, we made minor stylistic changes (its in line with our other UI design, nothing unordinary whatsoever) and we added a ViewportFrame with a WorldModel to give a real-time preview of your character appearance (we allow our players to wear any cosmetics we offer, wear new shirts and pants…).
It looks like this;
Because we want to make use of roblox emotes on specific actions in the menu (for example, equiping a new outfit or equiping a new accessory), we copy the appearance of player character (game.Players.LocalPlayer.Character) to a preset R15 rig, no matter what the rig of your LocalPlayer.Character is (R6, R15 or RTHRO).
At runtime, the folder structure looks like this;
Now, my character is correctly displayed in the viewport frame, on an R15 rig.
Nothing unusual so far, right? Just a viewportframe that makes use of the fairly recent WorldModel object, in order to display an animated character preview that consists of an R15 rig in order to make use of roblox emotes.
Now, things get slightly ‘hacky’ here.
Because this character preview is only on the client, and we don’t want any extra networking with the server, we decided not to make use of HumanoidDescriptions. HumanoidDescriptions can only be applied by a script (not a LocalScript), but for our use case we didn’t want any networking with the server to apply a HumanoidDescription, because its a preview of the character of your LocalPlayer.
Reasonable so far, but how do we copy over the appearance of our LocalPlayer to a preset R15 rig then?
We decided to handle the appearance manually, dynamically, just like developers had to do before we had HumanoidDescriptions to make our life easier.
(HumanoidDescriptions were only released in January 2019)
Here is how we handle this, code wise;
The setupVPF function is called every time when a player opens his menu (that is, pressing the toggle button to open the menu), to make sure that the latest character appearance is shown, to show the most accurate preview we can. (For example, when a player picked up a hat or changed his in-game appearance in some way)
In order to display any changes in appearance when the player is in the menu, we make use of these functions (these are used within the setupVPF function)
Because of this issue, I’m forced to revert this update until we/roblox can provide a fix.
We assume there might be a roblox bug with WorldModel, considering it’s a fairly new feature, or an issue with our ‘hacky’ implementation as an alternative for the HumanoidDescription system.
We’ve only had reports of server crashes with this update in our players their hands, therefore it’s likely that it is directly related to our implementation of a character preview by the use of a ViewportFrame and WorldModel, that came with this update.
I’m willing to give a copy of our games to a roblox employee if they want to take a deeper look.
If anyone has or had a similar problem, feel free to mention me - maybe there is some correlation that could lead to a solution to this problem.
EDIT: 29/07/2020
Was on vacation for a while. I re-enabled this system that caused crashes today.
A couple hours in we’re experience a server outage again in games that make use of this menu system.
2 of our players sent me their log files, they’re included here.
https://devforum.roblox.com/t/log-files-from-2-of-our-players-that-experienced-a-server-outage-same-server/696614
Once again, error code 277 for all our players, no error trace.
