The server microprofiler shows a completely healthy server on the over HUNDREDS I’ve recorded. Other than the random occasions of big SendData and Disconnect frames. The server has been entirely healthy pinging a discord webhook through a tick() every 60 seconds sharp.
I’m full on betting this is a RakNet bug or some kind of RakNet issue being abused thats causing this insane queueing.
Our lag has been happening over… and over… and over… It’s starting to be a daily occurance and it’s nothing we can do because we don’t have any RakNet logs. We can’t tell who’s sending the most RakNet data to the Roblox server because we simply can’t get that data…
Here’s yet another video, showing that everyone is complaining about lag. The server freezing like crazy, character positions not normal, chat being chunky.
https://streamable.com/7okxeq
Apparently Roblox Screen Recorder doesn’t record the Network Connection Health stats on CTRL+SHIFT+F4/F6 so I’ll have to record it again next time.
I’ve completely modified ChatService to not rely on Plr:Chatted by removing the C++ legacy fire event. I’ve completely renamed a few of the ChatService remote events so this isn’t a chat spam or command issue.
I’ve checked the game to have 0 unanchored parts. I’ve checked and made a script to force all non character parts to be SetNetworkOwner nil to have it server only. I’ve checked Server Scripts on the DevConsole and clicked on Rate (/s) to sort and see what’s firing the most and it’s nothing abnormal. I’ve reduced the attempts I’ve been using RemoteEvents even though the Server Logs doesn’t say anything about Remotes being fired too much from a player. I’ve even swapped between using Adonis or Basic Admin Essentials only, neither has a difference.
I’ve exhausted all my efforts, I’m blaming this lag issue on a RakNet buffer queue.
I really wish I could slap [ROBL CRITICAL] on this issue because this is happening to multiple games still.
We can’t do anything about it because I have a strong suspicion that this is either some DoS or RakNet exploit. The Roblox server is just fine and isn’t even having a long frame over 10 seconds but every client ingame is just frozen for upwards of 10 minutes, completely unable to play the game…
If a Roblox Staff would also want to know information about my captured packets from the game host IP, please DM me before 10 hours of this post, otherwise I’ll just dump all the zip files here of all the captured packets from RakNet.
This issue is very negatively effecting my game, making it almost unplayable due to these exploits. Please fix this as soon as possible. Exploiters are able to shutdown servers in my game through this and it seems there’s nothing I can do.
Upon every exhausted effort, I can finally conclude that this is someone using a DoS attack or RakNet exploit against the game servers that made the server and that the games we’re playing on.
The only fix for this is to either wait for Roblox engineers to add DoS prevention measures like Cloudflare Spectrum or for Roblox engineers to find a way to work around this unwanted traffic jam.
or for game developers to work on split server cross gameplay, a method of splitting players between multiple Roblox servers and either using MessagingService or HttpService with your own server to relay data between all the other servers to handle showing all the characters and game data.
We’re still yet having this issue for over 40 days.
Here is an entire list of game job ids that I have available that had the game lag for Roblox staff to view through it.
1591825071.25897,1f7c,6 ! Joining game ‘775b9260-07a7-4395-9056-c8eb835c439f’ place 2698066019 at 128.116.42.70
1592254411.40699,4f0c,6 ! Joining game ‘7b8c3158-4946-4e90-a8b5-2103c9a3a508’ place 2698066019 at 128.116.4.103
1592254411.40699,4f0c,6 ! Joining game ‘7b8c3158-4946-4e90-a8b5-2103c9a3a508’ place 2698066019 at 128.116.4.103
1592342524.64920,3f30,6 ! Joining game ‘f8afb967-ef90-4112-84fd-24308e0d5b1e’ place 2698066019 at 128.116.24.153
1592601368.21216,1dfc,6 ! Joining game ‘345ba639-e6f0-49a9-b726-706be203e46b’ place 2698066019 at 128.116.34.25
1592860096.39523,0798,6 ! Joining game ‘0743b4a1-88c5-455d-a0dc-496d904e2595’ place 2698066019 at 209.206.42.108
pcapng for section 1 through 3 events - 6-22-2020-1-3 lag events.zip (2.9 MB)
1592950723.75232,3f8c,6 ! Joining game ‘39fb1318-c90f-4aa0-8099-f5f1ece59768’ place 2698066019 at 128.116.35.88
This is still a game breaking issue as no body can play our game.
Here’s something that I’ve seen going around that some people are claiming is causing our game lag.
We’re still having this lag issue and it’s getting worse to the point that our group may fall apart along with many other groups due to the game being completely unplayable.
I dumped it and it contains a script which spams the SayMessageRequest remote. It is part of the default Chat. Put a debounce and/or length limit on the OnServerEvent connection in ChatServiceRunner and you’re good.
As the above post states, exploiters can spam the chat remote with a large amount of data (Such as strings with a large length.) to lag game servers. I would suggest possibly adding a small debounce to the chat remote (A minuscule debounce so your regular players aren’t affected by it.) and adding a check for the length of the chat message. Currently there is a max length that usual players can send (I believe the default setting is 200.) so you can cap messages at the length of 200 letters.
For instance, I modify the chat script to do this within my game to prevent exploiters from lagging my game by firing the chat remote with large strings:
[As a note the debounce shouldn’t be necessary with the length check.]
The script that manages the chat remote is located in: Chat.ChatServiceRunner
The script with the setting to change the maximum length chat is located in: ClientChatModules.ChatSettings as variable module.MaximumMessageLength
I have no idea as to why Roblox staff has agreed to remove my posts stating that I’ve already done this fix attempt by completely renaming the SendMessageRequest to something else with the issue still occurring in our games. This happens in other games also so I know it’s not one specifically targeting us and that it would need to be something similar in all of our games.
They even removed my message about adding a bounty to try to help get this issue fixed faster because this is still affecting our game since May.
Edit: As of now, I’m still keeping my bounty in place, $35 for whomever can help us out.
We’re already ratelimiting all remoteevents, remotefunctions, clickdetectors.
We’ve already disabled Roblox legacy C++ chat to no avail.
We’ve attempted to SetNetworkOwnership of the only 4 parts we have unanchored in our game to nil.
We’ve attempted turning StreamingEnabled on and off.
All the characters in are game are set to not collide with eachother using CollisionGroups.
We’ve reduced ChatService to not sent an RemoteEvent for a message that isn’t yet filtered.
Already attempted to click ‘start recording’ for the server microprofiller over 80 times with all results showing no issue of lag other than a single long frame of a second of a player receiving data on join or disconnecting, etc.
We’ve already had another Roblox staff ask us with a back and forth chat that hadn’t been responded to for a week now, inserting ‘sanity checks’ against the time it took to load a Heartbeat and if it goes over 10 seconds to log it with them seeing nothing in their logs about long frames.
We’ve already reduced the customized ChatService max character limit to be lower than default to fit the bubble chat more approperately ingame and had it kick players that go one character over this max limit.
We still have this issue and is willing to get suggestions about it.
Our next game session will be 8/10/2020 at 5:30 pm CST if anyone wants to join the group and try to catch the event happening again.
I haven’t had a chance to actually check the ping but everything in the game was being delayed by about a minute. Chat messages took at most 3 minutes to get through.
Yes of recent like last update Ping is going excessively high for no apparent reason. All my scripts etc havent changed from before yet theres ping levels exceeding what they did before which was 90 at the very most. Wierdest thing is that It seems Roblox player restricted, studio play mode doesn’t show such high ping
Now it is happening in our game too, but it is a little bit different. After some time, everything that comunicate with server stops (clock time, other players’ movement etc.), but client still can walk / drive / make client sided actions. After another few minutes player actions stop responding too and player have no option than rejoining the game. It happened at the exactly same time for all players on server. (Players have reported that issue trough VC they were in on another platform.) This bug also doesn’t change with players count. I had it twice - once on our game shift and once when I was there all by myself. Pretty weird. This game is just growing and is not open all the time - so we don’t have a lot of reports.
I can try your solution of kicking players, but I don’t think this will help. If it will, I will tell ya.
Screenshots
You can see both screenshots are taken about 2 minutes after server stopped responding (server time: 18:38, time screenshot was taken in: 18:40) … but maybe count of players can have affect, because when there was like 10 of us, it took a little bit longer before showing that Disconnected dialog (about 10 minutes).
i can’t find a solution for this like, it was working just perfect, even tried older versions of the game that worked perfectly but this still keeps happening.
i share my post about this, before i didn’t know it was due to roblox servers being unresponsive to all clients, the only thing to kinda soft these “lag” spikes is lowering the player count from 50 to 25, which the game could’ve handle 50 players before but not now.
