I’m starting to think that this is a RakPeer issue.
It’s apparent that Roblox uses RakNet and RakNet has options from peer to peer traffic for CongestionControl etc…
Is an exploit overloading server traffic to the point it’s hitting a buffer limit and we’re forced to wait this buffer limit out?
I’m unable to tell since I don’t think there even is lua access to see this information on a Roblox client to see if isLimitedByCongestionControl is even true.
If you truly want to get rid of exploiters using DoS through your game (application layer), you won’t get around auditing your entire server code, looking for things which can be spammed and are quite expensive.
Look out for:
Add debounces, usage trackers and/or rewrite badly performing code. It’s important to note that the “spamming” of a signal is not necessary to cause a DoS as it is possible to send malformed data which can lead to very long/infinite loops or similiar.
This isn’t anything being abused on the server that I can even see of.
The server microprofiler shows a completely healthy server on the over HUNDREDS I’ve recorded. Other than the random occasions of big SendData and Disconnect frames. The server has been entirely healthy pinging a discord webhook through a tick() every 60 seconds sharp.
I’m full on betting this is a RakNet bug or some kind of RakNet issue being abused thats causing this insane queueing.
Our lag has been happening over… and over… and over… It’s starting to be a daily occurance and it’s nothing we can do because we don’t have any RakNet logs. We can’t tell who’s sending the most RakNet data to the Roblox server because we simply can’t get that data…
Here’s yet another video, showing that everyone is complaining about lag. The server freezing like crazy, character positions not normal, chat being chunky.
https://streamable.com/7okxeq
Apparently Roblox Screen Recorder doesn’t record the Network Connection Health stats on CTRL+SHIFT+F4/F6 so I’ll have to record it again next time.
I’ve completely modified ChatService to not rely on Plr:Chatted by removing the C++ legacy fire event. I’ve completely renamed a few of the ChatService remote events so this isn’t a chat spam or command issue.
I’ve checked the game to have 0 unanchored parts. I’ve checked and made a script to force all non character parts to be SetNetworkOwner nil to have it server only. I’ve checked Server Scripts on the DevConsole and clicked on Rate (/s) to sort and see what’s firing the most and it’s nothing abnormal. I’ve reduced the attempts I’ve been using RemoteEvents even though the Server Logs doesn’t say anything about Remotes being fired too much from a player. I’ve even swapped between using Adonis or Basic Admin Essentials only, neither has a difference.
I’ve exhausted all my efforts, I’m blaming this lag issue on a RakNet buffer queue.
I really wish I could slap [ROBL CRITICAL] on this issue because this is happening to multiple games still.
We can’t do anything about it because I have a strong suspicion that this is either some DoS or RakNet exploit. The Roblox server is just fine and isn’t even having a long frame over 10 seconds but every client ingame is just frozen for upwards of 10 minutes, completely unable to play the game…
If a Roblox Staff would also want to know information about my captured packets from the game host IP, please DM me before 10 hours of this post, otherwise I’ll just dump all the zip files here of all the captured packets from RakNet.
This issue is very negatively effecting my game, making it almost unplayable due to these exploits. Please fix this as soon as possible. Exploiters are able to shutdown servers in my game through this and it seems there’s nothing I can do.
My game is being severely impacted by this exploit. Servers are crashing constantly.
Upon every exhausted effort, I can finally conclude that this is someone using a DoS attack or RakNet exploit against the game servers that made the server and that the games we’re playing on.
The only fix for this is to either wait for Roblox engineers to add DoS prevention measures like Cloudflare Spectrum or for Roblox engineers to find a way to work around this unwanted traffic jam.
or for game developers to work on split server cross gameplay, a method of splitting players between multiple Roblox servers and either using MessagingService or HttpService with your own server to relay data between all the other servers to handle showing all the characters and game data.
We’re still yet having this issue for over 40 days.
Here is an entire list of game job ids that I have available that had the game lag for Roblox staff to view through it.
1591825071.25897,1f7c,6 ! Joining game ‘775b9260-07a7-4395-9056-c8eb835c439f’ place 2698066019 at 128.116.42.70
1592254411.40699,4f0c,6 ! Joining game ‘7b8c3158-4946-4e90-a8b5-2103c9a3a508’ place 2698066019 at 128.116.4.103
1592254411.40699,4f0c,6 ! Joining game ‘7b8c3158-4946-4e90-a8b5-2103c9a3a508’ place 2698066019 at 128.116.4.103
1592342524.64920,3f30,6 ! Joining game ‘f8afb967-ef90-4112-84fd-24308e0d5b1e’ place 2698066019 at 128.116.24.153
1592601368.21216,1dfc,6 ! Joining game ‘345ba639-e6f0-49a9-b726-706be203e46b’ place 2698066019 at 128.116.34.25
1592860096.39523,0798,6 ! Joining game ‘0743b4a1-88c5-455d-a0dc-496d904e2595’ place 2698066019 at 209.206.42.108
pcapng for section 1 through 3 events -
6-22-2020-1-3 lag events.zip (2.9 MB)
1592950723.75232,3f8c,6 ! Joining game ‘39fb1318-c90f-4aa0-8099-f5f1ece59768’ place 2698066019 at 128.116.35.88
This is still a game breaking issue as no body can play our game.
Here’s something that I’ve seen going around that some people are claiming is causing our game lag.
We’re still having this lag issue and it’s getting worse to the point that our group may fall apart along with many other groups due to the game being completely unplayable.
Here is the script found from this youtube video.
Apparently uses wallysecure orbfuscation.
VR6h8Sf6.txt (26.4 KB)
If anyone wants to help decompile that script as I have no idea how to begin.
I dumped it and it contains a script which spams the SayMessageRequest remote. It is part of the default Chat. Put a debounce and/or length limit on the OnServerEvent connection in ChatServiceRunner and you’re good.
As the above post states, exploiters can spam the chat remote with a large amount of data (Such as strings with a large length.) to lag game servers. I would suggest possibly adding a small debounce to the chat remote (A minuscule debounce so your regular players aren’t affected by it.) and adding a check for the length of the chat message. Currently there is a max length that usual players can send (I believe the default setting is 200.) so you can cap messages at the length of 200 letters.
For instance, I modify the chat script to do this within my game to prevent exploiters from lagging my game by firing the chat remote with large strings:
Chat.ChatServiceRunnerClientChatModules.ChatSettings as variable module.MaximumMessageLength
I have no idea as to why Roblox staff has agreed to remove my posts stating that I’ve already done this fix attempt by completely renaming the SendMessageRequest to something else with the issue still occurring in our games. This happens in other games also so I know it’s not one specifically targeting us and that it would need to be something similar in all of our games.
They even removed my message about adding a bounty to try to help get this issue fixed faster because this is still affecting our game since May.
Edit: As of now, I’m still keeping my bounty in place, $35 for whomever can help us out.
We still have this issue and is willing to get suggestions about it.
Our next game session will be 8/10/2020 at 5:30 pm CST if anyone wants to join the group and try to catch the event happening again.
This has also been happening in A Bizarre Day (not my game)
https://www.roblox.com/games/2686500207/A-Bizarre-Day
I haven’t had a chance to actually check the ping but everything in the game was being delayed by about a minute. Chat messages took at most 3 minutes to get through.
Yes of recent like last update Ping is going excessively high for no apparent reason. All my scripts etc havent changed from before yet theres ping levels exceeding what they did before which was 90 at the very most. Wierdest thing is that It seems Roblox player restricted, studio play mode doesn’t show such high ping
Now it is happening in our game too, but it is a little bit different. After some time, everything that comunicate with server stops (clock time, other players’ movement etc.), but client still can walk / drive / make client sided actions. After another few minutes player actions stop responding too and player have no option than rejoining the game. It happened at the exactly same time for all players on server. (Players have reported that issue trough VC they were in on another platform.) This bug also doesn’t change with players count. I had it twice - once on our game shift and once when I was there all by myself. Pretty weird. This game is just growing and is not open all the time - so we don’t have a lot of reports.
I can try your solution of kicking players, but I don’t think this will help. If it will, I will tell ya.
You can see both screenshots are taken about 2 minutes after server stopped responding (server time: 18:38, time screenshot was taken in: 18:40) … but maybe count of players can have affect, because when there was like 10 of us, it took a little bit longer before showing that Disconnected dialog (about 10 minutes).
Experiencing this bug too. At first I thought it was my wifi, my device, but in reality it’s roblox’s servers.
It has become extremely difficult to join games for me.
my game is doing the same… [🎁 UPDATE] Steven Universe Future: Era 3 RP - Roblox
i can’t find a solution for this like, it was working just perfect, even tried older versions of the game that worked perfectly but this still keeps happening.
i share my post about this, before i didn’t know it was due to roblox servers being unresponsive to all clients, the only thing to kinda soft these “lag” spikes is lowering the player count from 50 to 25, which the game could’ve handle 50 players before but not now.
this describes my issue totally, have you found a solution?
