I just finished writing this and realised how long it is - oh well, at least I’m going in-depth.
EDIT 1: Added the only images I could find of the black hole issue.
EDIT 2: There’s a new way for people to gain access to server scripts. My Trello key was breached and the entire contents of my account was deleted in seconds. The game was thoroughly cleaned for backdoors - I’m extremely concerned.
This seems to be the most suitable category for my issue - however, if not, I apologise.
.
Lately, I’ve been experiencing two very weird and serious issues in my game and, despite spending days scouring for and removing any potential cause, I’m still stuck scratching my head while my player count dwindles.
I run a roleplay game which has recently gained a fair amount of traction - as in, we’ve gone from an average of 35 players per “start-up” (organised times for the game to be opened) to in excess of 70 and gain around 70-100 members in our group every two days. We vet every pending request to the group and decline any and all alternate / suspicious accounts. This has deterred a vast majority of exploiters.
However, ever since the game started to gain popularity, an issue began to occur which we’ve never seen before. It’s a bit hard to explain properly, but I’ll try my best.
Issue 1 - The "black hole"
At random times, a sort of “black hole” develops. This “black hole” appears at a random point in the map, although always near players. A favourite target of this “black hole” is the police station when the server opens, as this is where a lot of people are.
I’m calling it a “black hole” as it sucks in all unanchored objects from across the entire game. Cars, bodies, tools, lone unanchored parts - everything gets sucked towards this point and begins orbiting it at a very close range. It doesn’t destroy anything, it just sucks it all in.
This “black hole” typically has a sparkles effect. When selected with F3X, one-two tiny invisible parts are at the center of it - when deleted, the effect subsides. It can (and usually does) pop back up later in another place.
This issue alone strongly affected the game, as player’s purchased vehicles would fling from across the map and be sent into orbit, rendering them out of pocket and stuck on the side of the road miles away from where they wanted to be. It also disturbs other aspects of roleplay by sucking in corpses, police evidence markers, shell casings, dumped weapons - you name it, if it’s unanchored, it’s going to the hole.
We’ve come to the inevitable conclusion that this must be an exploit - it’s previously subsided after banning highly suspect players, it’s not caused by a script in game and it doesn’t happen in the server alone (as in, I’ve sat ingame for over an hour and a half to test and nothing happened.)
.
Gyazo gif of vehicle being flung away from a player to the black hole (the other vehicles are safe as they’re anchored a few seconds after the player exits them): https://gyazo.com/cd17435c68331ee4b5b5249bda0d404a
Example of what a “black hole” can look like - a car has been sucked into one that appeared outside the Fire Department.
Issue 2 - Server crashes
Now, while the “black hole” issue has been happening for a few weeks, a new issue popped up a few days ago - server crashes.
After anywhere between an hour and two hours, the server will stutter, lagging for about 5 seconds to begin with. After this, everything returns to normal for another 10 minutes, before the server gets hit with another lagspike - this time, longer. Another 5-10 minutes later, the server crashes.
We haven’t experienced this issue before, but it truly is game-breaking.
We haven’t added any new scripts that could be causing this - all core scripts have functioned well since 2018 (yes, I am working on rescripting the game as it’s pretty old - but point is, they’re stable.) and no new features have been added in the build-up to this.
Recent changes
When the “black hole” issue began happening, absolutely no major updates had been added. The only things added to the game were a few police cars and fire engines - all of which were originally created, scripted and thoroughly checked for any backdoors or malicious scripts that may have been inserted by a plugin. A few backdoors were found in the fire trucks - clearly from a plugin - and were promptly removed. This developer does not have TC access - the two of us who do only have reputable plugins.
When the server crash issue began, the only things that were changed were a few minor map updates (again - original creations, not risky free models) and a lighting overhaul. A new skybox was added alongside the new Atmosphere feature. Disabling this proved to be useless, as the issue continued.
Exploits
There are no publicly available scripts for our game in any major exploiting forums / servers - a community member with access to several of these offered to check. This, along with the fact that we strictly vet for alts - leaves us in a confusing place. If this is an exploit, it’s clearly either a single person or a small group of people who have access to it. Several people have been banned in connection to it, yet it continues to happen. If it was a backdoor, we’d have seen something a bit different rather than the same, semi-destructive thing happening over and over again.
However, we recently confirmed that at least three exploiters were ingame over the past couple days. They were pictured by a player somehow exploiting a remote that we thought we’d secured (by adding a group check serverside). These exploiters were also able to open a door that was linked server-side to a Trello board - I don’t mean walk through it with clientside btools, I mean unlock the door using the actual intended method - by clicking the handle. Usually, when a player clicks the handle, the RFID scanner glows red and beeps at them. When an authorized person (currently only me and an administrator) attempts to open it, the RFID scanner glows green, it chirps and opens. This is what happened when they used the door.
On a potentially related note, our Trello moderation wasn’t working last night. The intended function of this is to create a card on our Trello board when someone is banned. When a player joins, it checks for a card with their UserId on the Trello board, kicking them if one is found. This has worked for a year and a half. Last night, however, when banning someone, the script would return successfully, kicking them. However, no card was created (even though the return and kick function are after a card creation pcall which returns if it’s unsuccessful) and the player can happily rejoin with no worries. It also failed to kick players who had been previously banned when the system was working, despite the same players being kicked prior.
What we’ve tried
I’ve performed several thorough searches for backdoors and have come up clean every single time. As well as scanning with Kronos, I’ve searched all scripts for the following keywords:
-require
-getfenv
-MarketplaceService
-RunService
-string.byte
-string.reverse
I’ve tried rolling back the lighting update, theorising server memory issues related to the new setup - no luck.
I’ve tried running an empty server for over an hour and a half - nothing happened.
I’ve tried lowering the max players to 40, something that the game can definitely sustain - still happens.
Tonight, I’ll be reverting to the “last known stable game version” - as in, a version of the game that was published and played before these issues arose. I’ll update this thread with the results of this test after it’s done.
.
.
Any assistance, theories or ideas are greatly appreciated. We’ve had to delay a major update that’s been in the works for 3 weeks because of this. This has had me scratching my head for nearly a week now, and my game is greatly suffering because of it. I apologise if I don’t make much sense towards the end, I seem to have lost my English skills about halfway through writing.