Roblox’s Algorithm Changes Are Killing Our Game

ruddev_ethan · June 8, 2023, 11:58pm

Introduction

I run a team of 4 full time developers and a rotating team of part-time contractors called Ruddev Media. We’ve been responsible for many projects throughout the years but the project that our team has depended on over the last 4 years has been Bad Business. This is an FPS game that has 390 million visits, been featured in platform-wide events (Egg Hunt 2021, Ready Player Two), and was initially funded by Roblox in part of the Accelerator Program.

Starting sometime in mid-late April and early May, Bad Business has seen a sharp decline in new players, seemingly coming from us not appearing on people’s home pages anywhere near as frequently. Our average new users have halved between April and May. Our average new users from the home-page in particular have plummeted, looking to be just a quarter in May than what we had in April.

Based on our stats, I don’t believe we deserve this sort of punishment. We suffered a very slight dip in Day 1 retention in early May due to an unbalanced update (game dev is hard!) - but these trends were already occurring and it was nothing too out of the ordinary for the game. Aside from that statistic, everything else looks… normal? It’s best to just show you below.

Statistics

Disclaimer: Early March stats show extremely different statistics from the rest of the month, and the rest of the year. It looks like there was an adjustment to the calculation of these statements, but I decided to leave them in so as to not mislead anyone.

Disclaimer 2: I’ve included June stats so far. This is obviously not a full data set compared to the others, but I felt like it was important to show that this issue is ongoing, as well as to provide another data point after we made an update.

The first image here is our New User Home-Page Acquisition stats. As is the case with all the acquisition stats, I exported a period of March 1 - June 6 and graphed the data. The blue line is our daily stats, and the red dots are the monthly averages. With this I included the trendline for our averages as well.

This is another look at the averages and how they changed month to month - as you can see, our May averages are just 23% of what our April averages were. This is an intense drop, and June so far looks worse, despite us having just put out an update and bringing in an influx of interest to the game.

The next stat I wanted to display is our home-page impressions. Take note of how our peaks became more subdued over a very short period of time. These peaks are (predictably) over weekends. Our current peaks are lower than our dips previously.

And again a look at the averages of these months.

When looking at our overall new players statistic, we can see that it matches up directly with these changes in the algorithm. These downturns are all happening around the same time.

Take a look at the averages here - about half of these players have disappeared.

Conclusion

So - we need help.

The current trends for the game, seemingly due to something out of our realm of responsibility, is pretty damning. These rapid changes are endangering this game, and my team. Despite our ARPDAUs and ARPPUs being above average (according to dev stat benchmarks), this is just not sustainable. We saw a massive decline in revenue last month, one that meant I personally had to forfeit a paycheck.

We are not a huge game. And with that, we are not a huge team. And yet as of just a couple months ago, we were extremely stable, and seemed poised for future growth. Now our future with our game, our company, and this platform in general, have come into question.

Roblox - PLEASE rethink these changes to the algorithm, and please offer more transparency on algorithm adjustments in the future. I know that we are not the only game on the platform feeling these effects.

If you are a developer and have been noticing these changes in your acquisition stats, I highly encourage you to reply with them below! We need to show how this is hurting developers.

Page URL: https://www.roblox.com/games/3233893879/ANNIVERSARY-Bad-Business-3-24

Noble_Draconian · June 9, 2023, 12:36am

Our game, Super Skyward Towers has been having the exact same problems as @ruddev_ethan has described.

Our game’s KPIs have largely held steady up until May 2nd of this year. Retention, CCUs, etc have all held on the same trend with the usual spikes and dips around holidays. Our game averaged between 100 and 300 CCU during peak hours before may 2nd.

Funnily enough, our new users per source graph looks almost identical to ruddev’s if you overlay them:

Observe the huge dropoff as of May 2nd, and our usual trend being broken

Our impressions also took a massive hit on the 2nd:

Unique plays per source dropped on the 2nd:

DAUs took a huge hit on the 2nd:

If you look at these stats and compare them to our retention KPIs, you’ll notice our retention has largely remained the same. Only our acquisition and discovery has been majorly impacted, which objectively shows that user acquisition has been made worse:

Drastic algorithm changes like this need to be made in lock-step with the developer community. Suddenly harming businesses because “we wanted to remove a social friends sort” is simply unacceptable. We’ve been asking for better communication with discovery for years now. Last year, I made this post which was largely supported and responded to by the developer community:

It went unresponded to until I poked someone in slack, and nothing really came of it.

In yesterday’s AMA, Roblox responded to my feedback about discovery with… a non-specific answer that didn’t address issues like the one in this thread.

Please address this issue. I am tired of fighting for something that should be an automatic given on a platform that needs businesses to survive.

OutlookG · June 9, 2023, 12:38am

Reposting from here: Creator Roadmap (Pt. 2 of 4) - Grow and engage your audience - #54 by OutlookG

Whatever change was made to discovery about 6 weeks ago has severely damaged our games while boosting our games in our genre without any clear reason as to why.

Our games CCU has pretty much been chopped in half despite none of our key metrics changing one bit (playtime, retention or monetization), while other games in our genre with much worse metrics are being promoted more then us.

This game here has much more players then us despite our playtime being 100% better then theirs (6 minutes vs our 12 minutes), how does this make sense?

Why are we in particular are being kneecapped while other games with worse statistics are being promoted more then us?

Brian1KB · June 9, 2023, 12:42am

Super well put together post man, it does a great job of showing the issues that I think a lot of small studios are having. I’ve experienced very similar behaviour new visitor wise with my own game, and I’ve included a new visitor chart just so you’ve got some more data to work with. My game has particularly started to experience an extreme dip as of changes made seemingly within the last couple of days.

We very much need to start seeing some announcements made in regards to algorithm changes, as at the moment it is very much infeasible to run a small business on the platform due to the sheer unpredictability of monthly revenue as a result of these sporadic, unexplained changes.

Game link if it’s any use to you: SkyWars 🔥 [BY VOXELS!] - Roblox

OneEDM · June 9, 2023, 1:13am

Wanted to pitch in here and while currently I do not have a large enough game to be able to show graphs that show definite changes, I want to further add my input and opinions in this matter as both a passionate developer and player of Roblox.

First, the Roblox discover page is awful, and does a terrible job at surfacing small/ medium sized games. This attributes to games such as @ruddev_ethan’s Bad Business and countless others from surfacing to players who would enjoy.

Take a look at my recommended for you. Nothing about this is personal, I do not enjoy any of these games and these same games are recommended to almost everyone. Roblox’s continued efforts and claims of personalized recommendation are completely false.

Alright, you might say, but I’m sure Roblox has some amazing curated categories to alleviate this poor personalization.

You would be wrong.

Let’s look again at the discover page:

MANY OF THE SORTS CONTAIN THE SAME GAMES, THESE TWO SORTS ARE LITERALLY IDENTICAL.

Not only are these games already extremely successful, Roblox is literally wasting space when other games could be there. It still shocks me how bad this problem is.

Oh yes seeing what premium players is totally very useful.

Go check the discover yourself and see how many places the same games like Doors, BloxFruits, and Pet Simulator appear. Sometimes it can be more than 5.

All of this ranting to say, yes, there is a fundamental flaw in Roblox’s algorithm and game discovery systems. Roblox pushes the same big games and supresses those who are trying to work their way up.

Roblox Repeated Games for no Reason Bingo:
DOORS: 5
BLOX FRUITS: 7
Pet Sim X: 5

CashyMone_y · June 9, 2023, 1:23am

The above posts are pretty comprehensive, so I’ll just reiterate that I have the same problem as the people above, I went from around 1k ccu to barely holding 300 ccu in my game, Exploding Mayhem, as of a few weeks ago.

I also just wanted to say that Bad Business is a solid game, I hope things get sorted out for you!

OmegaInvictissimus · June 9, 2023, 2:23am

My game only has six million visits, but algorithm changes have massively reduced the amount of new players it receives. The amount of new players joining, which was sustained for about two years, has dropped by about seven times exclusively due to algorithm changes.

apenzijncoolenleuk1 · June 9, 2023, 4:52am

Scary results indeed!

I’m wondering what games are shown instead. If it’s the smaller games, I would be tempted to say that’s a good thing! But only if every large game is affected equally by this.

If it’s the larger games that are shown, that’s just bad…

realOmlet · June 9, 2023, 6:10am

As someone who had only small games dependant on the discovery system my impressions dropped to… 10 whenever someone tries to even search for my places or my friend’s using the keywords they can search for hours just to never find any of them… not sure how exactly this now works but it results in displaying only the top earning/popular in practice. I think that bad business actually got hurt bcuz whilst it’s not a small game it’s not a top game anymore either so it just gets thrown in the dead games section just like everyone else. Roblox in general recently has only bias towards bigger creators by limiting the access to distribution to smaller devs, notifying bigger devs about downtimes on private social media groups exclusively, updating the search tool rn etc… I wonder how’s UGC doing lol since the marketplace is also messed up. Also this change is quite old but it looks like they keep on reverting and readding it with slight adjustments since ppl r reporting this for quite a while… just now this is too harsh
EDIT: NOT ROBLOX TRYING TO COVER THIS UP Introducing Exact Match Search for Experiences another edit: this change is still here and my visits count and playtime dropped by 100% last week…

Alkan · June 9, 2023, 8:29am

One of my games also started behaving differently almost overnight right around the start of May as well.
What was weird is I had started making biweekly Friday updates, and it was doing well for my game’s popularity. Then one update(nothing out of the ordinary or changing any fundamentals) just flops on May 5th.

DevAndero · June 9, 2023, 8:59am

I think the algorithm has been problematic for a while now, one of my games started rapidly dropping off around February last year despite key metrics being relatively stable. Went from around 5-10k concurrent to like 100 and at this point it’s getting 10 on a good day.

clxr · June 9, 2023, 9:21am

I’m thankful that I’m not the only one with this issue. We launched a new game on April 21st. It boomed for 2 weeks, then around that first weekend of May it completely collapsed. What WAS averaging 5.5K players has collapsed and is barely averaging 2K.

I’m at a loss for words because I spent three years working on this game, sacrificed a lot and to see it falling flat because of an algorithm change discourages me from working on this platform anymore.

Maxwell_Edison · June 9, 2023, 1:07pm

I’m looking at my game’s statistics and I’m (either fortunately or unfortunately, based on your perspective,) unable to tell if this is impacting me. My game had extremely bad metrics at the beginning of the year, but where as most people say April was fine and May started this issue for them, I had a pretty mediocre April and my home impressions started ramping back up to normal as May progressed, being slightly lacking now in June.

I think my game is simply very ‘Spiky’ when it comes to this sort of thing in general though, so I doubt I’m particularly relevant in this case. I’ll keep an eye on my home impressions either way.

edit: My metric graphs on the “New users per source” page. Unlike most posting here, my statistics for May recovered back to pretty much normal (March having higher stats for other reasons as pointed out already)

ByGoalZ · June 9, 2023, 1:12pm

Same thing here. My Game has been doing pretty well for a while, growing daily.
Then it rapidly declined and kind of died within a few days.
I do understand that games can not be on their “peak” forever, but how in the world can the algorithm just drop games that quick without any changes to key metrics or changes to the game/designs? There is literally no reason.

This needs to be adressed in order to keep devs on this platform and make a living off it.

2jammers · June 9, 2023, 1:17pm

I barely use the discover page, and play the same game which averages around 200 players. But I never knew it was this bad.

Maxwell_Edison · June 9, 2023, 1:25pm

This is an incredibly valid observation. The fact Roblox arbitrarily decides what genres our games are outside of our control is bad enough - resulting in developers not even knowing which categories we have a chance of being in - But it has always been annoying that the most mainstream games are the only ones we’ll see on our front pages, even if we do not have any interest in those games, and even a basic personalization algorithm should know better than to recommend us the same 5-10 games over and over and over.

Roblox should simply not allow a game to show multiple times on this list - period. Choose the one that best fits a game, and put it in. I don’t need to see DOORS on literally every single game category on the front page just because it’s popular. If I see it once, anything more is redundant and not great for anybody, even the owners of these popular games, which do not benefit from extra impressions within the same page.

2jammers · June 9, 2023, 1:40pm

There should be some button inside of the three dots on the game page, something like “Do not see this game again” and it will not show up in the discover page.

Maxwell_Edison · June 9, 2023, 1:41pm

This is entirely beyond the point being presented, though. “Everyone should ignore the popular games” is not going to solve the issue of Roblox’s algorithms being somewhat un-optimized for actual discovery.

samuelagent · June 9, 2023, 2:50pm

Can this not be attributed to the growing competitiveness of games in the roblox development sphere? As roblox grows, games will have to overcome the competition

Noble_Draconian · June 9, 2023, 3:27pm

Nope. If you look at all the examples, games that were doing fine for a while (in my case, around a year) suddenly got hit at the same time. It’s 100% a problematic algorithm change.