Roblox’s Algorithm Changes Are Killing Our Game

Bug Reports Website Bugs
#105

Hi Developers,

I am Ivy_poisond, and I lead Experience Discovery. We heard your recent feedback on the shifts in experience exposure on Home, and understand your pain. There are some things we can and will improve. In the meantime, we also want to give you a better understanding of how we’re approaching recommendations, and what actions you can take to improve the discovery of your experiences.

How we are evolving recommendations

In the RDC 2022 Keynote, we shared our vision for evolving Home to improve the relevancy and diversity of creator experiences that are showcased. Our goal is to connect every user with the best experience for them and foster a diverse and successful creator ecosystem by connecting every experience with the most relevant audience. The way we do that is through iterative experiments.

What happened recently

Many of you shared graphs showing positive and negative changes to your impressions between mid-April and May. Thank you all for doing that and for your patience as we reviewed and analyzed the data.

We discovered that the combination of Discovery experiment launches and creator-side changes led to a lot of unique stories. The great news is that more experiences than ever are getting recommendation traffic from the home page, which has improved engagement for the overall developer community and improved the diversity of content offerings for our user community. More specifically, we saw a 25% increase in the average number of experiences with at least 100,000 monthly recommendation impressions from March to June. This means that while some experiences have received less traffic in recent months, more experiences are now finding their target audience. As we strive to enhance our algorithm further, we aim to grow our Roblox user community, their retention and help all experiences reach these target audiences effectively.

Improving our transparency & communication

We will be sharing better information on how changes to discovery may impact your experience. Our Creator Analytics team is actively working to increase transparency and actionability so that you have a clear understanding of how you can make changes to your experiences to get more distribution. For example, we’ve been testing an insights product that highlights when your experience is below benchmark on key metrics that may impact its discovery. We’ll also share the Creator Analytics roadmap to get feedback on actionability of the insights that we plan to provide.

How you can take action to improve discovery of your experiences right now

Our recommendation engine connects users to new experiences with consideration for real-time signals tied to these four key areas:

  1. Retention: percentage of users who return (day 1 and day 7 retention).
  2. Engagement: time spent in an experience (session time).
  3. Monetization: percentage of users who invest in your experience (payer conversion and ARPPU).
  4. Relevance of the experience and its metadata (i.e. images, title, and descriptions) to the user (impressions to plays).

To help you influence and make improvements against the signals mentioned above, take a look at your creator analytics dashboard to see if these metrics are above or comparable to similar experience benchmarks. In the coming months, we are also working to improve our acquisition analytics page to make the relevance metrics more transparent.

We recognize that Home is an important discovery channel for experiences like yours to acquire and grow users on Roblox. This is a journey and we will continue to iterate and improve how experiences are showcased to users. We are committed to keeping you informed along the way about specific changes and our plans in the coming weeks.

Please continue to share your feedback with us as we work on evolving content discovery and distribution.

Thank you.

36 Likes
Discovery does not respect the community, and desperately needs to change
Roblox game kicked out of search results - killed the game
#106

This really boils down to a classic Roblox post of “We hear you and we’ll do better” that’s too vague and doesn’t really address/ignores the issue brought up by hundreds of developers.

Firstly, Roblox is currently promoting WORSE PERFORMING GAMES above our game, actively destroying retention with Roblox by promoting games with worse metrics.

image
image1716×807 224 KB

We have higher stats in every single metric (that we can tell) then all of these games (our games in red boxes), and yet we are being punished while they are being rewarded. This is similar across the other genres our games are in and there is no clear explanation for this. This isn’t fair, acceptable or makes even any sort of business sense. The algorithm just promoting worse games that actively hurt Roblox and make less money.

Secondly, while the creator analytics team is fantastic, this doesn’t excuse the fact that this change was done without at least notifying creators that this was happening, instead we’re left wondering why our player count suddenly dropped overnight without any explanation, thinking that we did something wrong. All your achieving by doing this is placing additional stress on developers.

Thirdly, why does Payer Conversion have a influence on the algorithm in the first place? ARPPU and ARPDAU are much better indicators of how well a game has monetized. This especially demotes games with a large international audience such as ourselves who don’t have very good conversion rates but have genre leading ARPPU and ARPDAU metrics.

Specifically, ARPDAU should be the main metric the algorithm is tracking, as this most adequately shows how well an experience is monetizing and how much money its extracting out of users. Infinitely better then ARPPU and Conversion rate.

For example, our game 1% Win Obby has benchmark leading metrics in every single category from what we can tell (up to 100% better then the benchmark in some cases), including payer conversion and yet, we are not being promoted to as many users as these other games, which I can guarantee have worse metrics then us.
image

Peoples careers live and die by this algorithm, and right now it really feels like the algorithm is spinning a lucky wheel and giving players to whoever it lands on, no matter if they have awful metrics or not. I implore you, revert to the algorithm back in April, that algorithm was working perfectly fine.

When some of the biggest developers on the platform complaining, its clear something has to change.

34 Likes
#107

While I fully support and appreciate improvements in the recommendation algorithm, the recent changes have been an objective step backwards.

I’m going to avoid talking about home, as that boils down to a largely subjective argument of if it’s worthwhile to promote smaller experiences with worse engagement in the long run.

To focus on one issue, genre searching, I don’t think there is any data roblox can find or provide to suggest the recent changes have been positive.

When searching a genre, e.g. Obby or Tycoon, Roblox should promote the games with the best engagement, retention or monetisation. Not CTR.

If a user is searching for an Obby, they are going to find an Obby to join, so sorting by CTR is both pointless and incentivises foul practices such as constantly changing icons to trick users into thinking a game is new. This practice has increased since the recent algorithm changes and results in poor quality engagement.

Genre searches should return to the old algorithm as not only is it fair on developers who work hard to improve their core metrics, but it also provides objectively better engagement for Roblox as a whole.

11 Likes
#108

Thanks for the response. I’m concerned about this part:

This is Pet Simulator X:

image
image1374×288 44.2 KB

Our session time is also top 10%. Relevance is something we cannot control since the game is well established and has been out for almost 3 years now.

Despite this, Roblox is trying to hide our game:

image
image978×462 28.9 KB

Impressions are down to almost 1/3 what they were 2 months ago.

Part of me wonders how this is affecting Roblox’s bottom line. To my knowledge, all the top experiences have taken a pretty substantial hit… despite having a very high ARPPU. As a profit driven platform, why would you try to kill your biggest & most consistent revenue sources?

My theory is Relevance is now the key metric - which explains why my games page and recommendations are suddenly flooded with brand new games with clickbait-esque icons and titles. It feels like YouTube.

I shouldn’t have to guess. This was obviously a big change. More clarification would go a long way.

40 Likes
#109

I recently launched my game, Farmstead: Farmstead [BETA] - Roblox. About a week after a somewhat quiet release, Farmstead was put on the ‘up and coming’ sort. I reached 10,000 concurrent players shortly after, with fantastic revenue, engagement, and retention metrics. The only metric that was performing poorly was payer conversion. I second @OutlookG’s statement about payer conversion, and I would love to know why this is considered instead of a combination of ARPPU and APRDAU.

Just 6 weeks after hitting 10,000 concurrent players, Farmstead is now averaging about 600 concurrent players. In the last 4 weeks, I have seen my ‘Home’ impressions drop from 30,000,000 to 4,000,000, roughly halving each week.

I am not trying to blame the ‘algorithm’ for the shortcomings of my game; there are definitely areas I can improve, but seeing exposure drop so drastically in such a short period of time while having 30+ minute session times, 90+ percentile ARPPU and ARPDAU, 90+ percentile D7 retention, is incredibly disheartening.

I sincerely hope retention, engagement, and monetization (with a greater focus on ARPDAU and ARPPU rather than payer conversion) become the key signals to discovery. “Impressions to plays” as a key signal only encourages “clickable” games and does not correlate with overall user experience, platform revenue, engagement, or any other useful metric to developers or the Roblox platform as a whole.

13 Likes
#110

Can you link me to the new insights panel because I cannot find it

#112

It’s in a/b testing right now, you most likely don’t have it yet. I’ve been told full release is soon.

1 Like
#113

Disappointing, predictable & generic reply which ignores the points brought up by many developers on this post.

In regards to “what happened recently” – increasing the spread of players throughout games, provided players are finding a more “suitable” experience (e.g. one they will spend more time in) is entirely reasonable. What is not entirely reasonable is making such a large change, and not making any announcement whatsoever. This impact would have been obvious to those working on this change, and it should not have gotten to the point of this post needing to be made about it. Roblox is, as usual, oblivious & ignorant to the fact that people base their livelihoods upon the earnings of these games.

In regards to “improving transparency & communication”, this is the first proper statement that I’ve seen from the S&D team, it’s already extremely lacking when it comes to transparency & communication, and makes no commitments other than “we’ll do better”. Is it a stretch to simply detail the plans there? Does it involve announcements when large changes are being made that affect X% of creators? There needs to be some proper conditions for communication, and it can’t just be “we’ll do better”.

Additionally, the “insights” product you mention has already been brought up here in this post multiple times. You’ve got folks posting here who have insights far above the benchmarks. Their games are performing significantly worse than they were before. They have insights which outperform other games, and those games have 10x their players. These insights are fundamentally not useful if they are not connected to the behaviour of the algorithm.

Lastly, I suppose it was very much to be expected that you would state to work on retention, engagement, monetisation & “relevance”. Once again, this is already known information that has been brought up in this thread – there are games here that outperform other games in all of those areas, in matching genres, that have less players than them.

I’m sorry to be so pessimistic, but this is just what I expected from Roblox when I heard a reply was coming today. Some changes that I propose that I believe would remedy some of these issues

  • A single, algorithm score rating on the analytics page for your game – currently, you can double your monetisation, or even session length, and see no impact in new visitors. It’s not clear how long the algorithm takes to update, or why this is, and seeing a single “combined” score to know when you’ve done something that will have a positive impact is key.

  • An explanation of the surge behaviour that many new games seem to have been experiencing, to describe this, many new games at launch are experiencing tens of thousands of concurrent players, and then dropping down in to the low hundreds after a month. Can this behaviour be explained? Additionally, if the concept is that the algorithm sends large numbers of players to determine metrics surrounding the game, once the algorithm has “made a decision”, is it then possible for games to actually improve their metrics & be sent larger numbers of new visitors? e.g. I have increased my game’s metrics significantly, and have pretty much only gone down.

  • A commitment to notifications about algorithm changes when you expect for there to be a large impact on the creator community at large – a change of this size should obviously be included

  • An open channel between folks on the S&D team to shed light on issues with particular games, that are doing well in terms of all of their metrics, that don’t have a clear reason to be doing poorly. There’s simply no good reason why this shouldn’t be possible. The Analytics team have done incredibly well by starting a Guilded server, and being clear & open in communication, and the S&D team should be doing the same.

20 Likes
#114

Right now, it just seems like luck. As if the algorithm randomly selects a game to boost for a couple of days, only to drop it later.

Metrics don’t seem to help in any significant way. It’s basically a gamble…

I guess this gives the small games a chance, but I think we just need some consistency.

2 Likes
#115

It looks like the algorithm is trying to hide bigger games and promote random games with no players and with clickbait thumbnails and titles. Which looks like a pretty dumb move by Roblox (if that’s what is actually happening) because from what I know Roblox is trying to appeal to older audience and everything that I see in the discovery page is obbys and games like “Punch X simulator” which at least in my opinion won’t help Roblox.

2 Likes
#116

I also lost like 4x concurrent players as well as impressions due to the change even though the game did pretty well in many aspects listed.

1 Like
#117

They should just revert it to whatever worked best before and quit experimenting with this stuff when it can affect peoples income drastically.

I’m holding off on sponsoring my game simply because of what i’ve heard here so let’s hope they make up their minds and fix this mess.

2 Likes
#118

Respectfully, have you folks read the replies above? Many of us that have been negatively impacted by these experiments are above the average for the KPIs. There are games with worse KPIs being recommended over games with better KPIs.

Honestly, I could write an entire wall of criticism here for the non-answers that we’re consistently given, but I think Brian1KB’s reply, korky5000’s reply, BuildIntoGame’s reply and OutlookG’s reply all sum up what I have to say.
We ask for genuine dialogue yet we’re still given vague & closed-off answers while being told “we promise to do better”. We’ve been told this for years with no actual meaningful change.

As I mentioned previously:

Developers need open communication about these experiments. We are not getting this. Our businesses are being negatively impacted by the arbitrary unannounced surprise changes that Roblox has been constantly applying to S&D. If we cannot reliably run a business on this platform, we will move to a competitor.

While I thank you for the response, it doesn’t address any of our issues & dodges our questions.

18 Likes
#119

I agree, it just feels random, check this out

This game has 12k players and its litterally just free models, I dont see a reason to come back to this game every day

Even by putting myself in the shoes of a kid, I still can-t find a reason to enjoy the games roblox promotes me

#120

This is the 90-day new user acquisition for Specter, a game with > 300M visits. Around May you can see that the users from the home page (top blue bars) got a drastic cut of the overall percentage. We went from 2k-4k new users from the home page a day to less than 1000, and this drop happened in a few days. This didn’t recover back past 1000 until the 20th of June. We made no changes that would have hurt the game like this. In fact, we released an update at the beginning of May that should’ve boosted our stats.

image
image1000×406 12.4 KB

It’s always seemed to us that Specter is designed in a way that the algorithm does not like. We aren’t a P2W simulator, and the algorithm doesn’t like that. We have a lower conversion rate than most games partially due to the genre (horror). We have a generally older audience than a lot of games.

Yet for being one of the few games on the platform that is catering to a slightly older audience, we are constantly punished. Teleport failures are almost certainly affecting our session time and retention which hurts our algorithm rankings. Almost all of our analytics show us in the top 3%-6% of “similar experiences”, yet we receive essentially no promotion from the algorithm. Luckily we are a very social game and get a lot of new users from friends joining.

Developing on this platform is such a chore, we want to make ambitious projects. We want to create games for older audiences. Yet every feature that lets us do this actively works against us based on the stats. The algorithm seems to just favor whatever gets clicks the fastest, and that’s usually neon-colored clickbait. We’ve had to adjust our icons to be brighter to match the rest of the website, even though it’s a horror game. It honestly makes no sense, but there is no genre sorting, so this is the only option if we want to get players interested.

12 Likes
#121

Simply, make like 100 games a year. That’s what I do :sunglasses: They all suck but with this new algorithm I am sustaining like 100 players on every one of them and it adds up. All these old developers are one hit wonders and now are mad that roblox doesn’t want to reccomend the same games from years ago even if they are good because people get bored not of the game but roblox as a whole if its the same games OVER and over to put it simply.

1 Like
#122

Below are some of the insights for one my games that recently picked up about 100 concurrent players.

image
image1500×216 32.9 KB

While I am grateful I even have 100 playing on my game, it is demoralizing to see games have games of my own and games of others with far worse stats being pushed by the algorithm much more. Until about just a week ago, the game did not have a single bad insight. This makes me honestly feel like the algorithm is just selecting games by complete random and throwing them out there.

If you ever wonder why spam games are a problem on Roblox, this is why, and trust me they have become very tempting to make. Spam games are awarded while other games that have had lots of hard work put in to them, with good stats are punished.

5 Likes
#124

I don’t want to attack you personally, but I believe this is terrible advice to be giving people.

The terrible thing about it is that I’m pretty sure it works… Low effort games shouldn’t get rewarded over high effort games. That simply isn’t fair.

But I guess this is the current state of things, similar to the mobile gaming industry. Best we can do is hope Roblox adjusts their algorithm for the better.

7 Likes
#125

Algorithm changes, exclusively, have taken my game from getting 250,000 visits a month for a few years straight to suddenly dropping to only 50,000 visits a month. Hoping for a fix, or even a reversion to these updates Roblox has been making.

3 Likes
#126

Considering how many people are affected negatively by these updates I hate to say this but I am actually a big fan of the algorithm changes.

A few years ago I released a complete kit for building games themed around public transport. You could just grab it and you had a fully playable game with buses and trams.

I’m not gonna lie, a bunch of pretty well-made games emerged based on the kit. But I’ve also seen a few very low-effort, cash-grab ones too. And because of Roblox’s wacky algorithms, they gained a lot of recognition that they definitely did not deserve.

Today, when I look up the relevant keywords I struggle to even find them. And when I do find them, they seem to have only a fraction of the playercount that they used to have.

Interestingly enough, my own game about public transport seems to be entirely unaffected by these changes. And it’s back to being the top of it’s genre!

4 Likes