Just wanted to give an update on how the game is going - it has been growing, and quite literally everything roblox is telling us in the new “Insights” panel is that the game is doing great, and that our metrics are top of the line. We keep getting stars, hearts, top 3% 4% 5% in many categories…
But once again, discovery on Roblox’s end looks terrible and seems to have had nothing to do with our growth. We have had substantial increases in the Home, Friends and Other categories, while discovery from search (which includes the Discover pages according to official creator docs) has somehow decreased by about 20% during the same time period.
That game is new and didn’t exist before the changes this post is referring to. Given your metrics you should also be getting a lot more than the algorithm is currently giving you.
Hey everyone, so sorry about the trouble. This seems much more serious/detailed than a bug report. I’ve made sure we’re aware of this internally, and we are. Not much more I can do on my end. Godspeed.
Starting in March almost every single one of my games went down like 90% of my usual concurrent… I’m cashing out 10% of what I used to every month for THREE YEARS. Please fix this
I released a game towards the end of April, and had good success within the algorithm. Within a matter of days we were climbing very nicely, however right at the end of April we saw a sharp decline. Our stats (retention, ctr, playtime) barely changed when we blew up, and some improved, however we saw a rapid decline in impressions the following days without any reasons behind it.
We have started to grow slowly again, and have noticed that this growth aligns with some of our monetization metrics. However, given these high metrics (well above any benchmarks), the algorithm should be giving us many more impressions than it is right now.
(Take a look at the graphs below for our recent impressions)
I have released similar games in the past with worse stats, and had more success. What was the motivation behind these new changes? In my opinion, the previous algorithm was effective at what it needed to do, and distributed players well across top experiences.
As @BuildIntoGames mentioned previously, Pet Simulator X, with what I assume has good monetization analytics has seen a decline in impressions recently. I would be interested to hear if that has correlated directly with your ARPPU and ARPDAU or any other analytics.
Our game primarily relies on D7/30 retention/stickiness for it’s survival, but given our stats we should be seeing a lot more new users than we currently are.
Take a look at our past 60 days of Impressions/New Users:
We’re currently relying on natural growth, and constant updates (more frequent than we had planned) to keep the game alive. I’m not sure who made the decision for this change, and what motivated it, however as many others have already mentioned, this could cost top experiences millions of dollars as we head into Summer.
There is clear evidence of this change having a bad impact on the platform.
Any ETA on a change/update to the algorithm and discovery or at least some information on what influences it now?
I am Ivy_poisond, and I lead Experience Discovery. We heard your recent feedback on the shifts in experience exposure on Home, and understand your pain. There are some things we can and will improve. In the meantime, we also want to give you a better understanding of how we’re approaching recommendations, and what actions you can take to improve the discovery of your experiences.
How we are evolving recommendations
In the RDC 2022 Keynote, we shared our vision for evolving Home to improve the relevancy and diversity of creator experiences that are showcased. Our goal is to connect every user with the best experience for them and foster a diverse and successful creator ecosystem by connecting every experience with the most relevant audience. The way we do that is through iterative experiments.
What happened recently
Many of you shared graphs showing positive and negative changes to your impressions between mid-April and May. Thank you all for doing that and for your patience as we reviewed and analyzed the data.
We discovered that the combination of Discovery experiment launches and creator-side changes led to a lot of unique stories. The great news is that more experiences than ever are getting recommendation traffic from the home page, which has improved engagement for the overall developer community and improved the diversity of content offerings for our user community. More specifically, we saw a 25% increase in the average number of experiences with at least 100,000 monthly recommendation impressions from March to June. This means that while some experiences have received less traffic in recent months, more experiences are now finding their target audience. As we strive to enhance our algorithm further, we aim to grow our Roblox user community, their retention and help all experiences reach these target audiences effectively.
Improving our transparency & communication
We will be sharing better information on how changes to discovery may impact your experience. Our Creator Analytics team is actively working to increase transparency and actionability so that you have a clear understanding of how you can make changes to your experiences to get more distribution. For example, we’ve been testing an insights product that highlights when your experience is below benchmark on key metrics that may impact its discovery. We’ll also share the Creator Analytics roadmap to get feedback on actionability of the insights that we plan to provide.
How you can take action to improve discovery of your experiences right now
Our recommendation engine connects users to new experiences with consideration for real-time signals tied to these four key areas:
Retention: percentage of users who return (day 1 and day 7 retention).
Engagement: time spent in an experience (session time).
Monetization: percentage of users who invest in your experience (payer conversion and ARPPU).
Relevance of the experience and its metadata (i.e. images, title, and descriptions) to the user (impressions to plays).
To help you influence and make improvements against the signals mentioned above, take a look at your creator analytics dashboard to see if these metrics are above or comparable to similar experience benchmarks. In the coming months, we are also working to improve our acquisition analytics page to make the relevance metrics more transparent.
We recognize that Home is an important discovery channel for experiences like yours to acquire and grow users on Roblox. This is a journey and we will continue to iterate and improve how experiences are showcased to users. We are committed to keeping you informed along the way about specific changes and our plans in the coming weeks.
Please continue to share your feedback with us as we work on evolving content discovery and distribution.
This really boils down to a classic Roblox post of “We hear you and we’ll do better” that’s too vague and doesn’t really address/ignores the issue brought up by hundreds of developers.
Firstly, Roblox is currently promoting WORSE PERFORMING GAMES above our game, actively destroying retention with Roblox by promoting games with worse metrics.
We have higher stats in every single metric (that we can tell) then all of these games (our games in red boxes), and yet we are being punished while they are being rewarded. This is similar across the other genres our games are in and there is no clear explanation for this. This isn’t fair, acceptable or makes even any sort of business sense. The algorithm just promoting worse games that actively hurt Roblox and make less money.
Secondly, while the creator analytics team is fantastic, this doesn’t excuse the fact that this change was done without at least notifying creators that this was happening, instead we’re left wondering why our player count suddenly dropped overnight without any explanation, thinking that we did something wrong. All your achieving by doing this is placing additional stress on developers.
Thirdly, why does Payer Conversion have a influence on the algorithm in the first place? ARPPU and ARPDAU are much better indicators of how well a game has monetized. This especially demotes games with a large international audience such as ourselves who don’t have very good conversion rates but have genre leading ARPPU and ARPDAU metrics.
Specifically, ARPDAU should be the main metric the algorithm is tracking, as this most adequately shows how well an experience is monetizing and how much money its extracting out of users. Infinitely better then ARPPU and Conversion rate.
For example, our game 1% Win Obby has benchmark leading metrics in every single category from what we can tell (up to 100% better then the benchmark in some cases), including payer conversion and yet, we are not being promoted to as many users as these other games, which I can guarantee have worse metrics then us.
Peoples careers live and die by this algorithm, and right now it really feels like the algorithm is spinning a lucky wheel and giving players to whoever it lands on, no matter if they have awful metrics or not. I implore you, revert to the algorithm back in April, that algorithm was working perfectly fine.
When some of the biggest developers on the platform complaining, its clear something has to change.
While I fully support and appreciate improvements in the recommendation algorithm, the recent changes have been an objective step backwards.
I’m going to avoid talking about home, as that boils down to a largely subjective argument of if it’s worthwhile to promote smaller experiences with worse engagement in the long run.
To focus on one issue, genre searching, I don’t think there is any data roblox can find or provide to suggest the recent changes have been positive.
When searching a genre, e.g. Obby or Tycoon, Roblox should promote the games with the best engagement, retention or monetisation. Not CTR.
If a user is searching for an Obby, they are going to find an Obby to join, so sorting by CTR is both pointless and incentivises foul practices such as constantly changing icons to trick users into thinking a game is new. This practice has increased since the recent algorithm changes and results in poor quality engagement.
Genre searches should return to the old algorithm as not only is it fair on developers who work hard to improve their core metrics, but it also provides objectively better engagement for Roblox as a whole.
Impressions are down to almost 1/3 what they were 2 months ago.
Part of me wonders how this is affecting Roblox’s bottom line. To my knowledge, all the top experiences have taken a pretty substantial hit… despite having a very high ARPPU. As a profit driven platform, why would you try to kill your biggest & most consistent revenue sources?
My theory is Relevance is now the key metric - which explains why my games page and recommendations are suddenly flooded with brand new games with clickbait-esque icons and titles. It feels like YouTube.
I shouldn’t have to guess. This was obviously a big change. More clarification would go a long way.
I recently launched my game, Farmstead: Farmstead [BETA] - Roblox. About a week after a somewhat quiet release, Farmstead was put on the ‘up and coming’ sort. I reached 10,000 concurrent players shortly after, with fantastic revenue, engagement, and retention metrics. The only metric that was performing poorly was payer conversion. I second @OutlookG’s statement about payer conversion, and I would love to know why this is considered instead of a combination of ARPPU and APRDAU.
Just 6 weeks after hitting 10,000 concurrent players, Farmstead is now averaging about 600 concurrent players. In the last 4 weeks, I have seen my ‘Home’ impressions drop from 30,000,000 to 4,000,000, roughly halving each week.
I am not trying to blame the ‘algorithm’ for the shortcomings of my game; there are definitely areas I can improve, but seeing exposure drop so drastically in such a short period of time while having 30+ minute session times, 90+ percentile ARPPU and ARPDAU, 90+ percentile D7 retention, is incredibly disheartening.
I sincerely hope retention, engagement, and monetization (with a greater focus on ARPDAU and ARPPU rather than payer conversion) become the key signals to discovery. “Impressions to plays” as a key signal only encourages “clickable” games and does not correlate with overall user experience, platform revenue, engagement, or any other useful metric to developers or the Roblox platform as a whole.
Disappointing, predictable & generic reply which ignores the points brought up by many developers on this post.
In regards to “what happened recently” – increasing the spread of players throughout games, provided players are finding a more “suitable” experience (e.g. one they will spend more time in) is entirely reasonable. What is not entirely reasonable is making such a large change, and not making any announcement whatsoever. This impact would have been obvious to those working on this change, and it should not have gotten to the point of this post needing to be made about it. Roblox is, as usual, oblivious & ignorant to the fact that people base their livelihoods upon the earnings of these games.
In regards to “improving transparency & communication”, this is the first proper statement that I’ve seen from the S&D team, it’s already extremely lacking when it comes to transparency & communication, and makes no commitments other than “we’ll do better”. Is it a stretch to simply detail the plans there? Does it involve announcements when large changes are being made that affect X% of creators? There needs to be some proper conditions for communication, and it can’t just be “we’ll do better”.
Additionally, the “insights” product you mention has already been brought up here in this post multiple times. You’ve got folks posting here who have insights far above the benchmarks. Their games are performing significantly worse than they were before. They have insights which outperform other games, and those games have 10x their players. These insights are fundamentally not useful if they are not connected to the behaviour of the algorithm.
Lastly, I suppose it was very much to be expected that you would state to work on retention, engagement, monetisation & “relevance”. Once again, this is already known information that has been brought up in this thread – there are games here that outperform other games in all of those areas, in matching genres, that have less players than them.
I’m sorry to be so pessimistic, but this is just what I expected from Roblox when I heard a reply was coming today. Some changes that I propose that I believe would remedy some of these issues
A single, algorithm score rating on the analytics page for your game – currently, you can double your monetisation, or even session length, and see no impact in new visitors. It’s not clear how long the algorithm takes to update, or why this is, and seeing a single “combined” score to know when you’ve done something that will have a positive impact is key.
An explanation of the surge behaviour that many new games seem to have been experiencing, to describe this, many new games at launch are experiencing tens of thousands of concurrent players, and then dropping down in to the low hundreds after a month. Can this behaviour be explained? Additionally, if the concept is that the algorithm sends large numbers of players to determine metrics surrounding the game, once the algorithm has “made a decision”, is it then possible for games to actually improve their metrics & be sent larger numbers of new visitors? e.g. I have increased my game’s metrics significantly, and have pretty much only gone down.
A commitment to notifications about algorithm changes when you expect for there to be a large impact on the creator community at large – a change of this size should obviously be included
An open channel between folks on the S&D team to shed light on issues with particular games, that are doing well in terms of all of their metrics, that don’t have a clear reason to be doing poorly. There’s simply no good reason why this shouldn’t be possible. The Analytics team have done incredibly well by starting a Guilded server, and being clear & open in communication, and the S&D team should be doing the same.
It looks like the algorithm is trying to hide bigger games and promote random games with no players and with clickbait thumbnails and titles. Which looks like a pretty dumb move by Roblox (if that’s what is actually happening) because from what I know Roblox is trying to appeal to older audience and everything that I see in the discovery page is obbys and games like “Punch X simulator” which at least in my opinion won’t help Roblox.
Respectfully, have you folks read the replies above? Many of us that have been negatively impacted by these experiments are above the average for the KPIs. There are games with worse KPIs being recommended over games with better KPIs.
Honestly, I could write an entire wall of criticism here for the non-answers that we’re consistently given, but I think Brian1KB’s reply, korky5000’s reply, BuildIntoGame’s reply and OutlookG’s reply all sum up what I have to say.
We ask for genuine dialogue yet we’re still given vague & closed-off answers while being told “we promise to do better”. We’ve been told this for years with no actual meaningful change.
As I mentioned previously:
Developers need open communication about these experiments. We are not getting this. Our businesses are being negatively impacted by the arbitrary unannounced surprise changes that Roblox has been constantly applying to S&D. If we cannot reliably run a business on this platform, we will move to a competitor.
While I thank you for the response, it doesn’t address any of our issues & dodges our questions.
This is the 90-day new user acquisition for Specter, a game with > 300M visits. Around May you can see that the users from the home page (top blue bars) got a drastic cut of the overall percentage. We went from 2k-4k new users from the home page a day to less than 1000, and this drop happened in a few days. This didn’t recover back past 1000 until the 20th of June. We made no changes that would have hurt the game like this. In fact, we released an update at the beginning of May that should’ve boosted our stats.
It’s always seemed to us that Specter is designed in a way that the algorithm does not like. We aren’t a P2W simulator, and the algorithm doesn’t like that. We have a lower conversion rate than most games partially due to the genre (horror). We have a generally older audience than a lot of games.
Yet for being one of the few games on the platform that is catering to a slightly older audience, we are constantly punished. Teleport failures are almost certainly affecting our session time and retention which hurts our algorithm rankings. Almost all of our analytics show us in the top 3%-6% of “similar experiences”, yet we receive essentially no promotion from the algorithm. Luckily we are a very social game and get a lot of new users from friends joining.
Developing on this platform is such a chore, we want to make ambitious projects. We want to create games for older audiences. Yet every feature that lets us do this actively works against us based on the stats. The algorithm seems to just favor whatever gets clicks the fastest, and that’s usually neon-colored clickbait. We’ve had to adjust our icons to be brighter to match the rest of the website, even though it’s a horror game. It honestly makes no sense, but there is no genre sorting, so this is the only option if we want to get players interested.
