I don’t usually write large posts and specially don’t enjoy writing in such a critical tone as Roblox usually does a great job at solving issues that arise with the platform.
Unfortunately, today that is not the case with Discovery. This has been a downhill battle for a long time now, and despite some changes over time being negative to players, developers and businesses, it has never been as bad as it is now.
With the current direction Discovery is taking, I fear sustaining my business in Roblox will no longer be possible in the future, for reasons outside my control.
After reading many many posts about Discovery and searching the forums for posts regarding Discovery, it looks like this sentiment is shared by many other incredible developers who rely on the platform for their lifehood. As such, as you may understand, this is a very serious topic, and cannot be disregarded the same way a change to the Marketplace, for example, theoretically could be.
1. Why Discovery is so important
There are very few Roblox core components that are actually destructive to our income. Allow me to explain;
The website and engine are obviously vital to a developer’s success on this platform. The existence of our experiences and our ability to continue our workflows rely on these. However, our workflow can be adjusted if must be. The change to how Audio is shared is an extremely demonstrative example of this. Sure, it was game breaking for some experiences and surely wasn’t taken easily by a lot of developers… but it didn’t dictate “game over”. Truth is, as much as some changes might not be appreciated as much as fantastic other news, they have never seriously caused the doom of our experiences and presence on the platform. Outages may happen, but those are usually temporary (and almost never too damaging thanks to all the tooling provided nowadays) and aknowledged as non-intentional.
Discovery is one of the very rare, few mechanics that can be completely destructive to our experiences.
Due to the way Roblox is built as a platform (which I love, don’t get me wrong, and I’m sure is the reason many of us are on it), most experiences player find will be through their Home page or the Discovery page. This is dictated mostly by the algorithm, with a few exceptions such as friends joining an experience or sponsorships.
This approach works great when done correctly. When the “Recommended for You” sort was added, many developers saw a massive uprise in new user flow. This was great not only for us, but also for your average player.
Me personally, I tend to enjoy playing my own experiences, and due to being a long time Roblox player, still ocasionally play some Roblox experiences. The Recommended sort was surprisingly useful. One of my main problems as a player was always to find an experience I actually enjoy, and the Recommended sort gave me almost like some sort of “menu” for that. Looking back, this was probably the best Discovery had been at since.
Now, the algorithm is non-sensical, using metrics that promote low quality and low effort spam, measures that will eventually, if not already, be reflected on our income.
Without a proper flow of new users to our experiences, there is not much we can do as a developer that would save our experience. Without new players joining, any and all efforts are essentially rendered useless.
Before, if new user flow was decreasing, it was usually a clear sign that some metrics need improvement (mostly Retention and Session Time). This worked fine, because developers had to “step up” and get their experience in a better standard. This is now a pointless attempt. Changing metrics hardly affects the experience’s performance. In fact, having great metrics is not enough to keep your experience in the algorithm anymore.
2. Why can’t you just obtain those players yourself?
I have noticed this is mainly the most used argument as to why Discovery should not be as much of a worry as it has been becoming.
New users can be obtained through many different ways. With Roblox, those are mainly the Home page, the Discovery page, through friends, advertisements and sponsorships. Outside you may rely on influencers, such as YouTube content, livestreams, TikTok, etc. (among other not-as-used
methods)
For the reasons already outlined and that I will be expanding on further in this post, the Home and Discovery pages are no longer reliable. This leaves us with friends and paid boosting methods (advertisements and sponsorships).
The unfortunate truth is that neither of these are a viable option anymore.
Roblox has been A/B testing and slowly rolling out a new Friend Activity label feature where the Friend Activity sort is replaced with a gray label that shows up on other sorts of the Home page.
The problem with this, despite the obvious lack of a like/dislike ratio and player count (thus preventing the player from picking an experience from its face-value quality rating), this presents more problems for the player. What friend played this experience? Are they still playing it? When did they play it?
As counter-intuitive as it may seem, this actually disencourages the player from touching the experience. As a developer, this has a double-negative effect; the player is more likely to not join, and less experiences are being displayed overall.
This has been reported before, but alas, no response.
This brings us to the next and final remaining option: advertisements and sponsorships.
Now, obviously, Roblox grows more competitive each year that goes by, considering more people join the platform. This is not necessarily a bad thing, and could actually be good for the average player. This is actually a self-regulated system, which works great for its purpose. Pay more for your advertisement, expect more impressions relative to those who pay less.
The problem is that advertisements and sponsorships reach way too little people to provide any advantage to using them.
Before the wave of advertisement and sponsorship changes, and specially before sponsorships even existed, advertisements were incredibly powerful. I’d argue even too powerful. You’d see advertisements plastered everywhere on the website and as such was not a great user experience. Since then, advertisement locations have been reduced on the website, with them appearing in less places and not being so “in your face” as they used to be.
The current locations are perfect and do not require change. The issue is the lack of impressions.
Using advertisements, despite the extremely competitive environment, was a powerful way of boosting your experience’s market reach. However, these were cut down by more than half recently, rendering them nearly unusable for most experiences.
Sponsorships are a tricky topic. They used to serve their purpose, perhaps due, in some extent, to their novelty factor. I haven’t used them much so I will not comment a lot on them (feel free to share your story with sponsorships below this post), but from the little experience that I have with them, they are extremely inefficient. In my case, a single impression costed me 7 robux instead of providing a return. (it was a loss to run sponsorships, and this on an experience with a ARPDAU of 11)
With all this considered, relying on Roblox for new users is becoming unfeasible.
This leaves us with options from outside the platform, arguably the best way to hold yourself a good flow of new users that is much more independent from external factors.
It is essential to build a strong community and a large following, and influencer content certainly helps. However, we need new users, for the reasons described above. Without new users, our existing playerbase eventually will fade. (this is a certain fact, most will eventually stop playing, at different rates depending on the experience, but is just how it goes)
New users are met with a lot to take in if they are new to the platform. As such, a good part will stop playing within the initial minutes. This is not a bad thing and does not necessarily reflect bad user direction. It is simply a part of the first-time experience of most players. If they stay, great!
It’s also important to stress how much more difficult it is to obtain new users from outside the platform than it is through the algorithm. With the overwhelming amount of people playing Roblox every day, the algorithm will undeniably passively reach more impressions than you can outside the platform.
Back to the Discovery page - the “Recommended for you” sort is probably the most important way of obtaining players. This is where players will naturally be inclined to check for experiences on, as they (should) tend to cater to their interests.
When you join Roblox for the completely first time, this is what you see for the sort:
And that’s okay! Those are all big experiences, probably with great metrics…
… but this is the same sort on my 10 year old account:
Perhaps the sort is broken. But it’s been this way for quite some time now, with seemingly no change. With something as important as discovery, you’d expect more seriousness in fixing big issues with it.
With all this in mind, it is still undeniable that Roblox provides your experience with more users in-platform than you can out-platform.
Here’s an example of Unique Play Sessions Per Source from an experience with a community and reach outside the platform:
Despite the out-platform presence, Home is still nearly 2x as important as out-platform sources.
And sure, it is important to establish a community outside the platform for more reach, but an out-platform presence should not start dictating your experience’s survival when it hadn’t before.
3. Discovery is currently incredibly broken
Roblox has been making a huge effort to shift Discovery to be more player-driven for some time now.
It is true that Discovery has lacked immensely on this topic in the past, and this is probably part of the reason as to why so many of the changes made recently have been made. I believe this is with good intention, but the current direction is arguably counter-productive.
Let’s take this case analysis as an example:
This is an experience that received a massive update at the end of December of 2022, effectively reviving it. This experience is not new by any means, having existed since 2020. This prompted a massive algorithm spike in early January, which the experience managed to convince a good amount of the playerbase to keep returning until around February. When the values stabilized, that should have been it. What happened instead is that impressions were nearly completely cut off, with the experience receiving nearly no new players despite being updated and having better metrics than ever before. This is a non-sensical approach to organical growth management and has effectively caused the death of this experience. (used to have 2k CCU, now only has 20~)
This is not a unique case and has been reported many times. In fact, it’s such a pressing matter now that a bug report post about this has become one of the few Website Bugs post with more than 100 replies. I understand that the lead of Experience Discovery has replied to this last post, but the reply was to the sort of “we won’t do anything about it”, considering the arguments provided in the post and the overall tone conveyed.
Let’s step back a moment and understand the problem. Great metrics do not matter anymore. You will lose players.
There are many examples of experiences with great metrics, arguably the best in the platform even in some cases, that are still gradually losing more and more algorithm impressions every day. This doesn’t really make much sense. Sure, the experience may be getting older, but it is also getting better - and for a new player, it is still a completely new experience.
In the opposite hand, (relatively) new experiences are being pushed to the front of the algorithm, despite having worse metrics. Don’t get me wrong, I am all for new experiences getting their chance to shine. The point I am trying to make here is that, ignoring whether it would be fair or not to do this, it is incredibly harmful for existing experiences that new possible competition is receiving more attention by Discovery than your experience does, even though your experience has better metrics. This leads to an effect where players flock to the new experience and reduce your market share drastically, to the point where the new experience now has more players than you do. New experiences should only
receive more algorithm ranking than your experience if they provide better or similar metrics, but never worse. This sounds dystopian, but is already happening. This leads to sorts looking like this…
Back to advertisements and sponsorships; with Roblox’s growing market, the price required for the amount of impressions you’d get before has increased exponentially. For established developers, this isn’t that much of a worry, but for new developers it is. Roblox is well aware of this (the difficulty to get your experience out as a new developer)
and have been providing new experiences a lot of help in this matter through the use of the algorithm. A little boost is fine, but right now the algorithm favors new experiences too much.
It doesn’t make sense to do it like this as a business. At the end of the day, this is negative for the Roblox platform, not just developers. Having less recognizable experiences, with more incentive to spam new experiences, leads to an overall lower quality of experience in the platform, which may prompt users away and cut down user spending.
It seems as if Roblox is proud of this considering the amount of times new experiences hitting top sorts is mentioned in keynotes and presentations. This is something that does, however, not make much sense. More new experiences hitting top sort isn’t necessarily a good nor bad thing. It can mean multiple different things. Is it equality of opportunity Roblox is aiming for? If so, simply having a temporary boost and providing the developers with tools to expand their playerbase (such as advertisements and sponsorships) should suffice. There shouldn’t be positive discrimination, specially not in something like Roblox.
4. The Discovery metrics used for the algorithm are non-sensical
Before I begin, my source for this section is the Experience Discovery’s lead’s reply to the algorithm changes post and my own personal experience and analysis of experience analytics.
Roblox’s way of gauging a good ranking for an experience is currently the following:
- Retention: percentage of users who return (day 1 and day 7 retention).
- Engagement: time spent in an experience (session time).
- Monetization: percentage of users who invest in your experience (payer conversion and ARPPU).
- Relevance of the experience and its metadata (i.e. images, title, and descriptions) to the user (impressions to plays).
Before 2021, Discovery seems to have mainly relied on the first two (Retention and Engagement). This was fine but didn’t promote monetization-heavy or overall well-monetized experiences. Roblox introduced Monetization as part of the algorithm, and to this respect it was an improvement.
The way it is done, though, is not the best, and actually causes some problems.
You see, payer conversion includes all DAU. This means that even with a high ARPPU, if you have a really high DAU, your payer conversion is going to take a hit.
Payer Conversion being part of the algorithm incentivates the use of misleading tactics such as product spamming and intrusive popups. It is easy to artifically inflate Payer Conversion - just prompt a product in the player’s screen every time they perform a critical game action and haven’t purchased it already. It works, but it is incredibly harmful to the player’s experience. Experiences like these are usually frowned upon… so why is the algorithm promoting those? ARPPU is a great metric because it is not so easily influenced by these biases and still demonstrates if the experience has good monetization.
From personal experience, high payer conversion rewards the experience in the algorithm a lot.
As for Relevance, this seems to be a recent thing and the main culprit behind the drastic impression cuts some experiences have been seeing. This seems to rely on impressions to play… which… doesn’t make sense?
Let’s take a step back and analyze the discovery mechanic.
When you go to your Home page, you have different sorts below “Continue” (where you have your recently played experiences) where you can quickly find new experiences to join. If you do not find any of these interesting, you can go to the “Discover” page to find many more.
There is a “Recommended for You” sort that targets Relevance.
As a player, if I do not click on the Recommended for You sort, there is a good chance that I am not looking for an experience I’d usually play. Why is it that the only options I have listed to me are something that Roblox believes is relevant to me (and possibly other players)? Perhaps I just want something different.
I find myself closing Roblox to play other games instead due to lack of options in the Discover page.
Additionally, impressions to play favor the practice of clickbait immensely, and actively hurts different experience genres. For example, the horror genre tends to have a darker style, and that should be fine. However, since that also tends to not be clicked on as much, this leads to the algorithm ranking the experience lower. This is bad for developers, and bad for Roblox’s ability to provide the player with valuable options to play.
The algorithm present right now needs to be rethought. As it currently stands, it actively goes against Roblox’s attempts to age up the platform, harms developers and steers players away from the platform.
5. Discovery transparency with developers is severely lacking
Since 2019, there have been 0 announcements with “discovery” in the title, 0 announcements with “discover” in the title, 0 announcements with “algorithm” in the title, 2 announcements with “home” in the title, 3 announcements related to Search and Discovery with “search” in the title and 10 announcements with “sort” in the title.
That’s a total of 15 Discovery related announcements in a 4 years time. For comparison, simply searching the word “physics” (considering this only includes a fraction of all posts related to this category) already yields 27 results.
It doesn’t appear, from a developer perspective, that our concerns are being heard. Every time something is done with discovery that is protested about, it has almost never changed. The small amount of replies to our concerns from the discovery team have been disencouraging, with the latest reply by the Experience Discovery lead almost (truthfully, I apologize for the straight forwardness) feeling like a slap in the face.
Despite all the apparent Discovery problems, I have faith Roblox can remedy the situation and completely reverse the current path it has taken.
The real point to be made here is that it should had never gotten this bad.
Simple communication in the form of announcements would have gone a long way in allowing Roblox to measure impact and developer feedback, preventing Discovery from getting to a point where businesses are actively losing players and revenue.
I understand that it is difficult to scale a discovery algorithm for a platform as big as Roblox, but this is actually one of the reasons as to why it is extremely vital to include developers in the conversation. For many reasons outlined in this post and more, Discovery is a key component of our experiences’ survival, and it is vital developers are heard and informed on big changes to it.
More transparency is critical to Discovery’s success.
