Games Are Routinely Thrown Out Of the Algorithm

Bug Reports Website Bugs
#1

Reproduction Steps

Not sure how exactly to trigger this issue, I think it’s definitely related to games and places created after the end of 2022, my other older projects don’t have this issue.

Expected Behavior

I expect when I sponsor or update my game, for our position in the algorithm to remain relatively unchanged, assuming nothing about our stats has changed.

Actual Behavior

Everytime I’ve updated the game, or began running sponsored ads, our impressions due to recommendations drop to practically nothing.

This seems entirely inexplicable since prior to the action, it was growing consistently. In the case of updates, we don’t see any significant change to our time played, crash rate, retention, or monetization. Usually we see improvement since that’s the goal of the updates.

I’ve also regularly checked social media and categories on all platforms to ensure our traffic is coming from recommendations… it was.

This last update on the 13th of Feb, we improved our monetization yet again AND created an icon that out performs all our previous icons for clickrate. I ran some sponsored ads, and the game immediately died from over 1100 CCU to 500 overnight. The game still hasn’t recovered.

It’s really discouraging since it seems that working towards improving a game or advertising it actually impacts it negatively. As a developer, what am I supposed to do?

Issue Area: Roblox Website
Page URL: Throw Stuff - Roblox
Impact: High
Frequency: Constantly
Date First Experienced: 2023-01-01 00:01:00 (-05:00)
Date Last Experienced: 2023-02-16 00:02:00 (-05:00)

12 Likes
#2

FWIW, since late 2022, all of my experiences have had super weird traction with the algorithm - with or without ads. All of them have varying creation dates.

Not saying this is identical to what you’re describing in your post, but something relatively-recent with the algorithm has made previously-sensible historical trends incredibly variable. It seems inconsistent as to when experiences are shown and when they’re not shown to users. DAU and MAU change so frequently I don’t even know what my baseline is anymore.

Hard to tell what is and isn’t connected without transparency about algorithm changes, but figured I’d mention it as there’s a fair bit of parallels.

5 Likes
#3

I have experienced the exact same issues since late 2022, too. There was definitely some big algorithm change last year that as far as I am aware was never announced.

2 Likes
#5

Thanks for the report! We’ll follow up when we have an update for you.

6 Likes
#6

We are still suffering from the same problem on a regular basis. Out game “Poop Simulator” Poop Simulator [GLOBAL CHAT] - Roblox, despite having 2.8 million visits now lingers way down the search ranks. The game even appeared in a KreekCraft video before.

Searching for either the word “poop” or “poop simulator” in browser or app returns an unusual set of results, listing games with far few visits and far lower rating much higher when compared to our game.

FYI, I am unable to create a separate topic in the Web Site bug section for our problem despite being a very frequent contributor to this forum.

1 Like
#7

I feel like this has been happening to games I work on as well, specifically Zombie Rush and Catalog Cabin

1 Like
#8

Ditto, there’s zero consistency to search results, with even experiences that have less likes, favorites, default icon, last updated over a year ago will outrank my game for it’s main keyword. There’s been at least 20 bug report threads surrounding the search result/discovery issues with no resolution. Theres really no point to spend money on ads when your games search visibility will be arbitrarily tanked. It’s really weird that more transparency isn’t happening for this over a year long issue, with countless reports. It also directly leads to developers taking their ad spending OUTSIDE roblox, or just stopping it all together.

4 Likes
#9

Did you complete the recent Roblox survey? It included a question about game discovery, which will hopefully highlight the issue to them.

Survey link if required:
Short survey to improve the developer experience

Things went markedly downhill from about August 2022 it seemed. We’ve got quite a few games that have been finished and have high ratings, that disappear down the ranks. We made a game last year called Lost Friends that is almost undiscoverable. When searching for the game name, it is currently around position 2262 at which point the algorithm starts to return full matches on the search term of “Lost Friends”, along with a few other games with the same name. If that is not an example of how broken the search function is, I don’t know what they will accept as evidence it is entirely unfit for purpose.

4 Likes
#10

This issue may be related to: Discovery Documentation on Roblox docs

Roblox Discovery Documentation
Roblox Discovery Documentation1242×860 126 KB

Unless I am reading this incorrectly, it seems every time we update the name or description of our experience the discoverability is completely removed for up to 14 days. At which point, it gets reintroduced. So if you are updating every week and always changing the name or description, your game may have zero discoverability.

2 Likes
#11

This issue has been troubling me for many months now, and so I find myself frequently thinking about it. During these times, I have thought of a few things which might contribute to the problem. Of course, I have no insider knowledge of how the Roblox experiences algorithm works and why it broke so much last year, this is just my considerations.

I first question if it is possible Roblox is throttling R6 games. As an active community member on Roblox since 2011, I personally cannot stand R15’s art style and animations. For this reason, I leave my experience on R6. It is no secret that Roblox would like us to move to using R15. Therefore, I wonder if it is possible that at some point, Roblox decided to begin favoring R15/Rthro/3D Clothing compatible experiences over R6 ones in search results. If this is the case, it is something we need to be informed about so that we may have the most tools at our disposal to determine the future of our experiences.

I secondly wonder if it is possible experiences are sorted based on their use of new and beta Studio options. I remember in the past when Roblox decided to throttle experiences that did not opt to use Filtering Enabled, when that was optional. I wonder now if it possible it filters experiences not using things like adaptive timestepping and deferred signal behavior. My experience continues to use fixed timestepping and immediate signal behavior because these options work best for me, based on performance and bugs. I’m afraid adaptive timestepping especially is far too buggy to use. Perhaps more likely, I wonder if it is possible Roblox throttles experiences that are not compatible with Rthro, 3D clothing, and MeshPartHeadsAndAccessories. After all, some percentage of Roblox users pay for 3D clothing and presumably expect it to work in all experiences. Perhaps voice chat being enabled or disabled could also be taken into account, but I would hope not due to the restrictive server size limits for using voice chat in your experience. I will also include in this category the new TextChatService. Again, if it is true that Roblox throttles the search algorithm based on these options, we should be informed so that we can use that information to consider making changes to our “legacy” experiences in the future.

Thirdly, I wonder if it is possible that all of the translations of an experience’s title and description are used in the metadata for the experience search algorithm. What I mean by that is if you have your experience translated from English into Spanish, the text from both of those translations is used by the algorithm. I have to say that I hope this is not the case. The Roblox text filter simply does not work with non-English languages, and I am completely unable to translate my experience’s description into other languages due to this. Anyways, I suppose this option is a stretch, but I have at times felt paranoid and that me translating my experience’s information may actually hurt it on search results, or at least for English search results. Reassurances that this is not the case would be very valuable for me.

This topic is very near and dear to me because until mid-to-late 2022, my experience was in the top 10 search results for “army” and since then, has been drastically bumped down, currently to number 409. This has, as you can imagine, dramatically reduced the amount of new players joining my experience, easily reducing it by a factor of 10. Bizarrely, even if I assume every single point I’ve listed above is true, there are still dozens of experiences with less players, less likes, less favorites, and less visits listed far above my own experience.

It is my sincere hope, sir, that something in this post does some small effort in helping you all deduce the issue that I and many other users have experienced since last year. If something I listed here is correct and does affect the algorithm, for example using R6 or voice chat, etc. I would ask that you please let us know so that we can make informed decisions about what technologies to use in our experience. Please keep us up to date and know that I am very grateful for any work put into solving this issue. It would be my great honor and pleasure to help in any way I can, if necessary.

image
image1442×655 46.8 KB

4 Likes
#12

The same problem occurs when sorting clothes in the catalog (Problem with sorting clothes in Roblox catalog) and the problem appeared around November 2022. Something has changed in the sorting algorithm and clothes are sorted in a strange way and their popularity no longer matters. In addition, when you enter 2 words in the search box, results for any of the words entered are displayed instead of narrowing down the search results. Often, clothes that have nothing to do with the search topic and have only one or even none of the search words in the name or description are displayed much higher than popular clothes that perfectly match the search topic. My sales have fallen several times and have been at a lower level for half a year.

1 Like
#13

Wow, that’s really dumb.

What if you put your update logs in the description? Or just simply announce an update in your game title?

#14

Oh wow. We were seeing your game when we were launching ours (Terramon) and it was averaging 2k+ CCUs. Now it’s under 100.

We have noticed this sort of randomness in the algorithm while our stats seem very good judging from the analytics page.

It seems as if the engineers at Roblox are constantly running experiments and tweaking the algorithm without any notice. That’s really inconsiderate since for some of us this is our main source of income.

All of the Roblox spokespeople keep telling us the algorithm ranks based on engagement (and slightly monetisation) but they do not explain what does that mean. Is that average time per player? Per session? Do users that come from ads even count here?

We saw a spike of users in mid January (that was based on search and home discovery) that peaked at around 1.5k CCUs. Since then we have made countless improvements and updates but we never managed to see as much traction from the algorithm since then.

The lack of information on Roblox’s part is very frustrating. A lot of times we cannot really tell if we made a bad change in our game or the drops are due to some discoverability issue (bug or feature). Kudos to the analytics team that makes it easy for us to see how many players come from Home discoverability and also the performance of our game pages and icons. However, this just shines light on how artificially and non-organically the algorithm operates. We have retentions in the upper 90 percentiles, but that doesn’t seem to matter as the users keep decreasing.

The nail in the coffin is that in the past week the sponsors stopped yielding good results for us. Suddenly the CPP doubled for no apparent reason.

PS: I always keep track on games that are recommended to me and on our game’s page. We all seem to grow and shrink at the same time. It’s obvious it’s the discovery algorithm, but it’s impossible to prove from the creator’s end. Most likely it’s a feature not a bug, but at least there should be some documentation, announcement or general comms when platform defining changes like these, happen.

3 Likes
#15

I’ve been blown away by the amount of detail that YouTube provides creators on the visibility of content and how users find your content including which search terms are yielding the results.

Perhaps if we had the same sort of clarity from Roblox, us devs would be a little happier and could see why we end up with this massive decline in visibility of our games.

It is frustrating when we have an engaging game, high visit counts, reasonable ratings, excellent CVR on Monetization and yet we are penalised by being placed below other lower rates games. It makes no sense.

2 Likes
#16

Wow I had no idea you got penalized for changing your description or title. I’ll keep this in mind for the future for sure.

#17

Today I noticed that this experience, which is privated and has zero visits, zero favorites, and zero likes is showing up on search results before my experience which has zero six million visits. There is clearly a serious flaw somewhere in the search algorithm for experience searches that needs looking into. My friends are wondering if my experience was somehow flagged for botting or some other issue, but I don’t know how I would be able to check for that or get it resolved. Hoping for an update on this issue soon. Thanks

#18

I can personally confirm this as well, my game that has released on 1st January of this year has had an exponential growth period which suddenly stopped out of nowhere, we did no updates that could have halted user’s experience during this time.

Ever since Febuary our playercount weekly would raise by a good 100 players or so, however one day it reached the peak and has went on a downward decline no matter how much ads I run or sponsors (My ads get around .4% CTR)

Notes:

Purple: Few weeks after we have been actively running ads
Red: Sudden peak of the game’s interest and the sudden downfall
Yellow: An update we did recently addressing few bugs that caused a bit of bad reputation

image
image1278×660 82 KB

As you can see the sudden growth and toll do not make sense together, the game one day got promoted by the algorithm and one day spat back out, the game has an almost 80% rating so it’s not that our game is bad, quite the frank many actually enjoy it and join our community too.

Our optimization at current time may not be the best, but it doesn’t really matter as we have not changed anything about how the game runs ever since January

It would be really great if Roblox could at least keep their algorithm not being biased towards games that earn good amount of money for the website that COULD earn way way more if they were still kept in the algorithm.

2 Likes
#19

It seems this has just happened to our game as well. I tweaked the game description to include “Like the game for a chance to unlock new codes!” and our discoverability and CCU has drastically dropped.

image
image1103×1035 37.9 KB

I tweaked the description this last Sunday. I’m at a loss for words. I worked on this game for nearly three years and with one subtle change Roblox has completely removed us from discoverability.

3 Likes
#20

You definitely should include the missing section below that you forgot to screenshot in that statement unless it was recently added.

image
image808×186 16.4 KB

It looks like this means that it only applies if your title or description was moderated. The second statement says that the game would be able to repopulate back into search results after originally being removed for violating Roblox community standards. I look at plenty of games on the front page (plus a few of my games I updated the description for very recently) and they are still able to properly be searched for after they update the description or title.

It seems that there could be issues with just updating metadata in general that could be causing things to go weird. I hope that this does get addressed and properly worked on as if what I said was wrong about it being only applied to moderated titles and descriptions; this definitely would go against what Roblox said at RDC about wanting to make sure that discovery is improved.

2 Likes
#21

That is not what it says. It specifically states that any metadata changes may cause up to 14 days of waiting to repopulate discovery (regardless of moderation, that isn’t mentioned).

That is because it clearly states “up to 14 days”. Basic logic implies that it could be instant, 1 day, 2 days, or 14. So it seems that in most cases, your experience will be repopulated to discovery quickly, but in some cases it will not be until up to 14 days pass; and in extreme cases, it never repopulates (due to moderation, which is unclear if that includes text getting filtered or not) and doesn’t even warn you that you won’t repopulate discovery.

I don’t recall if this was there when I read it, but it’s irrelevant to users who weren’t moderated and that’s what this post is about.