I was looking to see if someone with more experience with the inscrutable Roblox algorithm could help me diagnose why my home recommendation impressions have seen asymptotic decline to near 0 over the course of months. After performing a linear regression analysis on these statistics, it seems the dominant predictive variables are W1 retention and yesterday’s home impressions (big surprise right?). Although, you can see how that wouldn’t be very actionable under the working assumption that W1 retention has dropped alongside drops in player counts (150 avg players to now 10) because this is a very social game, I don’t think I made the game that much worse by updating it (10.3% peak to 2.7% W1 retention today).
I understand that if anyone has gained any insight into the algorithm, they would probably be liable to keep it to themselves due to how precious the knowledge is. That seems to be causative for why there is so little community made information about the algorithm, although I imagine it is quite difficult to come upon anyways. I’ve been thinking of trying to get us to compile data so we can work out the key metrics which lead to home recommendations by doing some standard data analysis.
Anyways, I’ll post my graphs here. I understand medium and less so short term retention are my biggest issues, and that is where I’ve been focusing efforts. But I’d like to investigate the following questions:
What caused the asymptotic decline to zero despite in spite of updates to the game? Is it W1 retention drops or the natural course of the algorithm, something else?
Are these numbers what you’d expect? As in, does your game get more home impressions with worse stats than these or less home impressions with better stats?
Does Roblox tend to randomly “test” games by putting them on the algorithm? Is that liable to repeat again with this game? What causes these tests?
Hey, your game is doing really well currently.
Could you please describe how it happened?
We were releasing a new game and after a week of sponsors, we started receiving home recommendations. However, it keeps going down now. The stats are between the 50-90 percentile, mostly in the upper part so I’m a little confused…
This is due to being featured on Today’s Picks, which is a human curated selection of games. I’m very grateful for that exposure. But this is completely temporary. As for understanding what motivates Home Recommendation Impressions, I haven’t learned anything new since my last post. As you can tell by the lack of responses, other developers are not eager to share data and the algorithm is naturally mysterious as it is certainly an AI model.
As for what I do think I “know” related to your experience, is that when games get on the algorithm for the first time, there is an initial novelty bonus it seems to apply to give your game a nice launch. But after that the game will approach the stasis value of new players that you ought to be getting relative to your metrics. That’s what I figure goes on with us. And it is my current suspicion that 50-90th percentile benchmarks are to be interpreted differently. My big question is how well does a game that is perfectly 90th percentile in all metrics perform? If Roblox could let us know the benchmark for playtime in terms of 50th to 90th percentile and not in terms of ‘Top 10,000’ or ‘Top 2,500’ game, we would have greater perspective. My suspicion is that to do well, you should be well above 90th percentile in nearly every category. Being within the range is borderline acceptable. Because my suspicion is that a 90th percentile game is not very successful. You probably need to be in the top 1 or .1% to do very well for yourself, but we don’t see those percentile benchmarks. That’s my guess; but I know nothing.
It would help the effort here if you posted your statistics, particularly what they were right before you were added to the algorithm. You would help out people who have never been on the algorithm before see what they actually need to achieve to get their first burst of players.
