ohhh thanks! That helps a lot, i’ve always felt like the Qualified play-through rate is the metric I have least control over improving other than trying out a bunch of different thumbnails, but knowing that the initial onboarding has some influence too helps a lot!
We are trying to differentiate between brief visits and sustained interest with engaging game play.
You probably already understand but want to draw the distinction between visits, and engaged game play where users return to play again and also maybe with friends. The latter aspect drives increased retention on your game as you can see with various metrics available on the creator analytics dashboard. Signals outlined in the post are key factors in the new recommendation algorithm and positive strong growth in those signal will drive good discovery.
1 Like
Hi @maaz322, thanks for your feedback!
As a big tycoon enjoyer I understand your concern - the intention of this update is not to find only one specific kind of gameplay and push all users towards those types of experiences.
There are multiple signals that are used in conjunction. You don’t need to have every benchmark maxed out to be recommended - the new algorithm aims to help us elevate more different types of content and gameplay.
Here are some potential play patterns that can be measured against each other more easily now:
- fewer play days per week, but longer session times (capped at 60min per user per day)
- shorter session times but more play days per week
- medium session times but tons of coplay sessions
Of course, we always aim to improve the algorithm and will continue to test adding more signals that can help us recommend more high quality experiences across different kinds of genres and gameplay.
Hi @ph33sh, sorry to hear about this! (and sorry for late reply - I had this drafted and forgot to hit send 
)
Would you be able to send us a link to an experience with this issue?
It’s hard for us to tell from a general description, since many factors affect the impressions an experience receives (it doesn’t get picked up by any of the candidate generators, it gets ranked too low and users don’t scroll to see it, etc)
Just to set expectations, it will take some time for us to review this as there’s a ton of manual analysis involved. But we definitely want to ensure that the system is working as expected, and find out where there’s room for improvements!
I’ve been consistently in or above benchmark for every algo signal for the last few days, but I have been stuck at low impressions (10-50K) for 4 weeks now.
Recently, we’ve doubled the session time and retention to be above benchmarks for tycoons, but to no avail currently.
Is there a timeframe to this, does building the audience metadata or whatever impact this timeframe, etc?
any insight would help @skpm_plz
This at first seemed like a good update but after observing all my games. Unfortunately I have come to the conclusion that this update is very bad for people who make tycoons like me. I am a dev that has been MAJORLY impacted as I almost exclusively make tycoons and this update has impacted everything I do. All of my games have halfed in players right after the update and I have lost a significant chunk of my playerbase.
I also lost alot of revenue as a result.
I own many groups most notably being best tycoons studio with 2m+ members and salahdin with 350k+ members who have tycoons with hundreds and thousands of ccu.
Heres a graph from one of my tycoon proving such impact right after the update
My main tycoon group impacted as I have lost almost more than half of all my players and ccu is at an all time low comparing to the past year
Another group impacted
All the games under these groups were impacted 20+ tycoon games.
I feel extremely sad roblox would do this or an update would directly affect my players and revenue. I hope there is a fix to this very soon or it is big trouble for me.
5 Likes
Hi @webpotion,
March 29/30 was a weekend, so we expect to see a dip right after, going into Monday.
If you zoom out to a longer view, impressions should be more stable (i.e. in the same range, higher on weekends, lower on weekdays).
I picked a random tycoon from your group and impressions look stable over the last ~4 weeks, but of course I did not check all of them, so please let us know if there is a specific experience that sees this issue.
Hi @carbonscripts ,
Congrats on your launch and the improvements you’ve made!
The new algo signals are “7 day playtime” etc, so going forward it will take about a week to fully see the impact of changes.
I see that in the last month, you’ve run an ad campaign that’s kickstarted the algorithm exploration process (stage 4 in this post). At this stage, we normally see that impressions from Home grow to a peak, then settle somewhere lower than the peak that’s your new baseline.
As you’re still in the middle of stabilizing and the algorithm only changed this week, I’d recommend waiting another week or so to see where it lands at. (Also I see that you’ve made some thumbnail changes recently that seem to be helping - I’d continue to watch the impact of this too)
You probably already know this but just wanted to mention - Impressions are the top of the funnel so more is always “better”, but we prefer to index more on plays because if the each user gets the “correct” content immediately, then they won’t be scrolling as much.
P.S. I think you are doing a great job taking advantage of all our features. Thumbnail personalization, granular onboarding funnels, sponsored campaign - this all looks awesome 
1 Like
I don’t know to be honest if I recommend thumbnail personalization. Few days ago my QPTR of the only thumbnail I had reached 1.52% and it has been going downhill only after that reaching 0.81%. Yesterday I decided to do thumbnail personalization and added 1 more, the results were very bad. The old thumbnail had 0.5% while the new one 0.35%!! Also my Stats have been decreasing drasstically and I am not sure why. 4-5 days ago I had stable 18.5% D1 Retention for over 5 days, and now it is 9.5% or something. I am not sure if the new algorithm is messing up with the stats or I need to wait few more days for things to be good again?
Crazy Skidlets on Roblox
Hi @Kollector98,
Congrats on your launch as well!
From this screenshot we can see that the winning thumbnail for both 18+ and <18 segments was your original thumbnail (i.e. it is strictly better than the second one you’re trying). Usually we recommend giving the thumbnail personalization algorithm more time to explore, but in this case I think it’s pretty clear. I’d recommend adding more thumbnails to test with.
In general, as your impressions increase, we expect stats like QPTR to decrease (because it’s being shown to a broader audience). Based on your impressions stats I’d also bucket your experience into the “algorithm exploration” stage. Looking at your 30 day engagement and retention I don’t see any cause for concern yet.
I did notice the elevated D1 corresponds exactly to when you weren’t running a ad campaign. I don’t know what kind of campaign you have been running or if it is related, but we are continuing to iterate on our sponsored ad product as well to improve targeting and campaign objectives.
Edit: Wanted to mention that usually I like to look at the 30 day view instead of 7 day - it gives more information on whether a drop is a drop from a stable level, or if it went up then came back down
Hey, Thanks for the response however many of my games have seen a sudden drop after this update. I will provide another example below you can check out i also noticed my own games are not being recommended to me anymore… This means they wont be recommended as much to other players as before the update I used to check home recommendations constantly and saw many of my games there.
Below an example of one of my game shows extreme drop almost to 0, averaging 400 ccu then now 50 ccu averaging after the update.
4 Likes
This update will be very bad for alot of games on ROBLOX that has short gameplay like horror games, story games ETC. Just because they’re one-play time games it doesn’t mean ROBLOX shouldn’t put them in the algorithm. Just like most of the people watch a movie once and there’s no need to watch it twice just like @maaz322 said. Just because a game is unplayable it really doesn’t mean it shouldn’t be in the algorithm…
The old algorithm on the other hand, was great for long-term and short-term 1 play games. But now it’s only fair for the long-term progressive games.
5 Likes
We also had a significant drop in recommendations last day with no changes in the game. The game has good metrics on the Overview page. We don’t have a clue yet why this happened.
Game link: Planet Robinson 🚀🪐 - Roblox
2 Likes
Congrats on the launch of this game as well!
In the 60 day view, I see that you started to receive Home impressions on March 1, the algorithm did some exploration peaking on March 16 (around where this screenshot starts), and then tapered to today.
Since the new algorithm was only released this Monday (March 31), I don’t think the drop we’re looking at in this graph is caused by that.
This is my suspicion:
When you had fewer impressions, the algorithm first served it to a subset of users that were most likely to play. Your QPTR was stronger leading up to 3/10, and your D7 retention was stronger in that time as well. As the algorithm initial exploration continued, the metrics are starting to stabilize.
Unrelated to this experience’s stats, I was surprised by this being a benchmark.
My feeling as a user is “if you keep showing this to me and I don’t click on it (continues to be in Home Recommendations), I obviously don’t want to play it”. Currently we don’t have great controls for users to say “stop showing me this I don’t want it!!” but as we continue to improve Home, I don’t know if the test of “does it continue to show up” will be a good indication of whether content is being served to the correct audience.
Thanks for your feedback @kaanXBT.
As previously mentioned, there are multiple signals that are used together and a game doesn’t have to contain all elements to be recommended to users, e.g. a one-time play game that has a long playtime during that one time will be able to compete with progressive games with shorter session times.
We also used similar signals in the past, but now we have expanded to include more, specifically to capture more kinds of content.
We’re always looking to improve the algorithm - what are some measurable traits that you think make games with shorter gameplay good?
I don’t see the “Similar experiences in your benchmark”. My game also struggles to get enough home impressions to be successful despite solid metrics. Is this because the algorithm is struggling to find the right audience to recommend my game to? If so, will the new AI ads manager help fix that?
Hi @GonnaBoom ,
Since fluctuations do occur in general (based on user base, stats of other experiences you’re competing with, etc) I wouldn’t index too highly on one specific day yet, especially since impressions were not stable beforehand (they are growing!).
For reference, this is how I looked at your stats (also because I probably won’t have time to look at everyone’s individual experiences):
-
I looked at your graph and observed that impressions started increasing a little around 3/14 and a lot around 3/18.
-
I looked for something that could have caused this increase.
- Around 3/10 you released an update that seems to have greatly improved retention (congrats!).
- I looked for something that could have caused a decrease or stabilization.
-
Between 3/18 to 3/27 your QPTR was lower (before your newest update). This corresponds to the large increase in impressions. When impressions increase, we typically expect QPTR to decrease.
-
I was planning to look at your thumbnail personalization results but it looks like you just started a new campaign. Hope this one goes well!
Since many of the algorithm signals are 7-day signals, impression numbers can lag slightly relative to changes you make. For changes in impressions I usually take a look at a longer timeframe and see if anything changed.
But in this case I think it’s difficult to tell quickly(ish) based on the 1 day of data and currently the overall longer-term trend looks promising.
1 Like
Thank you for your reply. Let me tell you about thumbnails. Just a week ago when we released the update, I also uploaded a set of new thumbnails and launched a campaign. This is what the result looked like after a week.
Everything is fine in this screenshot. I decided to keep the two most successful thumbnails and replace the other three with new ones. And the next day I was woken up by the guys from my team with the news that our online numbers had dropped.
Checking the info I went to the thumbnails section and saw that they have almost no impressions or conversions in the game.
Since we didn’t change anything in the game itself, I decided that maybe my changes to the thumbnails were the reason for the drop in impressions. Just in case, I updated the thumbnails again. And at the moment this page is just zeros. I don’t know how often this page is updated, but before I thought it was showing actual data, and within half a day after the changes there should have been some numbers here.
I had planned to change the images in thumbnails regularly to find the most successful version, but now I’m afraid I’m going to break something.
Thanks so much for the insights!