Analytics: Recommendations Qualified Play Through Rate and Similar Experiences Benchmarks

[signal_zzz]

Hi Creators,

Today, we’re making two analytics updates to provide more transparency on how your experience is surfaced by recommendations:

1. Qualified play through rate to track impressions to qualified plays, which measures deeper engagement than plays.
2. Similar experience benchmarks to help you compare your KPIs with similar experiences that your users also play.

Qualified play through rate

Last year, we added a home recommendations conversion rate chart that measures the percent of users who play your experience after viewing it in recommendations.

Since then, we’ve updated our recommendations system to better surface experiences that drive deeper engagement and reduce clickbait.

Specifically, in recommendations for home and search, we’re now using a signal called qualified play through rate that tracks the percent of users who have a qualified play in your experience after viewing it in recommendations. A “qualified play” refers to a user’s intentional engagement with your experience, as opposed to accidental clicks or quick bounces.

With this in mind, we’ve replaced the recommendations conversion rate chart with a qualified play through rate for recommendations chart in Acquisition analytics. We’ve also added benchmarks for this metric based on similar experiences that your users also play. If your qualified play through rate is below benchmark, consider using our new funnel events to optimize your new user onboarding and testing new thumbnails that match your latest content updates.

Benchmark with similar experiences that your users play

Last year, we launched benchmarks to help you compare your experience with others based on our internal genre data. However, Roblox users often choose what experience to play across genres.

Today, we’re updating our benchmarks to be based on similar experiences that your users also play (with genre as a fallback).

We also use similar experiences (along with other signals) to decide what experiences are eligible to appear in recommendations, so these new benchmarks more closely reflect how our recommendations work.

To qualify for similar experience benchmarks, our model needs to find at least 50 similar experiences that have a large overlap in playtime and other signals with yours. You can see some of these experiences on your Experience Details page’s Recommended Experiences section:

Your similar experience benchmarks are updated daily. If you don’t qualify for similar experience benchmarks, you’ll still see benchmarks based on our internal genre data. We’ve added chart annotations to show when your experience transitions from genre to similar experience benchmarks:

New discovery documentation, explore mode, and a new experience overview

In addition to the changes above, we’re also excited to introduce:

1. New discovery documentation that explains in simple terms how our recommendations and other discovery channels work.
2. Explore mode to filter and view detailed breakdowns for any chart. To enter explore mode, click the expand button at the top right of any chart.
3. A new experience overview page (in testing) that will help you better understand what metric to focus on at a glance. We’ll share more about that in the coming weeks.

As always, let us know in the replies if you have any questions or feedback!

[HealthyKarl]

Sometimes I find new games through simple search parameters.

But what Roblox makes it hard for is when I type the exact Experience Name into the search bar, when their routine recommendation highlight temporarily expires, I am forced to scroll to find it. That seems wrong…

Hope that can be fixed.

[nicogamer12357 (ImNicoReal)]

Speaking of analytics, i cannot see enything on monitoring - performance, i used to be able to see the current active players but now i cannot, not even the analytics, it just says “You’ll see more performance statistics as your experience grows!”

[Qxual]

Sort of irrelevant, but are we going to get something similar for the Catalog/Avatar Shop? The algorithm doesn’t sort based on popularity/sales at least for 2D Clothing when searching for an item. It has been like that for the past year, a change would be great. Similar to the previous post about games, users would also have to scroll down to look for a popular clothing item when trying to search for its name.

[signal_zzz]

That’s weird. Can you link your experience detail page?

[signal_zzz]

Will pass feedback to search team thank you.

[bvetterdays (roman)]

This entire update sounds super cool. Looking through my own games, I don’t see the mini pop-up stating the benchmarks have been updated to similar experiences when I do believe there’s at least 50 other experiences available that could be used to gather this data. Any chance you guys could look into this?

Other than that, really loving the continued updates to analytics you guys always find a way to improve already-great features when it comes to the Creator Hub, so thank you

[HealthyKarl]

If there’s a Search Team.

Is there a Team that also influences the search results? Because I’d wonder how both communicate between each.

[VGVC2]

@signal_zzz

You can see some of these experiences on your Experience Details page’s Recommended Experiences section:

I’ve noticed in the past month or so, under my showcase game, actual showcase games (sometimes with similar vibes or aesthetics) are now shown in the Recommended section of the game page, that was a great change!

on Qualified Play Through Rate, this may change the whole game sort a lot, for the better! I have a good feeling there will finally be quality games being brought through the noise. (though I’m sure games with huge amounts of players despite being low-effort/having a very low rating will still remain prominent).

[signal_zzz]

If you hover over a data point on a chart with benchmarks, it should say the benchmark label whether that’s “similar experiences” “genre” or something else. We prioritize showing similar experience benchmarks if that’s available.

[Raki3dd (Raki)]

Hello, about the comparison with similar experiences, in the part that talks about the genre as an alternative, if for example you were playing a game with a horror genre, such as the Murder Mystery 2 game

When I leave the game, do other games that have the horror genre have to appear in the experience recommendations, or could I get other types of games that have a different genre?

By the way, I wanted to talk about something related to the use of Today’s Picks, several developers will agree with me on the misuse of this since Roblox is in charge of promoting some brands with this category, which is not fair anymore. that brands should play by the same rules as other developers.

I would recommend that so that this does not continue to happen, that developers have access to this type of promotion, which would compensate a lot and this would allow developers to play with the same rules that brands play.

[NiceDrift (TPC)]

Could we have charts for both genre-wide metrics and similar-experience metrics, instead of conditionally only selecting one of the two? Imo understanding the bigger picture of the genre is also important strategic information. It gives me a less objective understanding on how the experience is performing or trends occurring.

Also curious: if I push an update which jumps playtime by +20% and lands my experience in a different cluster, how dramatically will the similar experiences it’s compared to change? Because for instance I could suddenly see my game drop below the 50th percentile for that new set of comparisons despite improving engagement, rather than ranking above the 90th percentile like it used to be, in the span of 1 day.

I could see that being chaotic and unintuitive, especially for new devs, they might believe they made a mistake and revert the update?

[SnoutUpDev (SnoutUp)]

So… what happens if your game gets thrown into a benchmark range that is completely unreachable? Does that negatively impact recommendations? It’s very demotivating to see such charts.

[bvetterdays (roman)]

Right. Just to clarify, do these labels mean it’s still using the genre rather than similar experiences?

[C9OQI (Cooqi)]

@signal_zzz

A new experience overview page (in testing) that will help you better understand what metric to focus on at a glance. We’ll share more about that in the coming weeks.

I think I had this for a few days and then it was removed. It was really helpful and I was sad to see it go.

I CAN’T WAIT FOR THIS !!!

[nicogamer12357 (ImNicoReal)]

Creator Dashboard

It basically shows this. Nothing else.

[protagamestudio]

The Analytics changed in only one out of the seven experiences i have published.

Why is that happening?

[signal_zzz]

Which analytics are you referring to? If it’s the new experience overview page, we’re still testing it but hope to roll out to everyone in a few weeks. We’ll share a new devforum post when that happens.

[signal_zzz]

It looks like you don’t have enough users for us to show useful performance stats. Keep growing your experience and the stats should work.