I recently went through a few old games of mine to check some old analytic information and I noticed there were some inconsistencies between graphs and tables for qualified play through rate for recommendations. The game I was looking at had a sharp drop a while ago where the qualified play through rate dropped from ~2-2.5% for home recommendations to .5-.6% which prompted me to investigate some data so I could determine the cause. However when I was looking through different sources for home recommendation qualified play through rates I noticed they all showed different results despite the label for them being the same. The graphs are from qualified play through rate for users filtered for home recommendations, qualified play through rate (recommendations) and the table produced under the acquisition tab
All of these sources are for the same period of time (8 weeks) and arenât filtered for any specific people but they show wildly different results. I also manually went through the home impressions and qualified plays from home recommendations I got for those 8 weeks and calculated what the QPR for should be on my analytics page and got ~1.2% which still didnât match the .6% average it reports. Since the data sources are labelled exactly the same and should include the same data Iâm not sure where the inconsistencies come from.
Expected behavior
If the differences between the data is expected there should be some further labelling for clarity on why the results are different, currently they are just classified as âhome recommendationâ with no additional information so it makes it difficult for me to know why there is a large mismatch between the corresponding data sources
If the differences between the data is expected there should be some further labelling for clarity on why the results are different
You are looking at different metrics.
In your screenshots, the metric name is labeled differently and each column in the table says âUserâ. The tooltips for each metric identifies what the metric is calculating:
Qualified play through rate (recommendations) = Qualified plays divided by impressions for Home recommendations => That means this is on a per impression basis and the same user can have multiple impressions
Qualified Play Through Rate per User => This is per user, as stated in the metric name. The same user have multiple impressions but clicking through once in that day is counted as a play through.
Since users will often see the same experience multiple times in one day, the impression based number is much lower than the per user number
The analytics page currently doesnât really convey that well. There is no data source where you are able to look at pure home recommendation impressions, besides the area in thumbnail personalization which is locked to a week of data at most. All the other data sources I can see are only users with impressions. All the graphs that display data for home impressions are user based and directly above it as well. Additionally right above the QRP (recommendations) it says âEach signal is based on users who joined from Recommended for Youâ which sort of also misleadingly implies it is user based. The analytics page in general could use a lot of tweaks to enhance the clarity of it, especially since the documentation for the pages is pretty limited right now.
