Engagement Analytics are Wacky

[Redridge (Red)]

Thanks to the Analytics team now we have a bunch of places to track our game’s engagement. To be precise, I know of 3:

1. Developer Stats (Average Visit Length Dashboard)
   
   
2. Engagement Tab under Analytics (Average Session Time Dashboard)
   
   
3. Performance Tab under Real-Time (Session Time Dashboard)
   
   

These dashboards, all show completely different values. 1 and 2 are fairly consistent between each other. They show a difference of about 2-3 minutes, but it’s constant so at least we can assume wrong reporting.

The newest Performance one, is by far the wackiest, with values that don’t make any sense to me (and I assume for most users).
There are times where “Total” is between “Computer”, “Tablet” and “Phone”. This would make sense, since “Total” is probably the average of all sessions.

But then there are times where it’s much higher than everything else:

Make it make sense.

The values are also much higher than dashboards 1 and 2. Even if I pick the 1th percentile. The percentiles in general, do not seem to make much sense. I would expect the 99th percentile to be the longest 1% of sessions. SO for the 1th percentile am I getting the session that’s higher than the lowest 1% sessions? That shows a very high number (16 mins). It’s impossible that 99% of our sessions are higher than 16 mins when our average (as reported by us) is around 15.

The performance dashboard also has no bucketing. If I pick a date range of 30 days, you are showing me literally every hour of those 30 days. I would expect some transformation of the data points there. As a result, my laptop starts heating extremely fast when I pick a big date range.

So to conclude this post with useful action items:

1. Why are the metrics different in 3 places? Can we make them the same?
2. How does the “Total” in the performance dashboard work?
3. Did I interpret correctly the percentiles that you are providing in the performance dashboard?
4. Can we add some transform of datapoints when big ranges are selected? The data is noisy and doesn’t really make sense without transforms.

Expected behavior

I would expect all dashboards to show similar values. If not, we should at least get some explanation of how these metrics are measured and why are they so different from each other.

Page URL: https://create.roblox.com/dashboard/creations/experiences/3450993160/analytics/performance?rangeType=Last7Days&filter_Place=Terramon+[Adventures](9201515632)

[Focia19]

We’ve filed a ticket to our internal database, and will follow up when we have updates!

Thank you for the report!

[friedrich1717]

Hi Red:
thanks for reporting this confusion:
For screenshot 1. That is a legacy system that we want to fully deprecate and it is under our radar. that is not very accurate and you could stop referencing that one.
Screenshot 2. That is the most accurate one computing all the play sessions for your experience on average.
Screenshot 3.
99th percentile is the longest 1% of sessions, so it is longer than the average. I check your experience, if you pick 1th percentile, that is the lowest 1% of sessions and the number is smaller than the average, which makes sense. this one is minute level so one outlier (for example, one user that has played quite long time) For total greater than every platform, we are working on improving the way we compute the value to fix this issue and we will keep you updated.