So far from what I’ve observed. The Average Session Time metric on the engagement page doesn’t seem to be accurate. I understand that what it’s reflecting is simply total play time, divided by the number of sessions. However, now that the performance analytics accurately record session time, the metric on the engagement page is completely pointless.
There’s also a very strange disparity between this stat, and the real average session time. Almost every single one of my games shows “Average Session Time” as around 11 minutes; but one of these actually has a real average session time of 9 minutes, another one has a real average session time of 16 minutes, and another one has a real session time of 30 minutes. Yet the engagement page metric is really close across all three.
The way you have to think about Performance is that you did a release and you need to understand what people are doing to decide if you want to rollback.
If you have a bug that causes MOST people on a single platform to have a bad experience then you probably see that all your major PXs like P50 → P90 will be leaving much more quickly.
If you have a bug that causes MANY people on a single platform to have a bad experience then you will start to see that from P50 → P10
The problem with AVG is that it can hide (2) above, because the P10 might be leaving even faster, but since they are a smaller proportion of people (like new users maybe…) your AVG changes within the normal standard deviation but in reality, if you look over a longer period of time, it’ll cause your other metrics to trend down until it stabilizes.
What does PX mean?
For P90, imagine that you have 100 people and your order all of them by session time from lowest to highest. You pick the exact 90th person and that is their P90 session time
Same thing for P50, which is why it is a median and your median is unlikely to look like your AVG because for many Experiences their top users play a lot more than their median user, so you might see that your AVG is higher than your P50.
This is really like you need to use different tools for depending on the thing that you are trying to understand.
Ah I see, then it appears that those tools are broken. My P1-P50 numbers on PC and Console don’t change, despite the fact that my game has more than 100 CCUs per platform.
It’s also strange that even on a Last 1 Day view for P1-P50, all of the session times on each platform report higher than the total at any given timestamp. Yet the total, and subsequent average are always reported really low.
Hey there. While there is definitively something weird going on here (specific to this experience, I checked a few others and can’t reproduce the same situation) I wanted to explain something about Total. The series that you get on the chart are already aggregated, we don’t calculate Total from them because that would be incorrect. Ex. The Computer series is the Percentile X of all data points in a given minute, same as the rest, we can’t based Total on the Percentile X of (Percentile X of Computer, …) instead Total is the Percentile X of all the datapoints in a given minute independently of which Platform they are from. Now, Total being lower than the rest of the values doesn’t seem to be correct. Since we haven’t been able to reproduce it would be necessary to get which experience is this one so we can look at the specific case. Thanks for reporting!
Hey thanks for sharing. I have looked at the data and I think I understand what is happening here. Short answer is there is nothing wrong with the data but we do have already an open ticket to change the way we visualize larger time periods in performance charts due to it can cause what you are observing.
Problem
In realtime session duration total series seems to be lower than the rest of the series.
I noted that this can only be observed for larger periods of time and not for the Past 1 hour period which made me remember about an outstanding ticket we have to change the way we “aggregate” when serving larger periods of time. Allow me to explain: At the moment when you request data for the past hour we return with the original data resolution of 1 minute. But due to the number of datapoints if you request for the past 7 days we use a different resolution of 30 minutes, we internally use something called step in the realtime database which will pick a random data point to be the representative data point for the series in those 30 minutes. Since total is a different series to the rest is happening that we are picking different minutes as representatives and so it looks like total is lower when in fact there is a data point in the rest of your series with lower or the same value in that same minute. Our plan to mitigate this is to replace step with an average over the granularity.
It might still feel like total is lower than the rest, without disclosing any specifics about your metrics, check your P1 at Past 1 hour, note how Total matches almost exactly your 2 lower Session duration platforms which happen to be the ones with more samples (or players ). And this makes sense because P1 (total) == P1 (lower platform(s)). Total is not an average (across all platforms) is a Percentile meaning the 1% value when ordering all the samples at total falls around a similar value as it does for the 1% of only those lower platform values.
I would like to follow up with you as soon as we release the fix mentioned above and see if the problem persist. Again thanks for reporting and hope we can resolve this soon.
). And this makes sense because P1 (total) == P1 (lower platform(s)). Total is not an average (across all platforms) is a Percentile meaning the 1% value when ordering all the samples at total falls around a similar value as it does for the 1% of only those lower platform values.