When I create an experiment with ARPU as the goal metric, it shows a very small MDE. However, this is almost certainly incorrect since ARPU is a very noisy metric. Additionally, my experiments that run for 2 weeks+ all show a much wider confidence interval for the ARPU result.
Page URL: https://create.roblox.com/dashboard/creations/experiences/experiments
Are you confusing ARPU with ARPPU?
If you have a Payer Conversion Rate of 0.1%, then suddenly your user base for MDE goes down by 1000x - this would be like going from 39M to 39k over the lifetime of the experiment
If your ARPU is pretty low, let’s assume like 0.003 Robux, as an example that means a very small change for that value would be detectable since it means your entire user base had a change
So if it’s possible to detect a small change in ARPU, how come the confidence interval for ARPU is still very wide after 2 weeks?
do you have the link to your experiment that we can review?
Hello there, I know this one is a bit old but wanted to follow up and check if you still running into this issue. If so could you provide the link to your experiment so we can review. Thanks
