Hi Creators,
We just launched Early Harm Detection (EHD) for experiments. Until now, results for a new experiment took about 24 hours to show up, and during that time you had no way to know if your variant was hurting your game. With EHD, you get an early look at your key metrics so you can catch a serious regression and stop the experiment before a full day of players is affected.
EHD is only meant to help you decide whether to stop an experiment early. It is not meant for launch decisions. Please continue to rely on daily results when deciding which variant to roll out.
See Near-Real-Time Results In The First 24 Hours
During the first 24 hours of your experiment (or until the first daily results land), some metrics update every 5 minutes.
To protect your performance, we monitor against early harm thresholds every 5 minutes as well. A metric is flagged as critically harmed and highlighted in red, only if its entire confidence interval drops below these limits:
| Metric | Early harm threshold |
|---|---|
| Playtime | -10% |
| Average revenue per user | -20% |
| Payer conversion rate | -20% |
In the example below, harm is detected on playtime since its confidence interval is entirely below the 10% threshold. You can hover over a metric and click ‘View confidence’ to see its confidence interval.
If fewer than 10,000 players are enrolled across a variant and the control, we still show the lift but we do not make harm decisions until there is enough data to detect harm reliably.
Get notified when critical harm is detected
If critical harm is detected during the early harm analysis period, we send a notification to the game owner and the experiment owner so you can act quickly. Notifications are delivered through email and Creator Hub notification tray. Optionally, you can set up a webhook to send early harm notifications to your own tools.
When you receive a notification, review the experiment metrics and decide whether the experiment should be stopped early.
Catch sample ratio mismatch sooner
Sample ratio mismatch (SRM) happens when players are not enrolled into your variants in the expected proportions, which can invalidate your results. SRM is now checked every 5 minutes for the first 24 hours of your experiment, and then daily after that. If SRM is detected, an alert banner appears in the Results tab. We recommend stopping and restarting your experiment since the results of this experiment are not reliable.
Tips for using early harm detection
- Do not launch based on early results. Results in the first 24 hours only test for harm. Wait for your daily results and full experiment duration before making a decision.
- Let experiments run otherwise. Outside of critical harm or SRM in the first 24 hours, we recommend letting your experiment run for its full duration. Early novelty effects can cause results to swing.
- Plan for enough players. EHD needs at least 10,000 enrolled players across a variant and the control to flag harm. Use the minimum detectable effect when setting up your experiment to make sure your rollout is large enough.
Please let us know if you have any feedback or questions in the comments below!
