Hi creators,
Today, we’re excited to launch configs and experimentation to help you make dynamic updates to your games and measure their impact on your key metrics.
Configs: Launch features and update in-game values in real-time
Configs let you launch features and update in-game values in real-time without server restarts. You can use them to:
- Turn features on and off, such as a new onboarding flow.
- Update in-game values like daily rewards, weapon damage, and starter packs.
- Launch live events like a new holiday event or temporary player bonus.
How to create and publish configs
You can create, edit, and publish configs on both Creator Hub and Studio:
- Visit the Config page in your experience’s dashboard on Creator Hub or click Open Configs under the File menu in Studio.
- Create the config by entering in a key, type (e.g., string, number, boolean, JSON), and value.
- Copy the config snippet into the relevant script in your game (see documentation).
- Config changes are staged first so that you can test the change in Studio.
- Once you publish the staged change, it’ll be visible to all players.
Experiments: Run both in-game and matchmaking A/B tests
Experiments help you understand the “cause and effect” relationship between your in-game and matchmaking updates and your game’s key metrics. For example, did your new update really cause players to play longer, or did that just happen by chance?. You might create an experiment to test impact on your game’s:
- Engagement and retention: Onboarding flow, daily rewards, progression systems, matchmaking
- Monetization: Shop visibility and UX, starter pack types, item prices
How to create and run experiments
You can create and run experiments as follows:
- In Creator Hub, visit the Experiments page.
- Tap “Create experiment” and select between In-experience or Matchmaking experiment
- Follow the steps outlined in our documentation to create the experiment
Once you’ve created the experiment, you can monitor its progress and results as follows:
- In the Experiments page, hover over an active experiment and select “View”
- On the Details tab, you can see how many users are enrolled in each experiment variant to make sure that your experiment is working properly.
- On the Results tab, you can see your experiment results across your goal and learning metrics. You can also hover over a metric to view the confidence interval for that metric.
- When you’re ready to make a decision, you can ramp either the control or one of the variants of your experiment to 100% of users.
The metrics in the Details and Results tab typically shows up within 24-48 hours after you start the experiment.
Read our documentation and Sign up for our webinar
To get the most out of configs and experiments, we highly recommend that you:
- Read our detailed documentation on configs and experiments.
- Watch our RDC 2025 session with experiment best practices from top creators.
- RSVP to attend our webinar where we’ll show you how to use these products and answer any questions you may have live.
Best practices for running experiments
Experiments are an incredibly powerful tool if you follow these best practices:
-
Define a clear hypothesis stating the change, expected outcome, and why. Write your hypothesis as a cause-and-effect statement that includes the expected direction of the change in your goal metric and the reasoning behind the change. For example, “We can grow D1 retention by updating our onboarding flow to guide players to the first level.”
-
Use the minimum detectable effect (MDE) to decide if your experiment is worth running. We show you this metric when you’re creating an experiment. If the calculated MDE is very high (e.g., ±100%), then it’s likely that you will not be able to reach statistical significance for your goal metric. If the calculated MDE is very low (e.g., <0.01%), then it’s likely that you can actually run a shorter experiment or reduce the traffic allocation to your experiment.
-
Only access the config when you actually want to enroll a player. A player is added to the experiment the moment you call GetValue for the experiment key, so don’t fetch it early. Delay reading the value until the feature truly needs it to avoid enrolling players who never interact with that part of the experience.
-
Don’t act on your experiment results too early. Early swings in metrics could just be novelty effects, so wait for your experiment duration to complete before making any decisions (results take 24-48 hours to show up after experiment start).
-
Make decisions based on statistically significant metrics. When making a decision on which variant or control group to ramp, focus on any metrics that are statistically significant (shaded).
Metrics that are not statistically significant should be treated as if there was no change at all. Keep in mind that a significant shift in your primary goal metric can often cause your other (secondary) metrics to change as well. -
Many experiments present metric tradeoffs. Often an experiment will result in some statistically significant positive and negative signals. You will have to decide how to balance competing directions when deciding which outcome to choose.
-
Keep experiments independent when possible. If you’re running two experiments at once, be aware that a user can get enrolled in both and that there could be interaction effects. For example, if you’re testing increasing player health in one experiment and boosting weapon damage in another, some users might see both increased player health and weapon damage.
-
Document your experiment set-up, learnings, decisions along with next steps. Building up a body of knowledge and evidence can be just as valuable as a single launch decision.
-
Balance experiment results with intuition. Experiments provide strong signal, but you should feel empowered to decide what to build next. Use a blend of art and science to reach your goals.
FAQ
Expand to read more
What are best practices for using configs?
- Add configs ahead of time: The more values you add to configs, the more you’ll be able to update on the fly or experiment on later.
- Use JSON type for name spacing: Grouping a number of values together in a JSON type allows you to namespace similar keys together and helps simplify config management.
Does my config change go live immediately after I publish?
- When you publish a config change, it’ll go live within 5 minutes as servers receive the new value. If you’d like to, you can choose to have the changes go live over a 15 minute rollout instead.
How big does my player base need to be to run an experiment?
-
There’s no hard limit on how many players you need, but the more players you have, the more reliable your experiment results will be.
If you have a small player base, consider gathering ‘qualitative’ (e.g. text-based) feedback through the feedback dashboard and API.
Will players know if they’re part of an experiment that I set up?
- No, players won’t be notified. The goal of an experiment is to understand the impact of the changes you make, and by notifying players it may bias change how they would otherwise react to what you’re experimenting with.
How do I find the right test duration?
- The right experiment duration depends on how much traffic your experience receives, how much traffic you are willing to enroll in the experiment, and how big/small of a change in your goal metric you expect to see during the experiment. Based on your initial experiment set-up, the system calculates the corresponding minimum detectable effect (MDE). You can change your experiment duration such that the MDE calculated matches the change you expect to see.
Is there a limit on how many configs and experiments I can have at a time?
- You can have up to 1,000 active configs and run up to 10 in-game and 1 matchmaking experiment at any given time.
Why are the results stat sig one day, and not stat sig the next day?
- It is normal for results to swing back and forth, especially early in an experiment, where there is not sufficient data collected to make a conclusive decision. As the traffic enrolled to the experiment increases, the fluctuations reduce and the signals stabilize. A result might appear “significant” one day simply by chance, and then return to “not significant” once more data comes in.
- That’s why it’s important to wait until your experiment has run for the full pre-defined duration and obtain trustworthy results before making any decisions.
Why is the lift/drop so big, but it’s still not stat sig, and why can’t I use this?
- A large lift or drop might look meaningful, but statistical significance tells us whether that change is likely to be a real signal or just random noise. When you don’t have enough users, even big-looking changes can’t be trusted yet as the uncertainty and variability is too high.
- Acting on non-significant results can lead to false conclusions, like rolling out a version that doesn’t actually perform better. Once your test gathers enough data (runs for the planned experiment duration), the system will tell you if the effect holds up. Until then, treat large-but-not-significant results as if there was no change at all.
