Evolving Experiments and Analytics: Segmentation, Experiment Targeting, Early Access Program, and More

Updates Announcements
, ,
1

Key Takeaways

We’re rolling out updates to Experiments to support targeting, segmentation and early harm detection that let you know when an experiment starts to impact key metrics. You can also set conditional configs that serve different values by player type, and break down analytics by new segmentation attributes.

Find it in the Experiments tab of your game’s dashboard. Apply to the Analytics and LiveOps Early Access program to try out and provide feedback on new features.

Hi Creators,

We’re rolling out updates to Experiments, Configs, and Analytics that give you more control over which players see a change and clearer insight into how different segments respond. Experiments let you test how changes to your game affect the metrics that matter most, and creators who’ve run experiments have seen statistically significant gains from them.

We’re also now taking applications for an Analytics and LiveOps Early Access program. If you’re interested in trying out new features and providing feedback for any upcoming Analytics or LiveOps tools, please apply!

New Features

For the items below marked “coming soon”, we hope to push them live over the next few weeks and we’ll update this post when we do.

Conditional Configs

Conditional Configs let you serve different values for the same config key based on player attributes. For example, instead of one boss health value for everyone, you can set a lower value for new players, no separate configs required. Supported attributes include country, language, acquisition source, new versus returning, and payer status, among others.

For example, instead of applying one boss health value to everyone, you might set it to a lower value for a new player. You can now personalize gameplay and live operations without publishing separate configs for different audiences.

Each Conditional Config is built from three components:

  • Conditional rules define who matches, such as players in a specific country or active players.
  • Rule ordering determines which value is selected when a player matches multiple rules. Rules are evaluated from top to bottom, and the first matching rule wins.
  • Conditional values define what matching players receive. Players who do not match a rule receive the Default Value.

Learn more about how conditional rules, rule ordering, and values work, plus the full attribution list in our Conditional Configs documentation.

Experiment Targeting

Experiment Targeting lets you run experiments against specific player audiences instead of your entire player base.

For example, you can run experiments focused only on new players, compare difficulty changes by how long they’ve played your game, or make items available by country.

Experiments use the same targeting dimensions and attributes available in Conditional Configs. You’ll now be prompted when creating an experiment to add targeting.

You’ll need to add a config to run a targeted Experiment. Once you set one up, navigate to the ‘Experiments’ tab of your game’s dashboard to start an Experiment. This documentation dives into how experiments order the evaluation of different conditions and values, eligibility, and more.

image1
image11840×660 38 KB

Experiment Results Segmentation

You can now filter or break down experiment results by player segment, so you can figure out whether a new onboarding flow helped U18 users more than 18+ users, or whether a pricing change improved payer conversion for active payers but hurt it for others. Segmented results help you spot follow-on experiments and prioritize features that work for specific audiences.

This can help you answer questions like:

  • Did the new onboarding flow help more with U18 than 18+ users?
  • Did a pricing change improve payer conversion for active payers but hurt conversion for other players?
  • Did a gameplay change perform differently by country, platform, acquisition source, or tenure?

Early Harm Detection

Experiments now support near real-time monitoring during the first 24 hours through three key metrics:

  • Playtime
  • Average Revenue per User (ARPU)
  • Payer conversion rate

You’ll receive an alert if one of these metrics falls below a catastrophic threshold and is causing significant harm, giving you time to investigate and terminate an experiment early.

Updated Analytics Segmentation

We recently added new segment dimensions to Creator Analytics and Engine API calls, making it easier to analyze and act on differences in player behavior.

You can now filter and break down analytics based on:

  • In-experience activity status
  • Engagement level
  • Platform activity status
  • New vs. returning users
  • Acquisition source

Share Your Feedback

Thanks for helping us prioritize these updates, as many of you asked for them, especially for Experiments. We’d love your feedback on:

  • Which player dimensions you want to use for targeting
  • Which segments would be most valuable in experiment results
  • What additional metrics would help you monitor experiments
  • How you’d like to use Segments through Engine APIs

Let us know what you think, and what else you’d like to see added for Configs, Experiments, or Analytics!

52 Likes
2

This topic was automatically opened after 10 minutes.

3

This is a fantastic update to Experiments and Analytics.

Conditional Configs and Experiment Targeting in particular give creators significantly more control over how changes are rolled out. Being able to tailor configurations and experiments around new vs. returning users, engagement level, acquisition source, country, and other attributes should make testing much more meaningful than applying the same change across an entire player base.

I’m especially pleased to see Early Harm Detection coming soon. Near real-time monitoring of playtime, ARPU, and payer conversion during the first 24 hours gives developers an important additional safeguard when experimenting with production experiences. Being alerted when a test is causing significant negative impact means teams can react quickly rather than discovering the damage later through analytics.

Experiment Results Segmentation should be incredibly useful as well. An overall experiment result can sometimes hide very different outcomes between audiences, so being able to understand who benefited or was negatively affected should lead to much better-informed development decisions.

Overall, this feels like a substantial step forward for LiveOps on Roblox. Giving creators more sophisticated experimentation, targeting, segmentation, and safety tooling makes it easier to iterate confidently while still protecting the player experience.

Really looking forward to seeing these tools continue to evolve!

7 Likes
4

Our studio’s (and many others’) current workflow for pushing out updates is:

  1. test in a private testing place only accessible to devs & testers
  2. during an update, put the game into “maintenance” by setting the public game to private and only letting testers in
  3. let testers playtest the public game during maintenance
  4. once everything is good, open the game back up

This obviously isn’t ideal for many reasons, one notable case being that it freaks out roblox’s analytics due to the drop from tens of thousands of CCU to zero, then back up to the same amount a few hours later.

Would using targeted experiments as a non-disruptive maintenance variant for our testers be an intended usecase? I’m mainly asking this in case we abuse the system in ways it wasn’t designed or thought for :sweat_smile:

2 Likes
5

That’s actually a pretty interesting use-case. I’m wondering though, what made you host testing on the same universe as the main game?

3 Likes
6

Will the updated segments be made
available in AnalyticsService:GetPlayerSegmentsAsync

7

theyre not the same universe; theyre separate game ids. having a testing place is so we can iterate rapidly, while the maintenance period is to make sure theres no drift or critical bugs caused by prod data when loading in.

8

Oh yeah sorry, I meant for the maintenence period specifically.

Have you considered doing it in a private place under the same universe? It still kind of messes up some universe metrics like Session time by place, but generally prevents things like CCU getting tanked.
Just to be clear, I’m not trying to debate your testing choices, just want to hear the opinion of someone experienced.

1 Like
9

Why not use feature flags? Or, slightly riskier, publish your update and then simply monitor game health and quickly revert if anything looks wrong? Taking your game offline to test every update seems like a very strange choice.

2 Likes
11

Amazing update!

By the way if any staff seeing this post: Stolen experience

12

Holy.. it is just that easy..

I would thought this would involve more extra steps but nope. It’s incredibly easy to find and not difficult to setup and with very useful metrics. This is a very, very good addition.

13

why are Custom Events no longer a tab on analytics?
Is there a reason why it was removed, and is it being added back?

14

There’s a myriad reasons involved behind this decision that I don’t have the energy to explain in detail :head_shaking_horizontally:

1 Like
15

Yeah, that’s the same question I had.
I feel like most games have separate places for testing (like as Bee Swarm Simulator for instance) to avoid messing around with the main game

1 Like