A|B Tests: Pros & Cons

Feature Requests Website Features
#1

Hi there.

With the now impending release of Roblox Premium being confirmed, I figured it would be a good time to offer up some constructive feedback on the current testing process of rolling out features to the public. As some of you know, Roblox has recently adopted an A|B test program in which a random pool of users will be selected to test out a feature that’s in development. This has caused both positive and negative changes in the Roblox ecosystem, and I think that it’s perfectly fair that we indulge on it here in a professional matter. Here are some Pros & Cons that I was able to come up with off the top of my head.

Pros

  • Gives Roblox full control over what specific users can/cannot see
  • Allows for true criticism to be minimized to the users testing said feature
  • Allows data to be built up to determine whether or not the feature will be a success once rolled out to the public
  • Gives Roblox the opportunity to fine tune the feature under a live audience, regardless of the amount of users viewing said feature.

Cons

  • Causes confusion among normal users who don’t seek out answers for themselves
  • Creates major speculation between the two testing groups on social websites such as Discord and Twitter, which can lead to strong opinions being held over a feature which may not be released
  • Due to Roblox remaining opaque as possible with their new features, certain users will often ‘leak’ fake announcements like the one below in order to gain attention among other allocates (this specific image started floating around many months before Premium was ever publicly commented on)
  • Gives users who have been granted access to the new feature a sense of entitlement, though this can ultimately depend on the personal behavior of the user.

Obviously these bullet points are just my personal view on the situation at hand…I’d love to see what other users thoughts and opinions are on this subject, as I’ve yet to see any deep discussion on this rather radical change take place. Let me know what you guys think.

8 Likes
#2

I don’t think the problem is A/B tests but the fact that there is a complete lack of communication towards players about upcoming features.

Based on the facts that these images circulate, you can tell that players think the only source of news is the Developer Forum, which isn’t even meant for players, it’s meant for developers. There should be dedicated communication channels about (upcoming) updates that are clearly advertised to players so that they don’t even need to come to the Developer Forum or have to trust these fake images.

13 Likes
#3

Couldn’t agree more with you on that regard. It was odd to see the playerbase essentially discover Premium on their own by searching through Roblox’s LocalFiles. You would think they would of at least commented on the feature after it was evident that it was well under development a few months ago, yet they decided to keep their lips sealed until yesterday.

I see this same problem currently festering with the UGC update. Lots of little updates happening behind the scenes, lots of speculation around the subject, yet no official word on the feature since RDC over a month ago, which has led to rampant assumptions and incorrect opinions on the matter.

It seems as if Roblox has decided to keep all of their information in-house until they can fully get a grasp of what the new feature (UGC, Premium, etc.) can do under a public environment. This isn’t necessarily a bad thing to do, but with such a large playerbase, they should expect at least some to question whats going on internally. Especially power users, who have been using the platform professionally for years and expect to know whats going on.

6 Likes