Unrealistically low retention data

[Sinne_J (Sinne)]

Reproduction Steps

Ever since release of our game, Sword Haven, on 13th September, performance has been consistently unnatural, bizarre. Retention data was “fine”, we thought the game may have simply been a failure, didn’t think anything of it. However, there has now been 5 instances of extremely unnatural fluctuation in CCU (Concurrent Users), all of them explained in detail below.

To clarify, this is not a complaint or a pity post just because “your game is bad”. I understand it’s a possibility, but I am also highly confident there is a genuine issue in the Roblox algorithm or some other discoverability system, and I would like the situation to be assessed for the mentioned potential issues.

Instance 1

Explanation: We were stable since release until Sept 23th, averaging 1K CCU. Suddenly, on the night of the 23rd, with absolutely no external influence, such as social media promotions, the game jumped from 1K CCU to 4K in just a few hours. Then, on the 24th as the game continued to grow unnaturally fast, Roblox servers went down, and we halved in players. Normal, Roblox is down, players aren’t able to connect. But the game didn’t recover, it stayed at half, and continued downhill. The following day on the 25th Sept we had half the CCU as compared to the day prior. 26th Sept, went from 2.6K to 1K, although going from a Sunday to a Monday obviously always reduces in players, it’s not this much, not by more than x4, it’s unnatural.

Instance 2

Explanation: Following the last day of the first Instance, on the 27th, we were down to just 500 CCU. Once again, without any external influence, we jumped from 500 to 1.9K, and again, no explanation, completely randomly and unnaturally. This continued and remained stable until 29th Sept, where the next instance of this unexplainable activity occurs.

Instance 3

Explanation: After being somewhat stable at ~2K CCU, we suddenly, with no external influence, went to barely 1K on a Friday (30th Sept), where Roblox global CCU grows by ~25%. Completely random drop once again, unexplainable by any means.

Instance 4

Explanation: Exact same situation as instance 3. We were stable at ~1.2K CCU from Mon-Thu (Oct 3 - Oct 6), when we dropped to <800 on Oct 7th, a Friday, at a time where Roblox grows. No external influence, no explanation.

Instance 5

Explanation: Going from Sunday to Monday is usually a 42% drop. But somehow, without any external influence, we went up 20% instead (9th Oct Sunday - 10th Oct Monday).

For now, that is all the unusual activity on our game within the last 4 weeks. I understand natural fluctuation happens, and some of these fluctuations could’ve been, but this many times so consistently cannot be natural. This is my 6th game to be released, and I have never experienced anything like this. If a game falls, it just falls, it’s simply not good enough. Same the other way, if it’s good enough, it’ll grow, steadily and consistently, until it reaches its limit and either stabilizes or beings to fall. Consistently.

Now, retention.

This is the average day 1 retention for each group of games:

The average 2,500 experience averages around 50 CCU. So, with the average top 2,500 experience having a Day 1 retention of 12%, you’d only expect the absolute worst of the worst games, completely dead, having under 10%, right? Well, here comes in Sword Haven, our game, barely holding on to 10%, dipping as low as 6.5%. Day 7 retention just not even realistic, we’ve had 1 failed game previously, that couldn’t go above 300 CCU, and died completely within 2 days. That game had a day 7 retention of 7%. Sword Haven has 1%.

Both of these figures seem completely unrealistic to us, large majority of players we met are loving the game, mostly positive feedback, no performance issues or other notable issues. We have also been tirelessly trying to improve this with research backed features & improvements. For example, simple and interactive onboarding experience, first impressions, progression curve, etc… And it does nothing but continue trending downhill. These statistics deny all logic and sense, and we strongly believe something is wrong on Roblox’s end.

Only thing to mention is we have been sponsoring the game consistently on average 50,000 Robux daily. This couldn’t have had any effect since it was completely consistent since day 1 of release. The only time we weren’t sponsoring was for 2 days, Oct 12 and Oct 13.

To clarify again, this isn’t a complaint about my game failing. It is not a pity post because my game failed. It is something we believe is a genuine issue and is affecting more than just our game.

Expected Behavior

I expect realistic retention statistics.
I expect at least a somewhat consistent Concurrent Users chart.

Actual Behavior

The retention statistics are below anything I thought imaginable, and believe it is an issue with Roblox, not the game.
Concurrent Users are all over the place, randomly spiking, randomly crashing, unnaturally, over and over again.

Issue Area: Roblox Website
Page URL: ⚔️ Sword Haven - Roblox
Impact: High
Frequency: Constantly
Date First Experienced: 2022-09-25 00:09:00 (+03:00)
Date Last Experienced: 2022-10-20 00:10:00 (+03:00)

[DaMePlayzGames (MePlayzGames)]

This isn’t entirely related to the post, but where do you find the CCU data for your games at? Rolimons?

[unsigned_var]

You have to join rolimons’ discord server and request it.

[unsigned_var]

To the staff member reading this, perhaps it is related to player’s not seeing the game in their recent plays or favorites? I have heard about some cases about this, including this post.

[Sinne_J (Sinne)]

An update:

Instance 6

Once again, all numbers are always up on weekends. we were seeing great and consistent growth until around 3 am UTC, where we suddenly stopped, started declining, and never came back up. Theoretically, we should’ve reached 3k at the very least on the Saturday and Sunday, but instead, greatly declined. Going down from 2.3k on a Friday, to 1.8k on a Sunday, relatively is almost a 50% decline. Why? We had very good and consistent growth, large amount of sponsors running, a good recent update very much liked by the community, yet we went so far down.

We would hate to have to move on from a game in this way, having all this occur with no explanation. We were passionate about this game & so was our community, but it is simply not possible to grow or even sustain with these statistics. I’d also like to point out our other statistics, session length and monetization, are very consistent and doing very well. As well as our community, there is only positivity with everyone loving the game very much.

I really hope this can be looked into whatever the outcome may be. Thank you.

[unsigned_var]

A litte summary of our stats currently in terms of benchmark, these stats are either slightly above, or slightly underneath compared group.

• Top 200
  – New Visitors
  – Total Playtime
  – Daily Revenue
  – Average Revenue per Paying Users
• Top 500
  – Daily Active Users
  – Sessions
  – Monthly Active Users
  – Average Session Time
  – Paying Users
• Top 1,000
  – Average Revenue per Daily Active User
• Top 2,500
  – Conversion Rate
  – Day 1 Retention
  – Day 30 Retention
• Top 5,000
  – Day 7 Retention
  – Day 1 Stickiness
  – Day 7 Stickiness

Our Day 30 Stickiness doesn’t have an accurate benchmark indicator, but it’s currently at 0.07% (what?).

I just can’t make sense of how our stats can be top 200-500 in most departments, but then down to 2,500-5,000 when it comes to retention. Is this normal? We’ve calmed a bit down on the sponsors as well (30k/day), so it’s not like we have a constant stream of new players either.

This is all so confusing.

[focasds]

I totally agree with this post, I am honestly tired of every crazy bump the algorithm cases to my game and then makes the game suddenly die again.

On May 2021 my game spiked in players from 30 concurrent players to the 300 players out of nowhere. (No ads, no sponsors, no videos, literally nothing as I was just going to give up on the game until someone told of me of the sudden increase of players, it happened from the night to the morning)

The game suddenly dies as well as time passed by and reached even the 30s again. (This kind of data makes you infer your game is bad, but wait a moment and read below)

On March 2022 my game spiked again with a bigger increase, once again, it happened over night literally, with no sort of influence. My game started reaching the 200s again, then the 400s, and just like that until reach a peak on April 16 of 5k Players. I was in the process of updating my game during the time it grew in players since I literally started working on it again when I saw the first increase on March. Roblox literally took me for surprise.

By May 21, 2022 I had just 1k players and I updated with a horse update which bumped the game to 3.2k players. After that I continued updating the game still with the bigger update I was working on since March, when I released this update on July 16 I already had 300 players and when I made the live event for the game it literally just went up to 500.

---

Stats from my game:

My Average Revenue per DAU (ARPDAU) benchmark is at top 1k experiences.

---

Just like the owner of this post, I receive good feedback from the players and the Like/Dislike ratio is stable currently at 80.6%.

---

Other devs used to tell me to move on; however, how am I supposed to move on if I don’t know what’s causing the current game to fail. Do they want me to make the same mistake if that’s the case?

---

Game Link: The Lost Land [Halloween] - Roblox

[unsigned_var]

Any update on the issue? We still have 7-10% Day 1 retention, which really doesn’t make sense considering how well our other stats are doing.

[gigagiele]

Thanks for the report. We filed a ticket to our internal database.

[music_man1996]

Hey everyone. I spoke with the engineers. Apparently everything is working as intended, but this is still valuable feedback. Feel free to post as a feature request. But we are aware of the concerns here internally, I’ve forwarded this along, and thanks again. Gonna close out the thread.