DataStore and HttpService issues in private servers

#1

Hello!

We have received some reports for our users in Royale High whose data occasionally hasn’t been saving in private servers. From what I can tell, two potential issues may be going on.

#1. HttpService requests not always firing during server terminations in private servers. We have a backup server in place to save our player’s data in conjunction with Roblox Data Stores, which then the next time the player joins, we fetch and compare timestamps to be able to determine if the ROBLOX data did not save (we save the same timestamp to both roblox and our back end). We keep track of every time our back up data store gets used, and we do see several of these rewrites. It seems to range anywhere from 500 rewrites per hour to 3000 during peak times.

#2. DataStores aren’t always going through in private servers, with no error/warnings logged whatsoever. We implement :BindToClose() and are yielding the shutdown for the maximum 30 seconds, are still seeing reports. We log any time we are even close to hitting the data store cap, and we have no errors/queued warnings logged whatsoever.

The reports have been coming in for some time now, and we really need a way to make sure it doesn’t happen whatsoever, so I am very curious to know if any other developers have experienced these issues. The good news is that we’ve been able to link all reports to private servers, so that is likely the source of the issue. Thank you

Ice

2 Likes
#2

You haven’t explained the full extent of how you save data, but it sounds like you are only saving when the game shuts down, and probably when a player leaves. Unfortunately, shutdown is not a reliable way to save player data. If a server crashes, your players will lose all of their progress because the normal shutdown code is not run (and this may even be the culprit for your game if there is some private server -specific crash). The same can happen if DataStores are down/being throttled while a server is shutting down.

Your DataStore code will be much more tolerant to failure if you save every few minutes instead of just once while the server is shutting down or the player is leaving. You would save on server shutdown here too, but if the server does crash or there is a DataStore outage, players will only lose a few minutes of progress.

As for the underlying bug causing this (server crashing, DataStores not working in private servers, etc), I would recommend posting this in #platform-feedback instead of Development Support so an engineer can check to see if there are any recurring problems on the backend your game is encountering.

#3

Let me clarify:

We are saving at the following points.

  • Anytime something important happens in our game (user purchases currency, ect…) We are tracking all data store fails + throttled messages, both to Google Analytics and our own backend server. We do Not see any request dropped errors, or “data store was added to the queue”
  • Auto save intervals, every 100 seconds.
  • When a players leaves.
  • Shutdowns (through BindToClose)

I was referring to :BindToClose in particular because my suspicion is that either HttpService isn’t working well with it in VIP servers, so that it never has a chance to communicate with our back-end (hence the reports). Ultimately, it should be saving to data store, but we use our back up server to catch any cases where it is not.

So basically two things are an issue for us:

#1. Some data store requests are not going through, specifically in VIP servers where there are no error or warning messages logged. How do we know that? Well #1 reports - but also we track the number of times our back up server uses the data. This number is relatively high still. As I mentioned, 500-3000 times per hour.
#2. We’re not sure if Http requests are actually going out in all VIP servers. No errors on our NodeJS Application which runs on an ECS cluster in Amazon, so we’re thinking the issue is that the request never actually would get sent out in some cases. We closely monitor and have calculated the amount of traffic that our servers are able to handle.

Thanks!

Ice

#4

In that case, you will need to file a bug report. You can find instructions for doing so here.