Starting today, players have been seeing this message when trying to join a server in my game, Welcome to Farmtown. I haven’t received any such reports prior to today, and I’m seeing it myself.
This is tanking my player count so I’d appreciate a quick resolution. I don’t believe it’s caused by anything in my game, since it just started today and I haven’t changed anything that would seem likely to cause this. My game does send more data on startup than most, but far less than my other game that uses the exact same codebase and I have not observed that issue there.
What’s odd is that the game is apparently still running and is even connected to the server, but there’s no way to dismiss this dialog. This is leading me to believe it might be showing up in error, but since it’s such a vague message, I have no idea how to begin diagnosing it.
It’s notable that my game is using StreamingEnabled. This most likely differentiates it from other games, as many of my players have said they aren’t seeing this in other games.
Some further info: I’ve also been getting a lot of reports about just losing connection in general. It happened to me just now, after a while of being in the same server. Here’s the log:
1573715610.03345,3f60,7 AckTimeout: Last Packet received time: 10001 ms
1573715610.03345,3f60,7 Connection lost
1573715610.03345,3f60,7 Connection lost: connectMode: Connected, timeMS:2429421906, connectionTime 2425456855
1573715610.03345,3f60,7 Connection lost: AckTimeout 1, IsOutgoingDataWaiting 1
1573715610.03345,3f60,7 Connection lost - Cannot contact server/client
1573715610.03662,4580,7 Lost connection to 128.116.56.22|60265: ID_CONNECTION_LOST
1573715610.03662,4580,6 Lost connection to 128.116.56.22|60265
1573715610.11255,4580,6 OnDisconnect http req time:68.01ms
1573715610.11548,4580,7 Removing replicator instance: 1D565088
1573715610.25171,4580,7 Time to disconnect replication data: 134.364304
1573715610.25171,4580,7 Sending disconnect with reason: -1
So I’m pretty much just timing out, but why is it so common?
Yeah, I turned off StreamingEnabled today and I’m not seeing the issue anymore. Unfortunately, this isn’t much of an answer because my game is too large to not use streaming at this point.
Not sure who’s working on this now, but someone should probably look into this. @darthskrill, can you pass this along for me?
EDIT: Spoke too soon, it’s still happening with StreamingEnabled off. However, it does seem a little less common. Not sure if it’s less common enough to be significant since I don’t have a good way to log events for this. After reviewing some logs, it looks like there’s a common pattern of people disconnecting after receiving no packets at all for about 10 seconds (which is probably the timeout.)
I haven’t had any reports since, so that seems to have fixed it. I’ll follow up if I start seeing it again, but thanks! Please pass my gratitude on to the networking team, they’re doing great things.
EDIT: I did get one report, but it was from someone in a VIP server that may not have updated.
We did not re-enable the flag, however when we disabled the flag we did so in a way that may not have been propagated to mobile clients, I will make another tweak that will definitely be propagated even to the long tail. Do you know if the new reports were from mobile or Xbox users?
I’m not totally sure, but this seems fairly likely. It was much less pronounced, so I figured that there were tweaks made and the flag was re-enabled. If it was only mobile, that’d probably produce the same effect. I don’t have Xbox enabled for my game yet, either. Thanks!
We have not re-enabled the original offending feature, and we are not seeing other indicators of the original failure in our metrics. What error messages are the users seeing when they are disconnected?
Looking at Farmtown in particular, I do see some out-of-memory crashes attributed to your game which might manifest as connection failures. Can you try going in to a few instances of your game and inspecting the server memory in the developer console, to see if anything stands out?
As an update, we are investigating from our end. We’ve looked at the Data Store client code and it shouldn’t allocate more than 8MB cache, so 3470MB indicates something is going wrong. I’m reaching out to some other teams to figure out how we can do some more detailed debugging to understand what is happening.
In the meantime, is this issue continuing to happen for your game?
Hi mr_smellyman,
Upon further investigation, we found that if many DataStores are initialized and used by the DataStoreService, it’s possible to get past the 8MB limit mentioned before. Do you know if you create multiple DataStore objects using DataStoreService:GetDataStore in your game and use them frequently?
Yes, I do indeed. There’s one for each player. Is there a way to ensure the memory will get released once a player leaves?
It appears to monotonically increase every time I write a new data version, which happens every 30 seconds, even if the datastore writes are successful. That doesn’t seem quite right, hope it helps.
Is there any reason for you to create one for each player? If possible, I recommend you to switch to using a few / single DataStore with different keys for different players.
Not particularly, but there are plenty of code samples on the Developer Hub that use a datastore per player as well. That suggests to me that this is intended to be a valid use case.
However, it does seem a little less common. Not sure if it’s less common enough to be significant since I don’t have a good way to log events for this. After reviewing some logs, it looks like there’s a common pattern of people disconnecting after receiving no packets at all for about 10 seconds (which is probably the timeout.)