HTTP Error 500 when trying to contact datastores using ProfileService, resulting in Gacha Online failing to load

[SnowdustDev]

Reproduction Steps
This issue started occurring 2-3 hours before this post.
Some game servers for Gacha Online are getting HTTP Error 500 when trying to get data from datastores

Player are unable to load their characters because of this error.

No changes were published within the past week, and this issue only started a few hours ago.
Certain servers are still functional, and newly created private servers are still functional.
This error does not occur when running in Play mode in Studio.

Expected Behavior
Datastores can be reached successfully, allowing players to load their data, such as their custom characters

Actual Behavior
Players join the game, and the server gets an Error 500 when trying to contact the datastores for that player. Players are subsequently invisible and unable to play the game

Issue Area: Engine
Issue Type: Connectivity
Impact: Critical
Frequency: Often
Date First Experienced: 2021-11-09 00:11:00 (-07:00)

[SnowdustDev]

Made progress with debugging, seems as though only certain players are getting repeated HTTP 500s when trying to access their datastore, resulting in a repeated crash

[ShreddedPotat0 (Shred)]

Hi there thanks for the report! Teams are aware and working to resolve it.

We’ll reply in thread with any updates!

[SnowdustDev]

Further debugging leads me to believe ProfileService is doing a repeated task.wait() (or some other infinite loop), hanging the thread responsible for initializing players

[SnowdustDev]

A partial solution has been found, which involves adding a timeout within ProfileService. Game is now playable again

We still notice unusual datastore failures that are happening way more often now, causing players to be kicked for datastore HTTP 500 errors

[ShreddedPotat0 (Shred)]

Hi, this should now be resolved, please let us know if you are still experiencing an issue!