Hello.
For the past 4 or 5 days, I’ve been uploading new short sound effects (usually up to 7 seconds in length, on average 4 seconds) which have been stuck in pending for insanely long time. In the past, moderation / verification of the sounds used to take between 2 and 5 minutes at most, but recently it now takes more than a couple hours for a 7 seconds sound to proceed with the pending process and finally being able to be heard in studio. On top of that, it now seems like the pending process is done one at a time instead of being parallelized, because some sounds may become available while others are still stuck on pending for another hour, regardless of the duration.
This completely breaks the workflow, as sound feedbacks are as vital to any games as are meshes or images and having to wait for a sound to be done pending for hours to make sure the entire task is done is incredibly bad. I can not imagine how long it would take for the pending process to go through minutes-long sounds/musics. Last-minute sound tweaks are also now impossible, and games in productions will not be able to properly hotfix any sound issues that may arise in a quick manner.
I know that the Engineering Staff has been covertly looking for ways to improve the use of backend resources by intentionally limiting game’s server heartbeat down to 30 FPS in AB testing with no prior announcements whatsoever, and while this change has been reverted, I certainly wish that this sound moderation issue is nothing of that sort either, because after strolling through the announcement channel I haven’t seen any regarding moderation pending duration changes.
For me, this seems like a bug due to the lack of announcement, but feel free to tell me if this post is in the inappropriate category.
Expected behavior
I expect sound moderation to take the usual time it used to take for pending to go through. Or an announcement enlightening us on why the sound moderation now takes so long; and how / to what extent will we be impacted. Alternatively, allowing the developers to hear the audio in Studio despite the sounds still pending (with a warning hinting the developer that sound will NOT be heard in game, but can temporarily be heard on studio for development purposes) may be a good path forward.
Could you provide a few audio asset ids that took a long time to moderate? This would allow us to track the upload through our systems and determine where the bottleneck is.
Eventually they go out of moderation and can be heard in game successfully. However, I’ve uploaded thousands of similar sounds in the past with absolutely no issue whatsoever, they used to go through moderation / pending successfully in maximum 5 minutes. The fact that this now takes hours clues me in either on the fact that the moderation queue has some issues, or that the engineering are trying to save on resources like they’ve done recently.
Thanks for the information! I agree hours for audio moderation is way too much. There are a couple of different teams involved in the audio upload + moderation process so I’ll reach out to the relevant people and get back to you with an answer.
I’ve been dealing with the same issue since last weekend. It seems the audio moderation process takes around 24 hours to approve audios. I’m still waiting for my uploads from yesterday to get approved.
I’ve also had this issue going for a couple days. Though, some audios seem to be passing through just fine. I’m not sure why others haven’t been, it seems random.
We have discovered the root cause of this issue and are in the process of fixing it. The issue was that we had audio uploads get ‘stuck’ in a part of the audio moderation flow and pile up faster than we could deal with them. We have fixed the problem that caused the audio uploads to get stuck and are gradually seeing the total moderation time decrease to reasonable levels again. There may be another few hours before we fully are back to normal.
We had a misconfigured alert that prevented us from realizing this was an issue until we saw this devforum post. We have since corrected that alert so that in the future we should be able to resolve this issue before you all notice. Additionally we will be reviewing similar alerts to make sure they are properly configured as well.
Thanks for the report and sorry for the disruption to your workflows.
Awesome ! That was some insanely quick handling of the situation considering the technical and human challenges. It also always feels incredibly good to feel listened to.
Thank you for your thorough support with this issue and for your communication ! Have a great day
