Animations not unloading leads to server crashes

[verret001 (verre)]

Reproduction Steps
IssueReproducer.rbxm (2.5 KB)
(To note: This has a small sample set of animations, using a larger list of animation ids will illustrate the issue better)

Expected Behavior
When there are no references to an animation by any Animator instance, the server should dispose of the animations not in use. This was the behavior previously.

Actual Behavior
The server does not remove animations that are not in use. In animation-heavy games where there can be a total of gigabytes of animation data that builds up. This leads to server crashes.

Video of me running the reproduction with a large number of animations:

Workaround
I have not found any workarounds to force the animation data to unload.

Issue Area: Engine
Issue Type: Crashing
Impact: High
Frequency: Constantly
Date First Experienced: 2022-01-19 00:01:00 (-05:00)
A private message is associated with this bug report

[andourys]

I may also add in, it doesn’t just crash the server, it can also potentially crash the client due to the high amount of RAM being eaten up.

[AllowSleep (Gordon)]

Can confirm we see the issue too over @ Emotes CO and TTD 3.

Hoping for some kind of resolve soon - this is fairly critical and affects a large number of servers and therefore a lot of our players.

[0_0aztec (nick)]

I assume this problem is also related to the new addition of the AnimationClipProvider service. I am not able to load any animations using an animator object, and receive this errror.

[AllowSleep (Gordon)]

Yeah, I saw that FFlag change not too long ago and I’m just assuming that it’s the main cause of the problem.

[verret001 (verre)]

I decided to dig more into the FFlag changes. I used Rolimon’s average playtime tracking tool to see both TTD3’s average playtime as well as our own games to better identify when the issue started occuring.

This shows that the issue began around Jan 18th 2PM EST (I reported the 19th in the original post as that’s when I received the first reports)

There were two animation related FFlag changes around that time:
“FFlagUseNewAnimationClipProvider_3” was set to True at 1:51 PM EST
“FFlagFacialAnimationDirectImportFormat” was set to True at 1:55 PM EST

As such it would seem that the new animation clip service is behind this caching issue? The timeline matches up perfectly.

[thirdtakeonit]

Thanks for the report! We’ll follow up when we have an update for you.

[NotARobloxEngineer]

Thanks for reporting this. Would you be able to describe the repro steps that leads to this error message and your script not being able to load any animations ?
We have disabled the offending feature for now to avoid crashing animation heavy games, so you may not be able to reproduce this specific issue from now on, but I’d like to have a look at it.

[0_0aztec (nick)]

The error occurred when I called :LoadAnimation() on an Animator object, which was not yet parented to the game object. Once the AnimationClipProvider service was disabled, the error message became much more informative to the issue.

With the more informative error message, I was able to fix the bug instantly.
https://gyazo.com/752946e969d1ae7f8302af3c8d77e6e5

It seems that the service throwing an error just masked the more informative error message. Thank you.