Summary
Our two avatar-focused experiences log six figures of client-side content-loading failures per day that we have not been able to fix from the developer side. The failing assets are healthy: we spot-checked the exact ids from the Creator Hub error exports against the economy API, and every one is a live, unmoderated asset (example ids with names below, so this is easy to verify).
Affected experiences:
- Catalog Avatar Runway — Catalog Avatar Runway | Play on Roblox (placeId 97577741629233, universeId 10222476236, ~5,000+ CCU)
- Avatar Catalog Shopping — Avatar Catalog Shopping | Play on Roblox (placeId 86145498709536, universeId 10188861194, ~500 CCU)
All counts below come from a single past-day “Errors and warnings by count” export for the SMALLER experience (Avatar Catalog Shopping, ~500 CCU); the larger experience shows the same clusters at roughly 7x the volume (e.g. ~396,000/day MeshContentProvider failures).
These look like three faces of the avatar content pipeline failing under load, so we are reporting them together. Happy to split into separate topics, or provide full CSV exports and complete failing-id lists via DM.
1. could not fetch failures on healthy assets: six figures a day in production, no reason code, and no automatic retry
Daily volume (smaller experience; ~7x on the larger):
MeshContentProvider failed to process asset because 'could not fetch'— ~56,000/dayFailed to load object. PBR textures may not be visible in game.(the SurfaceAppearance under the same accessories) — ~44,000/dayHSRDataContentProvider failed to process asset because 'could not fetch'— ~10,000/day
Example failing ids, straight from the error export — each verified live and unmoderated via economy.roblox.com/v2/assets/<id>/details:
MeshContentProvider (all resolve as healthy Mesh assets, assetTypeId 4):
- 101071619261143 (“RightLowerLeg_OuterCage”)
- 18226221198 (“RenderMesh”)
- 96660452293980 (“default”)
- 122641932589139 (“Mesh”)
- 101700923999360, 101829502636686, 102829916074624, 113596143172015, 18402022909, 83944082659420
HSRDataContentProvider (resolve as healthy Mesh Hidden Surface Removal assets, assetTypeId 75):
- 100612010910687 (“Mesh Hidden Surface Removal”)
- 16594247308 (“Mesh Hidden Surface Removal”)
- 102550976684569, 105089156590982, 110752772305027, 112514369739616, 113148639492293, 118052310145386, 122455087498408, 131279838360417, 94175343774107, 96554700221180
Where it lands:
- Replicated player characters (
Workspace.<username>.Accessory (...).Handle.SurfaceAppearance,Workspace.<username>.Head.SurfaceAppearance). The engine fetches these during character appearance loading — the developer issues no requests here and has no pacing lever. - ViewportFrame rigs from
Players:CreateHumanoidModelFromDescriptionAsync(outfit browsing). One rig fans out 20-40 unique requests — each layered-clothing accessory is a mesh + HSR data + four SurfaceAppearance maps — so twelve outfits on screen is 300+ requests and a scroll through a catalog is thousands. - In-world display rigs we build and place ourselves (
Workspace.Game.CommunityStands.<n>.Character.Rig...), same content shape as 2.
The instance paths in the Failed to load object export are a mix of all three, so this is not confined to any one of our surfaces — and category 1 is not ours to pace at all.
What we measured, and the part we cannot explain:
When we first hit this (August 3), browsing an outfit grid left 179 of 258 meshes on one screen at AssetFetchStatus.Failure, with the console full of could not fetch and Unable to load ... SurfaceAppearance - ColorMap (Error: HTTP 429 (Too Many Requests)). Nothing reloaded on its own — a failed asset never self-recovered within a session, so cards and characters stayed half-dressed or untextured. The only things that brought content back were manual: an explicit ContentProvider:PreloadAsync on the same asset (which succeeds once the wave has passed, proving the content itself is fine and only the cached verdict blocked it), or the engine’s own console hint, “Change a TextureId property to retry”.
We then built mitigation around it (details below). Re-testing today on our current build, we can no longer reproduce a single failure from one developer machine: hammering the outfit grid with repeated full-canvas flicks, peaking at 126 outstanding requests in ContentProvider.RequestQueueSize, sampling every mesh, texture and SurfaceAppearance map on screen — 734 unique assets in one pass, 205 in each of eleven further passes across two different corpus slices — returned Success or None every time, and Failure zero times.
Yet production still logs 56,000 of these per day in the smaller experience and roughly 396,000 per day in the larger one. That gap is the reason we are posting. Whatever conditions produce it — cold client caches, weaker connections and devices, or many player characters loading at once on a populated server — are conditions a single developer client cannot reproduce, and the error message itself tells us nothing about which: could not fetch carries no HTTP status, no reason, and no indication of whether the client was rate-limited, the CDN failed, or the request timed out.
What we already tried (including fixes suggested in earlier threads on this error family):
ContentProvider:PreloadAsyncretry passes on failed assets — eventually recovers our own viewport rigs; cannot touch replicated characters.- Pacing every rig build behind
ContentProvider.RequestQueueSizehigh-water checks, plus concurrency caps that narrow during failure waves — reduced the volume, cannot eliminate it, and does nothing for replicated characters. - The console hint’s TextureId poke (“Change a TextureId property to retry”) — only works on instances we own; not applicable to another player’s character or to SurfaceAppearance content.
- Checked the
Use2022Materialsfix from Errors with specifically PBR materials/SurfaceAppearances — not applicable; that thread is a MaterialService issue, ours is asset delivery.
Questions:
- Can the error carry a reason?
MeshContentProvider failed to process asset because 'could not fetch'gives no HTTP status and no cause. With six figures of these per day and no way to reproduce them on a developer machine, we cannot tell rate-limiting from a CDN fault from a timeout. Even just surfacing the underlying status in the message would make this diagnosable by every developer hitting it. - Is it intended that a fetch failure is cached with no automatic retry for the rest of the session? A 429 is by definition transient, and we can demonstrate the asset fetches successfully moments later.
- Is there a supported way to invalidate the failure cache or trigger a retry — especially for another player’s replicated character, where the developer holds no content reference to
PreloadAsyncand owns noTextureIdto poke? - Is the per-client assetdelivery rate budget documented anywhere? We will gladly pace to a published number — we already pace to a guessed one.
Related: HSRDataContentProvider, MeshContentProvider failed to process asset because 'could not fetch' — that report was resolved as a developer-side TextureID-write loop. Our case has no TextureID writes on characters, and a large share of it is on engine-driven character appearance loading.
2. AnimationTrack limit of 64 tracks exceeded — stock animation ids on players’ OWN characters
Daily volume: ~33,000/day in the smaller experience; ~4.8M in the past-day export of the larger one. Virtually all on <Player> (the local character) with the stock ids 507765000 / 507766388 / 507766666 / 507766951 / 507767714 — the same duplicate default ids reported in:
- Excessive amounts of "Failed to play animation. AnimationTrack limit of 64 tracks exceeded" logs (staff engaged; the April fix moved our errors from
SelfAvatartoPlayer, matching that thread’s OP) - Default Character.Animate leaks AnimationTracks on R15 - hits 64 limit (duplicate default animation IDs)
We instrumented this and pulled the histograms off the live fleet today. The overflow is not 64 different animations — it is a handful of animations playing dozens of times each.
50 overflow reports sampled from live servers (each fires when a character’s Animator crosses the cap; the client walks Animator:GetPlayingAnimationTracks() and reports the id histogram):
| measure | max | median | min |
|---|---|---|---|
| concurrently playing tracks | 65 | 64 | 61 |
| distinct animation ids among them | 65 | 5 | 2 |
| copies of the single most-repeated id | 33 | 31 | 1 |
| tracks at (near-)zero weight | 65 | 64 | 60 |
So the median overflowing Animator is running 64 concurrent tracks drawn from just 5 distinct animations, with one of them playing 31 times simultaneously, and essentially all of them at zero weight — i.e. inaudible, invisible duplicates that still consume the cap.
The most-duplicated ids we captured, including Roblox’s own default animation assets:
id=507777826 x33 (default animation asset)
id=507767714 x32 (default animation asset)
id=74295126803410 x32
id=92095647791191 x30
id=122150855457006 x30
id=82598234841035 x29
id=507767968 x19 (default animation asset)
This is the same signature as the “duplicate default animation IDs” report linked above, with id-level data attached.
Other measurements, for whoever picks this up:
- The cap counts SIMULTANEOUSLY PLAYING tracks, not loaded ones (300 tracks loaded and never played produce zero warnings; refusal starts at the 66th concurrently playing track).
AnimationTrack.IsPlayingreturns true for a REFUSED track — only membership inAnimator:GetPlayingAnimationTracks()is honest, which makes naive verification read as a pass.- Of 55 reports, 37 are the player’s own character, 16 are other players’ characters observed from this client (the overflow replicates), 2 could not resolve a rig.
- We fork the stock Animate script to cache and reuse tracks, and audited every
LoadAnimationsite in our own code for track reuse. We cannot claim that fully exonerates us — but we do not play any animation 31 times at once anywhere, the worst offenders are Roblox default assets, and the same signature shows on other players’ characters. - We shipped a self-heal that stops near-zero-weight and redundant same-id tracks past 48 playing. The live reports show it sweeping 63 tracks on a character that is still at 64 playing — whatever creates them re-creates them immediately, so a developer-side sweep cannot win this race.
The ask here is small and would unblock every developer hitting this: put the animation id (or the Animator’s path) in the warning text. Today Failed to play animation. AnimationTrack limit of 64 tracks exceeded names nothing, which is why this has been open across several threads for months — we only got the numbers above by building custom instrumentation and shipping it to production. There is already a request for exactly this: Add Animation Id or path to "AnimationTrack limit of 64 tracks for one Animator exceeded, new animations will not be played
Full per-report JSON available on request.
3. Failed to load animation with sanitized ID on healthy catalog animations
Daily volume: ~9,000/day (smaller experience). Example failing ids from the export — each verified live and unmoderated via the economy API (assetTypeId 24, Animation):
- 102726115142727 (“JumpFixed”)
- 10921355261 (“Locomotion_Zombie_Walk”)
- 103685705441114, 105463045447284, 111726465575659, 114008465322602, 117075178031549, 119553632455878, 122150855457006, 86375733701002
These reach the client as worn avatar animations (animation-pack / emote children on players’ own avatars), so the usual workarounds do not apply: group/ownership audits (these are catalog assets the platform itself equipped on the avatar), server-side RunContext (the failing loads are the engine’s own client-side character animation path), and verifying outside Studio (this data is from production).
A possible link to section 2, offered as a lead rather than a conclusion: Failed to load animation with sanitized ID errors reports that failed client-side animation loads make the Animate script leak tracks toward the cap. Cross-referencing our two datasets, one id — 122150855457006 — appears both in our sanitized-ID load failures and in the 64-track histograms, duplicated 30 times on one Animator. That is one overlap out of ten sampled failing ids, so it is not proof; but if a failed load leaves a track behind, these two clusters are one bug, and whoever can see the load path internally could confirm or kill that in minutes.
Reproduction: join either experience above and browse avatar/outfit content, or simply stand in a populated server while characters load. The errors accumulate continuously in the Creator Hub error dashboard for both experiences. Happy to provide anything else needed to reproduce via DM.