Inventory can't be entirely scraped with ItemConfiguration endpoint (HTTP 504)

Hi, so I’m having some issues with the following endpoint:

https://itemconfiguration.roblox.com/v1/creations/get-assets?assetType={}&isArchived={}&limit={}&cursor={}

It appears that after going through several pages, the response time slows down drastically, and will throw HTTP 504 (Gateway Timeout) more frequently until it refuses to send back a response, which makes it nearly impossible to scrape large inventories for assets.

This error starts happening at around page 45 in my inventory (limit=100), and by page 56 it completely refuses to give me a successful response back. This also affects the Creator Dashboard, since this is the endpoint it relies on to fetch uploaded content.

Reproduction

While you could use something like Postman to make individual requests and copy the cursor over until this issue starts occurring (which I did to show that it’s not an issue with specifically an http library), I included a small Python script which reproduces this issue consistently in my inventory (or any large inventory) in less than two minutes.

Just make sure to replace your cookie for the account you’re testing with so it has proper authentication. This also requires the requests library to send HTTP requests.

Finally, I’ve been able to reproduce this in four areas:

  • Postman
  • Creator Dashboard (Firefox)
  • Python (Requests library)
  • C# (System.Net.Http)

Expected behavior

I expect for the request to not timeout, even if it takes longer to process the request on the server-side, and eventually give me a full list with the cursor(s) needed to advance.