Roblox recently removed the search.roblox.com API links, my GET requests there only return a nil JSON []
I’ve started using catalog.roblox.com and have gotten around the annoying cursor (I miss you &PageNumber=n), but I’m meeting {"errors":[{"code":0,"message":"TooManyRequests"}]} far quicker than I did through the search.roblox.com API.
Does anyone know of any documentation that describes the rate limits for catalog.roblox.com so I can add a throttler to my service as well as the already implemented caching?
Similarly, I would imagine the cursor is being used as an identifier to reduce scraping as well as for pagination (thanks to UGC inflating the DS…), I imagine there’s no way to jump to a specific page as in previous iterations of the API?
There is never documentation for rate limits, if there is it’s very rare, the only reason your search.roblox.com requests are responding with an empty array must be because of a bad request of some sort.
Interesting. Have they changed the URL parameters? It worked fine until fairly recently (and by that I mean a fair few months ago, it’s been a while since I touched this proj).
Yeah, I would have guessed that from the request response(s). As in OP, I am currently using catalog.roblox.com but would love further info with regard to rate limits and/or attempting to request specific pages without having to use the cursor pagination in an attempt to limit requests.
It’s unfortunate internal requests are treated as external despite only using this in a RBX server.
(Technically) I suppose not. I am using a proxy but I am making that request from the game server.
My comment is more in frustration as to why I need to do that in the first place when its an internal request, not entirely certain as to why the focus is on externally-minded API
Extremely frustrating knowing that it’s most likely going to stay with a very low rate limit, getting a list of UGC and Roblox-created items shouldn’t take me a few hours just to retrieve data for it to become outdated only minutes after. (FYI: I don’t need the endpoint anymore, my sales archive has been complete for a while before they enacted a stronger rate limit)
Like @0x6e7367 said, it’s not really reliable. Viewing the documentation of the subdomains directly is more reliable and more readable personally.
Along with their resource, you could also see my list, which at the moment only has undocumented and documented endpoints that allow GET requests. I haven’t had time to document properly yet.
FYI the catalog API will stop returning information when you get to cursor 36; even if you KNOW there is more assets it should be returning.
If you’re trying to scrape the catalog you should instead just try scraping the information off rolimons, or try splitting up your search queries into asset genre and asset type.
If you’re trying to make a decent list of most of the on sale assets, here is something I made recently. (disclaimer: this is not all on sale assets)
