I was trying to get the a table of all the items published by Roblox, using the HTTPService and the Roproxy Proxy to fetch the data.
When I run the code, it goes as planned until I reach page 34. In this case there is no nextPageCursor data on this, so I can’t continue trought the next pages. Each page has 30 itens and the last one only has 10. In total, I can only fetch exactly 1000 assets. Rerunning the code gives the same error.
So: Is this a limit of the proxy, a limit of the API or am I making a huge mistake in the way I’m trying to fetch this data? ( I’m a completly newbie at making Requests using HTTPService, but I was doing things based on This Documentation Page )
--- Function ---
function CatalogService:RetrievePages()
-- self.NextPageCursor is an attribute inside RNGService
if not self.NextPageCursor then -- Means that this is first time that scrape is happening
local Sucess, FirstPage = pcall(function()
return HTTPS:JSONDecode(HTTPS:GetAsync(self.URLs.MARKETPLACE_URL))
end)
if Sucess then
self.ac += 1 -- This is to index the page and it's number on the table
self.DataTables["CatalogPages"]["Page"..tostring(self.ac)] = FirstPage
if FirstPage.nextPageCursor then
self.NextPageCursor = FirstPage.nextPageCursor
else
warn("Next Page not found")
end
end
elseif self.NextPageCursor == "null" then -- null is to signal that there wasn't a nextPageCursor found in the page
warn("Canot scrape because there isn't more pages to scrape")
return
end
local Sucess, Page
while self.ScrapeData do -- So I can control if I continue the Scrape Process or not
task.wait(0.5)
if self.NextPageCursor and self.NextPageCursor ~= "null" then
Sucess, Page = pcall(function()
return HTTPS:JSONDecode(HTTPS:GetAsync(self.URLs.MARKETPLACE_URL.."&Cursor="..self.NextPageCursor))
end)
if Sucess then
self.ac += 1
self.NextPageCursor = Page.nextPageCursor or "null" -- null means that there is no next page to go
self.DataTables["CatalogPages"]["Page"..tostring(self.ac)] = Page
else
print(self.NextPageCursor)
warn("Fetch failed")
end
else
warn("Next Page not found")
end
end
end
Page 34 data:
