CatalogScraper - Scrape assets from the Roblox catalog

CatalogScraper

Scrape assets from the Roblox catalog with ease


Author: @Starnamics
Version: 1.0.0
License: MIT License


Download on GitHub :link:
Download on Roblox :inbox_tray:
View Source :computer:


Introduction :tada:

CatalogScraper provides developers with a way to scrape assets from the Roblox catalog.

CatalogScraper has support for all functionality of the catalog search API and allows for developers to fetch a huge amount of assets from the catalog, with virtually no limit to the amount of assets that can be fetched.

CatalogScraper is also fully typed, making it extremely easy to start using.

Installation :inbox_tray:

Installation of CatalogScraper is extremely easy, all you need to do is insert the module into ServerScriptService!

You can get the module from either the Roblox library or from the GitHub releases page

Once you’ve downloaded the module, open up studio and either insert it from the toolbox or from the file explorer.

Examples :books:

Scrape the Top 100 accessories from the catalog

This example shows you how to scrape the Top 100 accessories from the catalog and print them out in the console.

--// Require the module
local CatalogScraper = require(script.Parent:WaitForChild("CatalogScraper"))

--// Create scrape parameters
local ScrapeParams = CatalogScraper.ScrapeParams.new()
ScrapeParams.Category = "Accessories" --// Set a category
ScrapeParams.Subcategory = "Accessories" --// Set a subcategory
ScrapeParams.Limit = "10" --// Set the limit of asset results per request
ScrapeParams.SortType = "Favorited" --// Set the sort type

local Accessories = CatalogScraper:Scrape(ScrapeParams,100) --// Scrape 100 assets using the scrape parameters

for Number,Data in pairs(Accessories) do --// Loop through all accessories returned
	print(Number,"-",Data.Name) --// Print the accessory and it's ranking
end

Documentation :bookmark_tabs:

Scraping Assets

To scrape assets, you can use the CatalogScraper:Scrape() function.
This function will send requests to the catalog search API and will parse and compile the results into an array for you to use.

Due to limitations with the search API, you can only get up to 30 results per request currently, however you’re still able to request as many results as you’d like as the module will continue scraping until the amount of assets you wanted have been returned.

You are also able to scrape continuously until there are no more results by entering no amount to the arguments, however this may take a while to complete depending on the scrape parameters you entered.

Creating ScrapeParams

To create ScrapeParams, you can use the CatalogScraper.ScrapeParams.new() function and then edit the values of the returned dictionary.

Functions

CatalogScraper:Scrape()

Description:
Scrapes assets using the catalog search API

Usage:
CatalogScraper:Scrape(SearchParams,30,true)

Arguments:

  • Parameters: ScrapeParams
    Required
    – Parameters for scraping
  • Amount: number
    Optional
    – Amount of assets to scrape
  • LogOutput: boolean
    Optional
    – Should the amount of assets scraped be logged on each request

Returns:

  • Assets: {...Asset}
    – An array of the scraped assets
CatalogScraper.ScrapeParams.new()

Description:
Creates an empty ScrapeParams dictionary

Usage:

local ScrapeParams = CatalogScraper.ScrapeParams.new()
ScrapeParams.Category = "Accessories"
ScrapeParams.Subcategory = "Accessories" 
ScrapeParams.Limit = "10"
ScrapeParams.SortType = "Favorited"

Arguments:

  • None

Returns:

  • ScrapeParams: ScrapeParams
    – Empty ScrapeParams dictionary

Classes

Asset
AssetType: string,
BundleType: string,
Description: string,
CreatorHasVerifiedBadge: boolean,
CreatorName: string,
CreatorTargetId: number,
CreatorType: string,
FavoriteCount: number,
Genres: {...string},
AssetId: number,
ItemRestrictions: {...string},
ItemStatus: string,
ItemType: string,
LowestPrice: number,
Name: string,
Price: number,
PriceStatus: {...string},
PurchaseCount: number,
UnitsAvailableForConsumption: number,
ProductId: number,
ScrapeParams
Category: string
CreatorName: string
CreatorTargetId: number
CreatorType: string
Cursor: string
Genre: string
Keyword: string
Limit: string
MaxPrice: number
MinPrice: number
SortAggregation: string
SortType: string
Subcategory: string

Thank you :wave:

If you encounter bugs, need support or have feedback for me, please let me know in the replies of this topic! :bug:

33 Likes

Extremely handy resource! I was wondering if you can filter out certain assetTypes or fetch for specific assetTypes. For example, you want to fetch from the Accessories category but you don’t want T-Shirts.

2 Likes

At the moment, it’s not currently an included feature but you should be able to write some code to do it by looping through the asset table and checking the asset category with MarketplaceService:GetProductInfo()

3 Likes

Star, back at it again with another useful resource.

4 Likes

Is there a way to save the results of the previous scrape() function and do another that would continue where the first one stopped? Like going onto the next page, or scrolling down farther past where the first scape results ended.

2 Likes

At the moment there is not, however I may add this in a future update once I am a bit less busy!

3 Likes

That’s great, nice. I also noticed there is an issue where if the scrapeparams category is clothing and the subcategory is either a shirt, pants, or t-shirt then there isn’t anything scraped and I get an error. I don’t know if maybe I just missed something and those don’t go together but trying out some things got me nothing.

2 Likes

This could be due to Roblox API charges with the new clothing types and everything, I’ll look into it soon.

2 Likes

Is there a way to scrape only 1 item with the item’s itemId?

I would probably just use one of the built in functions to access this.

AvatarEditorService:GetItemDetails

1 Like

This is incredibly useful! I am trying to get a list of all accessories uploaded by Roblox but the limit seems to be 1000 assets before it stops returning a nextPageCursor. Is this a limitation of the Roblox/RoProxy’s Catalog API?

I believe it’s a limitation from the Roblox API

I seem to be getting an error when I try to use it:

image
May it be caused by how I’m trying to scrape? If not, how could I fix it? I’m thinking it might be a problem with the proxy url, but I might just be setting this up wrong with the ScrapParams.

local Param = CatalogScraper.ScrapeParams.new()
Param.CreatorName = "Roblox"
Param.Category = "Collectibles"
Param.Subcategory = "Accessories"
Param.SortType = "Favorited"

local Limiteds = CatalogScraper:Scrape(Param)