How do I store and query millions of user created items efficiently?

Help and Feedback Scripting Support
,
1

hola, developers

In one of my projects I’m adding the ability for users to create and publish their own items in-game. Each item has these keys

{
  id = "########-####-####-############",
  name = "######",
  color = "ffffff",
  type = 1,
  date = 1628195892,
}

A unique id, a name, color and type the player provides, and the creation date of the item.

Storing an item is easier said than done. But querying using filters is almost impossible. I want to let players search for an item using filter options. Which currently is just the name, color and type.

I’m sure using an external database would be a smart thing to do if I’m trying to prepare for millions of items. But I’m trying to see if there’s a way to use standard Roblox datastores to save time and resources.

In my conclusion, I believe it’s impossible to store millions of items (the irony, I know) But I made this post to see if there’s someone smarter who can nudge me in the right direction.

Any help is appreciated, thanks

2

You’re tackling an interesting challenge! Using standard Roblox data stores for filtering can be tricky since Roblox’s built-in system isn’t optimized for advanced queries. Typically, DataStoreService is great for key-value storage but lacks indexing and search capabilities.

Here are a few approaches that could help:

1. Using Dictionaries for Filtering

You could store item entries in a structured dictionary format. For example, saving entries under keys based on their attributes:

DataStore:SetAsync("items/type/1", { item1, item2, item3 })
DataStore:SetAsync("items/color/ffffff", { item4, item5 })

This allows for segmented filtering but has scalability limitations.

2. Compression & Partitioning

Instead of storing individual items in one large datastore, partitioning by type or category might reduce lookup time. For example, you could separate items into multiple datastore entries:

DataStore:SetAsync("type_1", { item1, item2, item3 })
DataStore:SetAsync("color_ff0000", { item4, item5 })

Then, when searching, you can retrieve only relevant chunks instead of scanning everything.

3. Using Ordered DataStores

OrderedDataStores allow for numerical-based queries, so if you encode item properties numerically (e.g., converting colors or types into sortable numbers), you might improve search efficiency.

4. External Storage for Large-Scale Needs

If you anticipate millions of items, an external database solution like Firebase, MySQL, or even Google Sheets via HTTP service might provide better filtering.

While Roblox’s datastore isn’t designed for querying millions of entries efficiently, with clever structuring and some filtering tricks, you can optimize your setup for practical searches within limits.

Would love to hear if any of these approaches work for your game! :rocket:

3

Who says there will be millions of items?

I wonder how the platform you are on right now stores all of the user created content…

4

No disrespect but judging by your username and the content in your reply you seem to have just pasted a chat bot’s reply.

The items will be user generated, and in mass. So millions of items is very likely.

Unlike an actual database, Roblox datastore requests are limited. I can’t modify data at will like I would with a database (unfortunately)

Again I appreciate the help but I doubt gpt will be coming in handy with this one

3 Likes
5

You could build manual indexes for filters, which trade storage and write complexity for querying ability. This would add to data stored, but optimize querying.

1 Like
6

good job Chat GPT, now write it without ChatGPT

2 Likes
7

bonjour,

I’ve decided to use an external database to store the items. To anyone who may have a similar problem, I encourage the use of an actual database. You can store as much as you can pay. And no rate limits!

I presume you’re referring to creating a datastore for each major filter e.g. color, type. I can see it working well but I can envision storage limits after a few thousand items.

2 Likes
8

Yes, for the filters. That would compound your original problem, I know.
You would probably be able to use that with an external database or a mix of the two.

1 Like