How Can You Handle A Huge Database?

Help and Feedback Scripting Support
#1

How can you handle a huge database? Like lets suppose I am creating an in-game ROBLOX forum, and I will need to store a lot of data, how would I do that? There is a 260k character limit. I know that I can just create multiple datastores but what if those multiple datastores get filled up too?

1 Like
#2

You can always use this:

You would have to either rent or own servers outside of Roblox however, it allows for more requests than data stores and larger databases as the file sizes then depend on system storage which I would assume would be larger than 260K characters.

1 Like
#3

I am aware of the fact that I can use a third party database but they cost money, and the ones there are free are not good for serious projects. I do not really want to use a third party database for now.

#4

As far as I’m aware then you’ll just have to deal with multiple data stores being filled up over time.

#5

Data Compression + Efficient Data Structure is one method to make the DataBase more efficient

You can use BitBuffer, StringCompression and any other module you see fit as well

I don’t think that’s a major concern, DataStore functionality isn’t very flexible I would rather use a 3rd party service for that kind of system because there are limits and such which makes creating a in-game ROBLOX forum very challenging.

Don’t worry about multiple DataStores getting filled up, just make more

1 Like
#6

I never heard of BitBuffer or StringCompression. What are they? Also just storing data efficiently and making it so it takes less space won’t prevent my datastores from getting filled up. Also as I said I want to rely on ROBLOX’s datastores because I do not want to pay for databases and the ones that are free are not good for serious projects and storing a lot of data. Also if my datastores fill up, I need to manually create new ones. That is what my main concern is. What if I run out of storage? No data will be saved until I manually create a new datastore.

#7

I understand that you want to use DataStore, I only meant to state my opinions on the situation.

It doesn’t prevent it from filling up but it slows down the process quite a lot so it’s worth mentioning.

You can Automate that with code, unless I’m miss understanding something

BitBuffer64 - Slow

StringCompression - Not perfect

You can use

HTTPService

to measure the Data Size (String Length)

1 Like
#8

That is what I wanted to say, woops.

Well, I haven’t really thought of that, how would I actually do that? I actually have an idea but the method I have in my mind seems really inefficient. Basically I will just use 1 datastore but different keys (I forgot that you could just store different keys) . And lets suppose the key is 1, if the key has reached its 260k characters limit it will create a new key which is basically the key + 1. So the key will be 2. And I can split the data. I can have all of the data registered in a dictionary, JSON encode it and split the string. And then when fetching the data I can gather the data from all of the keys, merge the JSON list string and JSON decode. So every time the datastore fills up I can create a new string. I do not know if this idea is good tho.

And thanks for linking me the articles.

2 Likes
#9

This seems like a good idea for a game that involves lots of player data (building game, etc.).

#10

If you are using more than 260,000 characters per key, what are you doing? Don’t store it all in one key. For example, a fourm might have a key like “creatoruserid:topiccount”. Can you give a example of data you are storing? You should not exceed this limit without massive amounts of data.

If you have too much data, try using data compression. There are some good examples above.

If you are confident that you really need a separate data storage, you would have to host/rent a server which can take get and post requests using httpsService.

1 Like
#12

As others have mentioned you can either use DataStore or a web server with HTTPService.

There is a 260k character limit. I know that I can just create multiple datastores but what if those multiple datastores get filled up too?

Perhaps you could explain what you mean here, why is this an issue and how would several datastores help?

#13

That limit is per key, not per datastore. You can :SetAsync multiple keys in the same datastore with 260kb of data. Basically, this means that single datastore can store an infinite amount of data.

Regardless, the only way I can see a forum being implemented in-game is with an actual database for several reasons:

  1. You can’t search Roblox’s datastores. If your forum gets large enough that you’re using 100s of keys, you’ll need to query all 100 of those keys when doing your search. You’re going to hit the request limit very quickly.

  2. There must be a 6-second delay between each :UpdateAsync() (and yes, you’ll have to use UpdateAsync, not SetAsync, otherwise you’re going to overwrite posts from other servers. I’ve been told that UpdateAsync isn’t even enough to ensure consistency since UpdateAsync only uses the last known value, not its current value!) You’re going to encounter a lot of throttling if players in your server decide to post at the same time. Source. It’s under the section “Server Limits”

  3. In addition to having to handle throttling writes to a key, you’ll also need to throttle requests globally. While you can handle #2 by queueing/batching requests in a queue in the server (at the cost of posts being sequential and in realtime), this limit is practically insurmountable. Source. It’s under the section “Game Limits”

By the way, the wiki has this to say about avoiding hitting limits:

Your forum use case is neither of the two.

TL;DR: Invest in an external database for your forum because using Roblox’s datastores for the same purpose is pretty much impossible once you hit enough players.