[Release] External key/val database through HttpService

Resources Community Resources
,
#1

So in one of the projects I’m working on I had a few issues with how Roblox’s DataStore service works. I saw some people have been kinda using google docs which I found super duper weird so I wrote my own offsite server solution in NodeJS. I’ve decided to release this to everyone because it’s damn useful. Consider this the basic readme and tutorial for getting it all set up.

This solution has no limits (though it will cost money eventually), and is fully customizable. I chose to deploy my external server to Heroku but since it’s a simple Node.js app you can obviously deploy it on Amazon, Google, or your PaaS of choice.

Heroku has the advantage of having a great free-tier which you can use for development. It’s also really easy to integrate a Redis database, so there’s that as well I guess. This guide is only for Heroku setup. If you wanna deploy to another platform that’s up to you.

  1. First off, grab the files [here]
    (Google Drive: Sign-in)

  2. I’m not gonna guide you through creating a Heroku account… just do that

  3. When you log into Heroku you should see the dashboard. It looks kinda like this

    1255×774 70.2 KB

  4. Click the “New” button and create a new app. Name it and set it’s region to USA (Roblox’s servers are in USA so your http requests will be faster in this region)

  5. Before we deploy we need to add our actual Key/Value storage soution. Go to the “Resources” tab in your project page and add “Heroku Redis”. There’s a free tier, I recommend adding that for now. It should look like this if you’ve done it correctly.

    1136×533 34.3 KB

  6. Now, in your project’s dashboard in the overview you should see some basic instructions on how to deploy your app using Heroku’s CLI interface. Follow these instructions exactly and upload the heroku-server folder you downloaded earlier.

    1152×643 56.3 KB

  7. And that’s it! Your app should now be running at your-app-name.herokuapp.com

  8. Lastly, require the DatabaseModule in your game. It functions quite like DataStore. The api looks like this.
    Values are auto-(de)serialized using roblox’s jsonEncode/Decode.
    GetAsync will return a cached value if requested within the customizable CachLength. HardGetAsync will always request new data.

database:PostAsync(string key, object value)
database:GetAsync(string key)
database:HardGetAsync(string key)

For those of you fancy people who know Node.js if you want, it really wouldn’t be difficult to add JWT Auth to the app and create a secure server… Otherwise, this version is accessible and editable to anyone with the URL so keep it safe.

–Henry

8 Likes
#2

I’m curious, why Redis? Ease of development or another reason?

1 Like
#3

Redis is insanely fast for large databases as it (kinda) stores everything in memory… It’s also the closest relative to Roblox’s DataStore, being key/val focused. I wanted the API to be as similar to Roblox’s as possible to avoid confusion.

Like, yeah I could have gone with a sql table for the key/val but eh…
it was also a learning opportunity ¯_(ツ)_/¯

1 Like
#4

(There are some draw backs to Heroku free plan IIRC, https://www.heroku.com/pricing “Sleeps after 30 mins of inactivity” and some limits on requests per minute etc)

#5

Great application! However, I notice you use HttpService:GetAsync to “put” data. I’d recommend using HttpService:PostAsync to handle storing of data, as it’s a method intended for doing such.

1 Like
#6

Yeah I was lazy with Node and just sent the params in the url rather than thru post. ¯_(ツ)_/¯
It doesn’t matter performance-wise so I just left it.

#7

However, it does affect how much data you can send in the request.

2 Likes
#8

This seems very useful, but can we rely on these services to stay up as long as our games do?

2 Likes
#9

Interesting, I’m highly considering this instead of datastores - the amount of issues I’ve had with datastores is crazy, so much data loss and unreasonable limits.

#10

Will definitely look into this :slight_smile:

#11

It’s all great until the database fails and you have to deal with that. That’s why I’m not using my own website that I’ll be paying for for two years. What do I do when it fails for a couple hours? I’m screwed.

#12

I think the biggest issue with datastores is that there is no real way to share data across multiple places, hence why these sorts of servers are increasing in demand. ROBLOX could easily fix this by either allowing people to disable the auto-hub-redirect functionality on a place or by telling us what placeId and instanceId they were trying to join so we can fix it for them. Similarly, people shouldn’t get “illegal teleport destination” (which was filed as a bug by TG101) if they try to join a server. Full idea here Tell me what game a user was trying to join in a universe

1 Like
#13

Redis naturally sets backups for like every 5 minutes, you don’t have to worry about failure.

#14

Heroku is used by Netflix, Uber, and a bunch of other large name startups for development, testing, and deployment. It’s not going anywhere anytime soon. Alternatively use Amazon?

#15

Sorry to necro this post,
but is this still a viable solution for a custom database, or is there a more effective way of doing it?

1 Like
#16

Best case is to build your own web datastore. You can start learning the subject and decide what’s best for your game but I prefer people use Digital Ocean over Heroku. My guide coverd how to do thid and setup a SQL server on it. There’s also mLab which is cheap for testing startups but gets expensive to upgrade unless your game can cover it.

#17

This should still be just about as scalable as any external data store. It is still quite simple but should get the job done.

1 Like
#18

Remember that Redis is (an) in-memory (database). Any data you save will also be saved in the memory, hence its speed and beautifulness. There is a way to disable this behavior tho, and there is also a way to make a cluster setup, so that you can distribute the data evenly across multiple servers, so that the data does not exceed the RAM.

This is just a heads up to everyone considering using Redis.

1 Like