Amazon Server Storage Module

Amazon Storage Server

I made an Amazon server located in Ohio to store game data across universes. It uses the Dynamodb structure and caches get requests in RAM for some time and updates the caches from posts for a couple minutes.

What can it do?

Currently it can:

  • handle 300000 cached requests per second,
  • return cached data in sub 1ms,
  • get data from Dynamodb in 37ms on average,
  • do posix multithreaded processing,
  • store complicated Lua tables and arrays without userdata.

Why should you use it?

Because with this, you;

  • can easily save data across games and universes,
  • don’t have to worry about error handling and server errors,
  • can handle all of your data receiving and sending needs with 2 simple functions.

Amazon.rbxl (17.5 KB)

This is a place that comes with an Amazon wrapper module in ServerStorage and a TestScript in ServerScriptService. Run the game and the game will load data and then save data to the Amazon servers. You’re free to use the simple wrapper however you like.

Scripts
--Test script inside ServerScriptService
local Amazon = require(game.ServerStorage.Amazon) 

local UserId = math.floor(tick()/120) --Get an user id valid for ~2 minutes

local Data = Amazon.LoadPlayerData(UserId) --Immediately load player data of the user id

local PlayerData = {}

if Data[1] and Data[1]==591 then --No data code
	print"no data"
	PlayerData.Points = 1
else
	PlayerData = Data --There is data and we can add it
end

print("Points:",PlayerData.Points) --Current data value

PlayerData.Points = PlayerData.Points + 1 --Increase it by 1


Amazon.SavePlayerData(UserId,PlayerData,true) --Add it to save queue, the 3rd argument is importance


Amazon.Step() --Make a save step and actually send the data to server 
--(since importance was set to true, step here does nothing)

--Wrapper module inside ServerStorage
local module = {}

local requrl = "http://18.217.51.64:9002/"
local get = "getdata/"
local post = "postdata/"

module.AuthKey = "Game1"

local http = game:GetService("HttpService")

module.AutoSave = true --True or False

module.Verbose = true

local fwarn,fprint=warn,print
local function print(...)
	if not module.Verbose then return end
	return fprint(...)
end
local function warn(...)
	if not module.Verbose then return end
	return fwarn(...)
end

local SaveRequests = {}

module.SavePlayerData = function(UserId,Data,Important) --Number, Table, Bool
	assert(UserId,"UserId can't be nil",2)
	assert(Data,"Data can't be nil",2)
	assert(typeof(UserId)=="number","UserId needs to be a number",2)
	assert(typeof(Data)=="table","Data needs to be a table",2)
	Data.UserId = UserId
	module.SaveRaw(Data)
	if Important then
		module.Step()
	end
end

module.SaveRaw = function(Table)
	Table._ltik=tick()
	table.insert(SaveRequests, Table)
end

module.LoadPlayerData = function(UserId) --Number
	assert(UserId,"UserId can't be nil",2)
	assert(typeof(UserId)=="number","UserId needs to be a number",2)
	local res,body = module.GetRaw{Body={saves={{key=UserId}}}}
	return body.saves[1],res
end

module.GetRaw = function(Table) --Run for when player joins to get their data
	assert(Table,"Request can't be nil",2)
	local url = requrl..get..(Table.AuthKey and Table.AuthKey or module.AuthKey)
	local method = "POST"
	local headers = {}
	headers["Content-Type"]="application/json"
	local body = Table.Body and http:JSONEncode(Table.Body) or "{}"
	local res = http:RequestAsync{
		Url=url,
		Method=method,
		Headers=headers,
		Body=body
	}
	if res.StatusCode<=299 then
		body = http:JSONDecode(res.Body)
	else
		warn("Req failed:", res.StatusCode, res.StatusMessage)
	end
	return res,body
end

module.Step = function()
	if #SaveRequests>0 then
		local reqSave = {}
		for i,tab in pairs(SaveRequests) do
			reqSave[i]=tab
		end 
		table.sort(reqSave, function(a,b) return a._ltik<b._ltik end)
		
		SaveRequests={}
		
		local res,suc
		repeat
			local t = tick()
			suc,res = pcall(function() return http:RequestAsync{
				Url=requrl..post..module.AuthKey,
				Method="POST",
				Headers={["Content-Type"]="application/json"},
				Body=http:JSONEncode{
					saves=reqSave
				}
			}
			end)
			print("time http taken to save: " .. tick()-t)
			if not suc then warn(res) end
		until res and (res.StatusCode<=299)
		
		local immediateRedo
		local body = http:JSONDecode(res.Body)
		if body.saves then
			for i = 1,#reqSave do
				if body.saves[i] and typeof(body.saves[i])=="table" then
					if not body.saves[i][1] or body.saves[i][1]>299 then
						warn("error while saving code",body.saves[i][1])
						warn("error while saving reason",body.saves[i][2])
						table.insert(SaveRequests,reqSave[i])
						immediateRedo=true
					else
						warn("save success")
					end
				else
					warn("not saved",body.saves[i])
					table.insert(SaveRequests,reqSave[i])
					immediateRedo=true
				end
			end
		else
			immediateRedo=true
			for i = 1,#reqSave do
				table.insert(SaveRequests,reqSave[i])
			end
			if body[1] and typeof(body[1])=="number" then
				warn("save global error:",body[1],body[2])
			else
				warn("unknown error:",res.StatusCode)
			end
		end
		if immediateRedo then
			module.Step()
		end
	end
end

game.Players.PlayerRemoving:Connect(function()
	if module.AutoSave then
		ypcall(module.Step)
	end
end)

coroutine.wrap(function()
	while wait(3) do
		if module.AutoSave then
			ypcall(module.Step)
		end
	end
end)()

return module

Amazon Module API

void module.SavePlayerData(number UserId, table Data, bool Importance)

Will add your Data to a queue. If Importance is set to true, it’ll call Step function right after and immediately save the data. The given Data table can have any complexity that’s valid for JSON in it.


table Data module.LoadPlayerData(number UserId)

Yields. This function takes an UserId number and returns Data that was saved. If there was an error, it’ll return {ErrorCode, ErrorReason} as a response.


void module.Step()

Yields. An important function which does the actual saving to the server. When you call SavePlayerData as not important, it adds the data to a queue and actually saves and sends the data to the Amazon Server when it’s called. This means your non-critical data should be set as unimportant and be waited for more data to be sent together.

Every request to the server, strains the server so it’s a better idea to send a whole bunch and let it solve it in parallel. Every request also takes away from the currency of the key, exhausting it.


module.AuthKey = "Game1"

This is the key field of the module. This is either the public key, or private key given to you by me. It represents the name of the table your data is getting saved to. The public keys are public and can be edited and read by anyone.


module.AutoSave = true

This field does a step call whenever someone leaves the game and every 3 seconds, if you don’t want to worry about step function calls yourself.


module.Verbose = true

Boolean field prints and warns everything for debugging.


18.217.51.64:9002 is the current IP:Port address of the server. It will be transferred to a domain later.

For now, I’m going to keep it in an open beta state to collect some errors while you (devforum) uses it freely after which I’ll offer the service for some price. I would like you to test its robustness with the public key “Game1”. I might wipe out the table if I think it’s being abused so don’t put any sensitive information in to the public key and if you would like to use it for some kind of production release, send me a message through devforums and I’ll give you a free upto 200 MB data private key to use for yourself.

In the future, I will add ordered data stores and leaderboard systems but before that, I need to see if the current system works as robust as possible.

The system is always subject to change and I update it pretty much daily. Your public key data can and will probably get lost while I close and open the server. This won’t be a problem as I will be doing load balancing type Amazon server array where updates will be pushed slowly and will be handled in parallel for validity checking the updates.

45 Likes

How much does this cost to sustain?

1 Like

The Dynamodb costs nothing and the server costs about 20$ a month.

2 Likes

I’m just intrigued by this. Dynamodb costs nothing? Haven’t exceeded the free plan? Also $20 for the web server? So I’m assuming you made a middle ground for communicating with the db server? Or the server you’re using comes with Dynamodb?

I’ve wanted to do something similar to this just for the project for a while now, and seeing you do it has been quite motivational.

Dynamodb hasn’t exceeded the free plan. The web server isn’t just the middle man, it has leaderboard services, dashboards, cache utilities and analytics, programmed by me.

2 Likes

Oh I see! Thats cool. Thanks for the quick replies. This has been really informative, and will consider using your service in the future.

2 Likes