Pondering DataStores - is simple caching enough?

Help and Feedback Code Review
#1

Hi there, I posted recently on DataStore2 - this is a separate post asking for review on my own Data Stores interface implementation.

For a brief background, I’m a long time developer, new to Roblox. When I started down the road of storing data, the Data Store API quotas were an interesting problem to confront. Nothing new - APIs have quotas all the time but this was a bit unique in that most databases have quota/connection limitations that far exceed the Data Store limitations i.e. most applications the size of a typical Roblox game would never need to think about hitting connection or throttling issues with e.g. an auto-scaling SQL database.

Anyway, I did what any developer would do being new to the community and unaware that others had confronted this problem - I wrote my own solution :slight_smile:

It rests on caching the data store data “as long as needed” to not run into quotas, based on the number of data stores the game needs. Implementation below. It seems to work fine but has never encountered any kind of scale.

Later, I came across DataStore2 (linked above) and dug into the code to understand the differences. The first thing that strikes me is the complexity difference - my solution is much simpler.

I did post some questions I have on the DataStore2 implementation but that aside,

My default assumption is that mine is simpler because I have missed something or there is something I do not understand. Hence, this post :slight_smile:

If you have a minute to take a look and give your opinion, I’d love to hear it. NOTE: this is not intended to be a fool-proof implementation for others to use! There are statically defined numbers and caveats in the code that are specific to my game. I’ve called out those cases I’m aware of.

The question is, once those limitations were cleared up and the code configured in a way that was extensible to other games, what problems would it encounter in a game at scale? Or more broadly, should I continue down this road or have I misunderstood something and this implementation will fall apart due to (X)?

Thanks for your time.

local RunService = game:GetService("RunService")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local DataStoreService = game:GetService("DataStoreService")

local JunosTools = require(ReplicatedStorage:WaitForChild("Modules"):WaitForChild("JunosTools"))

--
-- The following is cache management for handling throttled updates to the data store
--

local CachedDs = {}

-- manage a local cache to throttle updates to the data store
local UpdateCache = {}
local UpdatingStoreLock = false
local StoreCache = {}
-- how often we write to the stores
-- THIS IS HIGHLY DEPENDENT ON THE NUMBER OF STORES YOU HAVE
CachedDs.UPDATE_FREQUENCY_SEC = 8

local Debug = false

---
--- Q update 
---
function CachedDs.queueSave(storeName, storeKey, value)
	if Debug then print("CachedDs.queueSave storeName "..storeName..", storeKey "..storeKey) end
	-- a better solution would be to write into an internal Q and pop those off in the update cache
	-- A Q SOLUTION WOULD BE REQUIRED FOR A ROBUST IMPLEMENTATION
	if UpdatingStoreLock then
		wait(0.3) -- not exactly deterministic but I think it should work
	end

	if not UpdateCache[storeName] then
		UpdateCache[storeName] = {}
	end

	if not StoreCache[storeName] then
		StoreCache[storeName] = DataStoreService:GetDataStore(storeName)
	end

	UpdateCache[storeName][storeKey] = value
end


--
-- Write the data
--
function persistToDS(store, key, val, throttle)
	if Debug then print("persistToDS key "..key) end
	if Debug then print("persistToDS val "..JunosTools.dumpOut(val)) end
	
	if throttle and throttle > 0 then
		if Debug then print("^^ throttled "..throttle.." waiting...") end
		wait(throttle)
	end
	local setSuccess, errorMessage = pcall(function()
		store:SetAsync(key, val)
	end)
	if not setSuccess then
		-- A ROBUST SOLUTOIN WOULD NEED TO RETRY
		warn('CachedDs - persistToDS ERROR '..errorMessage)
	end
end


--
-- Called when we do the actual persist to data store
-- 
local function checkCache()
	local storeCount = JunosTools.getTableLength(UpdateCache)
	-- return the total time we'll throttle writes
	local totalThrottle = 0
	if Debug then print("CachedDs checkCache - writing to "..storeCount.." stores") end

	if storeCount > 0 then
		UpdatingStoreLock = true
		if Debug then print("^^ -- storeCount > 0, lock set") end
		
		-- THIS SHOULD BE BASED ON WRITE FREQUENCY HARD-CODED ABOVE (+ TAKE INTO CONSIDERATION NUMBER OF STORES)
		local baseThrottleTime = 0.5
		local curStore = 0 -- track current store for throttle time - start with no throttle time
		
		for storeName, updateTable in pairs(UpdateCache) do
			local keyCount = JunosTools.getTableLength(updateTable)
			local curKey = 1 -- 1 because if curStore is 0, cancels out, else, start with base

			for key, val in pairs(updateTable) do
				local writeThrottle = baseThrottleTime * curStore * curKey
				totalThrottle += writeThrottle

				local runner = coroutine.wrap(persistToDS)
				runner(StoreCache[storeName], key, val, writeThrottle)
				
				curKey += 1
			end

			curStore += 1
		end

		UpdateCache = {}
		UpdatingStoreLock = false
		if Debug then print("^^ -- UpdateCache reset, lock unset") end
	end	
	return totalThrottle
end

-- track step count to throttle updates
local StepCount = 0
-- don't let multiple heartbeats fire updates at the same time (unlikely, but could be bad)
local HeartbeatLock = false

--
-- Check for when we do updates to the data store
-- IS HEARTBEAT THE WRONG THING??
-- 
RunService.Heartbeat:Connect(function(step)
	StepCount += step

	if math.fmod(math.ceil(StepCount), CachedDs.UPDATE_FREQUENCY_SEC) == 0 and not HeartbeatLock then
		HeartbeatLock = true
		StepCount = 0.1
		if Debug then print("CachedDs Heartbeat - check update cache") end
		local throttle = checkCache()
		wait(throttle)
		HeartbeatLock = false
	end
end)

--
-- Persist when server shutsdown to catch up anything left in the cache
--
game:BindToClose(function()
	if Debug then print("CachedDs checking cache on server close") end
	
	local minThrottle = 1
	local setThrottle = checkCache()
	
	if setThrottle < minThrottle then
		setThrottle = minThrottle
	end
	print("Waiting "..setThrottle.." seconds on shutdown to complete DB writes")
	wait(setThrottle)

	if setThrottle ~= 0 then
		print("CachedDs writes done!")
	end	
end)

return CachedDs
Pondering DataStores - Why DataStore2?