Suphi's DataStore Module

Resources Community Resources
#289

OK, thanks for the clear reply.

What would be the best way to deal with these outages that occur while players are in the server?

Right now, I have the following state change listener that just retries opening the datastore (from your tutorial video):

local function keepTryingToOpen(dataStore)
	while dataStore.State == false do
		if dataStore:Open(DATA_TEMPLATE) ~= DataStoreModule.Response.Success then
			task.wait(5)
		end
	end
end
local function stateChanged(_, dataStore)
	if dataStore.State == false then
		keepTryingToOpen(dataStore)
	end
end

Is this the recommended way to deal with in-game outages?
I always do the following checks on the datastore object:

if dataStore.State ~= true then
	return false
end

Assuming a player’s session never succeeds in reopening from the micro outage, wouldn’t this lead to worse data loss as the player continues playing?

I think ProfileService recommends kicks the player in that case, though I could be wrong.

#290

That’s all you can do just keep trying to reopen until roblox comes back online

if you watch my advance video i show you how to notify the player when there datastore closes

i personally would not kick them from the game
id just show a notification on there screen and still let them interact with parts of the game that does not use the datastore

but your free to handle it anyway you like

3 Likes
#291

Would using ‘find’ in MarketplaceService.ProcessReceipt not cause any issues when this gets called on join? (from your basics video)

local dataStore = DataStoreModule.find("Player", receiptInfo.PlayerId)

From the roblox docs they say the callback is only ran when the player joins or when they buy another DevProduct. Worst case situation I can imagine is that the player data setup (under PlayerAdded), runs after the ProcessReceipt callback, making it so you’re always stuck until buying a new DevProduct.

#292

Your right Roblox does not state if ProcessReceipt will fire after PlayerAdded and even if it did the datastore state might still be false if it fires to quickly after playeradded

so in order to solve this problem we could do

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

it should be safe to use open here because PlayerRemoving should fire after ProcessReceipt causing the datastore to still get closed

if you want to be extra safe you could also record the purchases

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
  
    if dataStore.Value.Purchases[receiptInfo.PurchaseId] then
        -- the item was already given lets not give it again
        return Enum.ProductPurchaseDecision.PurchaseGranted
    end

    -- set this so other servers don't give the item again
    dataStore.Value.Purchases[receiptInfo.PurchaseId] = true 
    dataStore.Value.Coins += 1

    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        -- remove this so that when ProcessReceipt fires again we give them the coin the next time
        dataStore.Value.Purchases[receiptInfo.PurchaseId] = nil
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end
2 Likes
#293

Ok that seems much more robust.
I’ll use the ‘.new()’ instead.

Worst-case scenario is that player has to wait 5 mins or so right (using default settings)? Their session still being ‘open’ on another server.

#294

if we read MarketplaceService | Documentation - Roblox Creator Hub it says

The player needs to be in the server for the callback to be invoked,
but does not need to be in the server for the result of the callback
to be recorded on the backend.

So my understanding is that ProcessReceipt can only be called while the player is on the server and that should be before the PlayerRemoving event

so even if you call datastore:Destroy() while your in the middle of opening the destroy will be scheduled by the task scheduler to take place after the open has finished so the worst case should never be possible

but if ProcessReceipt did get called after PlayerRemoving then the datastore will get stuck open until the server shuts down but I don’t believe this is possible if it does happen we should send a bug report to Roblox

2 Likes
#295

I think the docs you mentioned are an indirect support for yielding within the callback.

This is apparently allowed (almost encouraged even). I just found this by searching on the forum.

So another solution could be to have a limited number of tries for ‘find()’ with small delays.

I say ‘solution’, but I doubt one call to ‘find()’ actually gives issues. The ProcessReceipt probably runs a little later than PlayerAdded.

Thanks for your help :+1:.

#296

you have to be carful when yielding because it could allow the player to exploit

MarketplaceService.ProcessReceipt = function(receiptInfo)
    -- try load datastore
    task.wait(10)
    -- try load datastore
    -- if datastore loads give coin
end

lets use this as a example

what a player could do is enter a server this would fire the ProcessReceipt event then they can quickly exit and enter another server because the first server is still yielding the second server will also fire ProcessReceipt then they could exit that server go back to the first server and then the coin will get adding to there datastore then once they get the coin quickly go back to the second server and because its still yielding it will also give them there coin after the 10 seconds has ended

the longer the yield the easier it would be to do and would also be possible to do with more then just 2 server

the way to solve this is to save dataStore.Value.Purchases[receiptInfo.PurchaseId] = true so that a server can not give the same item twice

or can be solved by not yielding

in this example

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

there is a very small yield when we call dataStore:Open and dataStore:Save but because of session locking wont allow the second server to open the datastore until this server closes it also if we did not have session locking the yield should be so small that the chance of the player doing the above mentioned exploit would be most likely imposable

so I would not encourage yielding unless you save receiptInfo.PurchaseId or unless your keeping the datastore open so that you prevent the second server from yielding

2 Likes
#297

OK thanks.
I understand the issue now. I’ll use ‘Open()’, since that seems like the easiest solution :+1:.

Quote from your previous reply:

I’m not sure why this would cause the session to stay locked until the server shuts down? Is that server actively ‘rejecting’ other requests to open the data?
In that case, what if the server crashes, and is unable to gracefully close the session. What would happen?

From what I understand the session stays locked for 5 minutes after a server crash.

1 Like
#298

When you call DS:Open() the server will keep that session open until you call DS:Close() or DS:Destroy() or the server closes/crashes and it times out

if ProcessReceipt gets called after PlayerRemoving that means the DS:Destroy() that is inside PlayerRemoving was already called and now ProcessReceipt has just opened a brand new session but this brand new session will never get destroyed so it will stay open until the server closes/crashes or the player comes back into the same server and then exits again

here is another example that should work fine

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.find("Player", receiptInfo.PlayerId)
    if dataStore == nil or dataStore.State ~= true then
        dataStore = DataStoreModule.hidden("Player", receiptInfo.PlayerId)
        if dataStore:Open(template) ~= "Success" then
            dataStore:Destroy()
            return Enum.ProductPurchaseDecision.NotProcessedYet
        end
    end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        if dataStore.Hidden then dataStore:Destroy() end
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
         if dataStore.Hidden then dataStore:Destroy() end
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

but I feel this is overkill I think ProcessReceipt should never get called after PlayerRemoving so the previous example should work fine

2 Likes
#299

Hey there. Is it possible to use SDM without needing access to the DataStore API? I’m currently working at a private place file for my project over actual game place.

image
image921×50 13.9 KB

It would be great if there’s an alternative method that allows you to use the module - but it won’t save if it doesn’t have access to the DataStore API over giving errors and breaking the entire code.

#300

set
SaveInterval = 0
and
SaveOnClose = false

or just dont open()

#301

How would I be able to use the module if I don’t Open()? Also, it seems to occur when constructing new. I believe this module wasn’t made compatible without the use of the DataStore API.

#302 
local DataStoreModule = require(11671168253)

local template = {
    Level = 0,
    Coins = 0,
    Inventory = {},
    DeveloperProducts = {},
}

-- This function is currently not used
local function StateChanged(state, dataStore)
    while dataStore.State == false do
        if dataStore:Open(template) ~= "Success" then task.wait(6) end
    end
end

game.Players.PlayerAdded:Connect(function(player)
    local dataStore = DataStoreModule.new("Player", player.UserId)

    -- dataStore.StateChanged:Connect(StateChanged) -- dont connect to statechanged here
    --StateChanged(dataStore.State, dataStore) -- dont try to open the datastore object

    -- Temporary
    datastore:Reconcile(template) -- reconcile the template onto datastore.Value
    dataStore.StateChanged:Fire(true, datastore) -- fire statechanged to true to trick other scripts
    -- End Temporary
end)

game.Players.PlayerRemoving:Connect(function(player)
    local dataStore = DataStoreModule.find("Player", player.UserId)
    if dataStore ~= nil then dataStore:Destroy() end
end)

now to use the datastore you do

local dataStore = DataStoreModule.find("Player", player.UserId)
if dataStore == nil then return end
-- if dataStore.State ~= true then return end -- temporarily remove state checks
dataStore.Value.Level += 1

or another option is to just make a make module that acts like the datastore module

local dataStores = {}

local module = {}
module.__index = module

function module.new(name, scope, key)
    if key == nil then key, scope = scope, "global" end
    local id = name .. "/" .. scope .. "/" .. key
    if dataStores[id] ~= nil then return dataStores[id] end
    local self = setmetatable({}, module)
    self.Id = id
    self.State = false
    dataStores[id] = self
    return self
end

function module.find(name, scope, key)
	if key == nil then key, scope = scope, "global" end
	local id = name .. "/" .. scope .. "/" .. key
	return dataStores[id]
end

function module:Open(template)
	if self..Value == nil then
		self.Value = Clone(template)
	elseif type(self.Value) == "table" and type(template) == "table" then
		Reconcile(self.Value, template)
	end
    self.State = true
end

function module:Destroy()
    dataStores[self.Id] = nil
    self.State = nil
end

function Clone(original)
	if type(original) ~= "table" then return original end
	local clone = {}
	for index, value in original do clone[index] = Clone(value) end
	return clone
end

function Reconcile(target, template)	
	for index, value in template do
		if type(index) == "number" then continue end
		if target[index] == nil then
			target[index] = Clone(value)
		elseif type(target[index]) == "table" and type(value) == "table" then
			Reconcile(target[index], value)
		end
	end
end

return module

or another option is to modify the original module

Lock = function(dataStore, attempts)
	local success, value, id, lockTime, lockInterval, lockAttempts = nil, nil, nil, nil, dataStore.__public.LockInterval, dataStore.__public.LockAttempts
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		lockTime = os.clock()
		--success, value = pcall(dataStore.MemoryStore.UpdateAsync, dataStore.MemoryStore, "Id", function(value) id = value return if id == nil or id == dataStore.__public.UniqueId then dataStore.__public.UniqueId else nil end, lockInterval * lockAttempts + 30)
		success, value = true, "LockId" -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if value == nil then return "Locked", id end
	dataStore.LockTime = lockTime + lockInterval * lockAttempts
	dataStore.ActiveLockInterval = lockInterval
	dataStore.__public.AttemptsRemaining = lockAttempts
	return "Success"
end

Unlock = function(dataStore, attempts)
	local success, value, id = nil, nil, nil
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		--success, value = pcall(dataStore.MemoryStore.UpdateAsync, dataStore.MemoryStore, "Id", function(value) id = value return if id == dataStore.__public.UniqueId then dataStore.__public.UniqueId else nil end, 0)
		success, value = true, "LockId" -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if value == nil and id ~= nil then return "Locked", id end
	return "Success"
end

Load = function(dataStore, attempts)
	local success, value, info = nil, nil, nil
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		--success, value, info = pcall(dataStore.DataStore.GetAsync, dataStore.DataStore, dataStore.__public.Key)
		success, value = true, nil -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if info == nil then
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = {}, {}, 0, 0, ""
	else
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = info:GetMetadata(), info:GetUserIds(), info.CreatedTime, info.UpdatedTime, info.Version
	end
	if type(dataStore.__public.Metadata.Compress) ~= "table" then
		dataStore.__public.Value = value
	else
		dataStore.__public.CompressedValue = value
		local decimals = 10 ^ (dataStore.__public.Metadata.Compress.Decimals or 3)
		dataStore.__public.Value = Decompress(dataStore.__public.CompressedValue, decimals)
	end
	return "Success"
end

Save = function(proxy, attempts)
	local dataStore = getmetatable(proxy)
	local deltaTime = os.clock() - dataStore.SaveTime
	if deltaTime < dataStore.__public.SaveDelay then task.wait(dataStore.__public.SaveDelay - deltaTime) end
	dataStore.__public.Saving:Fire(dataStore.__public.Value, proxy)
	local success, value, info = nil, nil, nil
	if dataStore.__public.Value == nil then
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value, info = pcall(dataStore.DataStore.RemoveAsync, dataStore.DataStore, dataStore.__public.Key)
			success = true -- ADD this
			if success == true then break end
		end
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = {}, {}, 0, 0, ""
	elseif type(dataStore.__public.Metadata.Compress) ~= "table" then
		dataStore.Options:SetMetadata(dataStore.__public.Metadata)
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value = pcall(dataStore.DataStore.SetAsync, dataStore.DataStore, dataStore.__public.Key, dataStore.__public.Value, dataStore.__public.UserIds, dataStore.Options)
			success = true -- ADD this
			if success == true then break end
		end	
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.__public.Version = value
	else
		local level = dataStore.__public.Metadata.Compress.Level or 2
		local decimals = 10 ^ (dataStore.__public.Metadata.Compress.Decimals or 3)
		local safety = if dataStore.__public.Metadata.Compress.Safety == nil then true else dataStore.__public.Metadata.Compress.Safety
		dataStore.__public.CompressedValue = Compress(dataStore.__public.Value, level, decimals, safety)
		dataStore.Options:SetMetadata(dataStore.__public.Metadata)
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value = pcall(dataStore.DataStore.SetAsync, dataStore.DataStore, dataStore.__public.Key, dataStore.__public.CompressedValue, dataStore.__public.UserIds, dataStore.Options)
			success = true -- ADD this
			if success == true then break end
		end
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.Version = value
	end
	dataStore.SaveTime = os.clock()
	dataStore.__public.Saved:Fire("Saved", dataStore.__public.Value, proxy)
	return "Saved", dataStore.__public.Value
end
1 Like
#303

Hi

I’m saving character-related data per player (inventory items). When the player leaves I would like them to remain as an NPC in the world so they can still be attacked by others.

If they get killed (after the player left), they should lose their inventory contents. What would be the correct way of doing this?

I was thinking to edit the data, but the session is closed on player leaving, so that probably won’t work…
Then there’s also the issue where they would join another server while their data is being set in the other server.

Would keeping the session locked for like 30s if the player combat-logged cause any issues? If they didn’t combat log, it would instantly unload the session so the player can join another server faster.

#304

if a player leaves and enters another server can they still die on the original server? if no then I think

server A keeps there session open/locked for 30 seconds
then server B does

if datastoreIsLocked then
    datastore:Queue("Close", 5) -- send a message that lasts up to 5 seconds asking server A to close the session
    task.wait(5) -- wait a bit for the other server to get the message and close the datastore
end

another option is to close the datastore then server A uses updateasync on a separate datastore to tell the next server that they died the problem with this is that if the player enters after they die on server B then it will miss the update to the datastore

1 Like
#305

The player can die on the server while they joined another server. I think I should just keep the session locked for longer?

I have a loadingscreen that waits until the player data loads. As soon as the other server is done I close the session, and that would allow the player to join that other server right?

Having them wait for some time after combat logging is acceptable, since it’s kinda on them for having combat logged.

I’m just worried what would happen if the player joins the same server after combat logging.

#306

its kind of hard to say there are a few scenarios you need to handle
if the player never logs back on
if the player logs back onto the same server
if the player logs back onto a different server
if a player dies while there on another server
if a player dies while there on the same server

and because I don’t know how you want to handle each of these scenarios I cant really make a recommendation

1 Like
#307

I’ll assume my proposed solution of keeping the session locked for 1 extra minute after a player leaves the server.

First 3 cases would just be that they lost their inventory items. The server clears their data so it’s ready for the next join. These shouldn’t be an issue at all from what I understand.

4th case would put the player in a loading screen until the other server is done setting their data (session closed). Then the player would be able to play from the new server.

Last case seems to be the complex one. It should probably prevent the player from playing until after the 60s. Keep them in a loading screen until after the server is done with their character. Server wouldn’t close the session in this case, so the player can exit the loading screen as soon as their data is updated.

Would something like that work?
I think the first 3 cases are no problem since the ‘old’ server essentially has free roam over the data. No other server trying to unlock it.

Also the maximum time I can keep the session open probably affects the failsafe where another server tries opening a deadlocked session?
What’s the maximum time I would be able to wait?

#308

ive never seen that display for data before (referring to your picture), is that a roblox feature of datastore?