Suphi's DataStore Module

apenzijncoolenleuk1 · September 16, 2024, 6:46am

OK, thanks for the clear reply.

What would be the best way to deal with these outages that occur while players are in the server?

Right now, I have the following state change listener that just retries opening the datastore (from your tutorial video):

local function keepTryingToOpen(dataStore)
	while dataStore.State == false do
		if dataStore:Open(DATA_TEMPLATE) ~= DataStoreModule.Response.Success then
			task.wait(5)
		end
	end
end
local function stateChanged(_, dataStore)
	if dataStore.State == false then
		keepTryingToOpen(dataStore)
	end
end

Is this the recommended way to deal with in-game outages?
I always do the following checks on the datastore object:

if dataStore.State ~= true then
	return false
end

Assuming a player’s session never succeeds in reopening from the micro outage, wouldn’t this lead to worse data loss as the player continues playing?

I think ProfileService recommends kicks the player in that case, though I could be wrong.

5uphi · September 16, 2024, 12:51pm

That’s all you can do just keep trying to reopen until roblox comes back online

if you watch my advance video i show you how to notify the player when there datastore closes

i personally would not kick them from the game
id just show a notification on there screen and still let them interact with parts of the game that does not use the datastore

but your free to handle it anyway you like

apenzijncoolenleuk1 · September 22, 2024, 7:36am

Would using ‘find’ in MarketplaceService.ProcessReceipt not cause any issues when this gets called on join? (from your basics video)

local dataStore = DataStoreModule.find("Player", receiptInfo.PlayerId)

From the roblox docs they say the callback is only ran when the player joins or when they buy another DevProduct. Worst case situation I can imagine is that the player data setup (under PlayerAdded), runs after the ProcessReceipt callback, making it so you’re always stuck until buying a new DevProduct.

5uphi · September 22, 2024, 1:47pm

Your right Roblox does not state if ProcessReceipt will fire after PlayerAdded and even if it did the datastore state might still be false if it fires to quickly after playeradded

so in order to solve this problem we could do

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

it should be safe to use open here because PlayerRemoving should fire after ProcessReceipt causing the datastore to still get closed

if you want to be extra safe you could also record the purchases

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
  
    if dataStore.Value.Purchases[receiptInfo.PurchaseId] then
        -- the item was already given lets not give it again
        return Enum.ProductPurchaseDecision.PurchaseGranted
    end

    -- set this so other servers don't give the item again
    dataStore.Value.Purchases[receiptInfo.PurchaseId] = true 
    dataStore.Value.Coins += 1

    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        -- remove this so that when ProcessReceipt fires again we give them the coin the next time
        dataStore.Value.Purchases[receiptInfo.PurchaseId] = nil
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

apenzijncoolenleuk1 · September 22, 2024, 3:43pm

Ok that seems much more robust.
I’ll use the ‘.new()’ instead.

Worst-case scenario is that player has to wait 5 mins or so right (using default settings)? Their session still being ‘open’ on another server.

5uphi · September 22, 2024, 8:42pm

if we read MarketplaceService | Documentation - Roblox Creator Hub it says

The player needs to be in the server for the callback to be invoked,
but does not need to be in the server for the result of the callback
to be recorded on the backend.

So my understanding is that ProcessReceipt can only be called while the player is on the server and that should be before the PlayerRemoving event

so even if you call datastore:Destroy() while your in the middle of opening the destroy will be scheduled by the task scheduler to take place after the open has finished so the worst case should never be possible

but if ProcessReceipt did get called after PlayerRemoving then the datastore will get stuck open until the server shuts down but I don’t believe this is possible if it does happen we should send a bug report to Roblox

apenzijncoolenleuk1 · September 23, 2024, 5:07am

I think the docs you mentioned are an indirect support for yielding within the callback.

This is apparently allowed (almost encouraged even). I just found this by searching on the forum.

So another solution could be to have a limited number of tries for ‘find()’ with small delays.

I say ‘solution’, but I doubt one call to ‘find()’ actually gives issues. The ProcessReceipt probably runs a little later than PlayerAdded.

Thanks for your help .

5uphi · September 23, 2024, 8:07am

you have to be carful when yielding because it could allow the player to exploit

MarketplaceService.ProcessReceipt = function(receiptInfo)
    -- try load datastore
    task.wait(10)
    -- try load datastore
    -- if datastore loads give coin
end

lets use this as a example

what a player could do is enter a server this would fire the ProcessReceipt event then they can quickly exit and enter another server because the first server is still yielding the second server will also fire ProcessReceipt then they could exit that server go back to the first server and then the coin will get adding to there datastore then once they get the coin quickly go back to the second server and because its still yielding it will also give them there coin after the 10 seconds has ended

the longer the yield the easier it would be to do and would also be possible to do with more then just 2 server

the way to solve this is to save dataStore.Value.Purchases[receiptInfo.PurchaseId] = true so that a server can not give the same item twice

or can be solved by not yielding

in this example

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.new("Player", receiptInfo.PlayerId)
    if dataStore:Open(template) ~= "Success" then return Enum.ProductPurchaseDecision.NotProcessedYet end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

there is a very small yield when we call dataStore:Open and dataStore:Save but because of session locking wont allow the second server to open the datastore until this server closes it also if we did not have session locking the yield should be so small that the chance of the player doing the above mentioned exploit would be most likely imposable

so I would not encourage yielding unless you save receiptInfo.PurchaseId or unless your keeping the datastore open so that you prevent the second server from yielding

apenzijncoolenleuk1 · September 23, 2024, 9:07am

OK thanks.
I understand the issue now. I’ll use ‘Open()’, since that seems like the easiest solution .

Quote from your previous reply:

I’m not sure why this would cause the session to stay locked until the server shuts down? Is that server actively ‘rejecting’ other requests to open the data?
In that case, what if the server crashes, and is unable to gracefully close the session. What would happen?

From what I understand the session stays locked for 5 minutes after a server crash.

5uphi · September 23, 2024, 9:44am

When you call DS:Open() the server will keep that session open until you call DS:Close() or DS:Destroy() or the server closes/crashes and it times out

if ProcessReceipt gets called after PlayerRemoving that means the DS:Destroy() that is inside PlayerRemoving was already called and now ProcessReceipt has just opened a brand new session but this brand new session will never get destroyed so it will stay open until the server closes/crashes or the player comes back into the same server and then exits again

here is another example that should work fine

MarketplaceService.ProcessReceipt = function(receiptInfo)
    local dataStore = DataStoreModule.find("Player", receiptInfo.PlayerId)
    if dataStore == nil or dataStore.State ~= true then
        dataStore = DataStoreModule.hidden("Player", receiptInfo.PlayerId)
        if dataStore:Open(template) ~= "Success" then
            dataStore:Destroy()
            return Enum.ProductPurchaseDecision.NotProcessedYet
        end
    end
    dataStore.Value.Coins += 1
    if dataStore:Save() == "Saved" then
        if dataStore.Hidden then dataStore:Destroy() end
        return Enum.ProductPurchaseDecision.PurchaseGranted
    else
        dataStore.Value.Coins -= 1
         if dataStore.Hidden then dataStore:Destroy() end
        return Enum.ProductPurchaseDecision.NotProcessedYet
    end
end

but I feel this is overkill I think ProcessReceipt should never get called after PlayerRemoving so the previous example should work fine

Obsentiel · October 1, 2024, 10:36pm

Hey there. Is it possible to use SDM without needing access to the DataStore API? I’m currently working at a private place file for my project over actual game place.

It would be great if there’s an alternative method that allows you to use the module - but it won’t save if it doesn’t have access to the DataStore API over giving errors and breaking the entire code.

5uphi · October 1, 2024, 10:44pm

set
SaveInterval = 0
and
SaveOnClose = false

or just dont open()

Obsentiel · October 2, 2024, 4:49am

How would I be able to use the module if I don’t Open()? Also, it seems to occur when constructing new. I believe this module wasn’t made compatible without the use of the DataStore API.

5uphi · October 3, 2024, 8:57am

local DataStoreModule = require(11671168253)

local template = {
    Level = 0,
    Coins = 0,
    Inventory = {},
    DeveloperProducts = {},
}

-- This function is currently not used
local function StateChanged(state, dataStore)
    while dataStore.State == false do
        if dataStore:Open(template) ~= "Success" then task.wait(6) end
    end
end

game.Players.PlayerAdded:Connect(function(player)
    local dataStore = DataStoreModule.new("Player", player.UserId)

    -- dataStore.StateChanged:Connect(StateChanged) -- dont connect to statechanged here
    --StateChanged(dataStore.State, dataStore) -- dont try to open the datastore object

    -- Temporary
    datastore:Reconcile(template) -- reconcile the template onto datastore.Value
    dataStore.StateChanged:Fire(true, datastore) -- fire statechanged to true to trick other scripts
    -- End Temporary
end)

game.Players.PlayerRemoving:Connect(function(player)
    local dataStore = DataStoreModule.find("Player", player.UserId)
    if dataStore ~= nil then dataStore:Destroy() end
end)

now to use the datastore you do

local dataStore = DataStoreModule.find("Player", player.UserId)
if dataStore == nil then return end
-- if dataStore.State ~= true then return end -- temporarily remove state checks
dataStore.Value.Level += 1

or another option is to just make a make module that acts like the datastore module

local dataStores = {}

local module = {}
module.__index = module

function module.new(name, scope, key)
    if key == nil then key, scope = scope, "global" end
    local id = name .. "/" .. scope .. "/" .. key
    if dataStores[id] ~= nil then return dataStores[id] end
    local self = setmetatable({}, module)
    self.Id = id
    self.State = false
    dataStores[id] = self
    return self
end

function module.find(name, scope, key)
	if key == nil then key, scope = scope, "global" end
	local id = name .. "/" .. scope .. "/" .. key
	return dataStores[id]
end

function module:Open(template)
	if self..Value == nil then
		self.Value = Clone(template)
	elseif type(self.Value) == "table" and type(template) == "table" then
		Reconcile(self.Value, template)
	end
    self.State = true
end

function module:Destroy()
    dataStores[self.Id] = nil
    self.State = nil
end

function Clone(original)
	if type(original) ~= "table" then return original end
	local clone = {}
	for index, value in original do clone[index] = Clone(value) end
	return clone
end

function Reconcile(target, template)	
	for index, value in template do
		if type(index) == "number" then continue end
		if target[index] == nil then
			target[index] = Clone(value)
		elseif type(target[index]) == "table" and type(value) == "table" then
			Reconcile(target[index], value)
		end
	end
end

return module

or another option is to modify the original module

Lock = function(dataStore, attempts)
	local success, value, id, lockTime, lockInterval, lockAttempts = nil, nil, nil, nil, dataStore.__public.LockInterval, dataStore.__public.LockAttempts
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		lockTime = os.clock()
		--success, value = pcall(dataStore.MemoryStore.UpdateAsync, dataStore.MemoryStore, "Id", function(value) id = value return if id == nil or id == dataStore.__public.UniqueId then dataStore.__public.UniqueId else nil end, lockInterval * lockAttempts + 30)
		success, value = true, "LockId" -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if value == nil then return "Locked", id end
	dataStore.LockTime = lockTime + lockInterval * lockAttempts
	dataStore.ActiveLockInterval = lockInterval
	dataStore.__public.AttemptsRemaining = lockAttempts
	return "Success"
end

Unlock = function(dataStore, attempts)
	local success, value, id = nil, nil, nil
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		--success, value = pcall(dataStore.MemoryStore.UpdateAsync, dataStore.MemoryStore, "Id", function(value) id = value return if id == dataStore.__public.UniqueId then dataStore.__public.UniqueId else nil end, 0)
		success, value = true, "LockId" -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if value == nil and id ~= nil then return "Locked", id end
	return "Success"
end

Load = function(dataStore, attempts)
	local success, value, info = nil, nil, nil
	for i = 1, attempts do
		if i > 1 then task.wait(1) end
		--success, value, info = pcall(dataStore.DataStore.GetAsync, dataStore.DataStore, dataStore.__public.Key)
		success, value = true, nil -- ADD this
		if success == true then break end
	end
	if success == false then return "Error", value end
	if info == nil then
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = {}, {}, 0, 0, ""
	else
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = info:GetMetadata(), info:GetUserIds(), info.CreatedTime, info.UpdatedTime, info.Version
	end
	if type(dataStore.__public.Metadata.Compress) ~= "table" then
		dataStore.__public.Value = value
	else
		dataStore.__public.CompressedValue = value
		local decimals = 10 ^ (dataStore.__public.Metadata.Compress.Decimals or 3)
		dataStore.__public.Value = Decompress(dataStore.__public.CompressedValue, decimals)
	end
	return "Success"
end

Save = function(proxy, attempts)
	local dataStore = getmetatable(proxy)
	local deltaTime = os.clock() - dataStore.SaveTime
	if deltaTime < dataStore.__public.SaveDelay then task.wait(dataStore.__public.SaveDelay - deltaTime) end
	dataStore.__public.Saving:Fire(dataStore.__public.Value, proxy)
	local success, value, info = nil, nil, nil
	if dataStore.__public.Value == nil then
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value, info = pcall(dataStore.DataStore.RemoveAsync, dataStore.DataStore, dataStore.__public.Key)
			success = true -- ADD this
			if success == true then break end
		end
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.__public.Metadata, dataStore.__public.UserIds, dataStore.__public.CreatedTime, dataStore.__public.UpdatedTime, dataStore.__public.Version = {}, {}, 0, 0, ""
	elseif type(dataStore.__public.Metadata.Compress) ~= "table" then
		dataStore.Options:SetMetadata(dataStore.__public.Metadata)
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value = pcall(dataStore.DataStore.SetAsync, dataStore.DataStore, dataStore.__public.Key, dataStore.__public.Value, dataStore.__public.UserIds, dataStore.Options)
			success = true -- ADD this
			if success == true then break end
		end	
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.__public.Version = value
	else
		local level = dataStore.__public.Metadata.Compress.Level or 2
		local decimals = 10 ^ (dataStore.__public.Metadata.Compress.Decimals or 3)
		local safety = if dataStore.__public.Metadata.Compress.Safety == nil then true else dataStore.__public.Metadata.Compress.Safety
		dataStore.__public.CompressedValue = Compress(dataStore.__public.Value, level, decimals, safety)
		dataStore.Options:SetMetadata(dataStore.__public.Metadata)
		for i = 1, attempts do
			if i > 1 then task.wait(1) end
			--success, value = pcall(dataStore.DataStore.SetAsync, dataStore.DataStore, dataStore.__public.Key, dataStore.__public.CompressedValue, dataStore.__public.UserIds, dataStore.Options)
			success = true -- ADD this
			if success == true then break end
		end
		if success == false then dataStore.__public.Saved:Fire("Error", value, proxy) return "Error", value end
		dataStore.Version = value
	end
	dataStore.SaveTime = os.clock()
	dataStore.__public.Saved:Fire("Saved", dataStore.__public.Value, proxy)
	return "Saved", dataStore.__public.Value
end