Details on DataStoreService for Advanced Developers

[RuizuKun_Dev (Kitsune)]

@buildthomas

OnUpdate core functionality is broken

Is this going to be fixed anytime soon?

I thought I should ask because you might know or able to acquire news about this broken feature everyone is relying on and want fixed.

---

Very Informative Thread!

[Tiffblocks]

@Anaminus

Perhaps the string is (incorrectly) assumed to be unicode encoding rather than unencoded bytes?

This isn’t a bug, strings in data stores explicitly only store valid UTF-8.

[TheGuyWithAShortName (Sam Huckaby)]

I developed a global marketplace about a month ago for my game, but couldn’t ever ship it because it only worked if both players were in the same server (aka not global. Defeats the purpose) This OnUpdate bug seems to be the issue. No wonder. I hope this gets fixed soon. If it does, someone please @ me.

[buildthomas]

You can subscribe to notifications on the linked bug report and you’ll get an email/browser notification when someone replies there.

[Person299]

This is quite possibly the best guide I’ve ever read on anything related to ROBLOX.
Awesome job, I wish I had this years ago, it would’ve saved me a lot of time.

[ArtemisTheDeer (Artemis)]

Heads up, there’s a bug going around right now related to potential data loss if you try to call the same key in succession within a short period of time (<4s between attempts), more info here.

Data loss starting two days ago

Heads up, after talking to the amazing @RoughSphereBlox (Seriously, you’re the real MVP here), this was the issue with Vehicle Simulator’s data code (But likely applies to other games too that try repeated attempts to load data): If you make a call to getasync, it caches for 4s (Per the wiki documentation). If the request throttles/errors, and another request is made within that 4s period, getasync will return nil w/o erroring (Because nil was what was cached). Due to holiday traffic, this bug…

[Avigant]

Are you sure that set requests do not set the read cache? I believe they do.

local services = {
	data_store = game:GetService("DataStoreService")
}

--local data_store = services.data_store:GetDataStore("name_0")
--local data_store = services.data_store:GetDataStore("name_1")
local data_store = services.data_store:GetDataStore("name_2")

game.Players.PlayerAdded:Connect(function(player)
	wait(10)
	local value = data_store:GetAsync("key")
	data_store:SetAsync("key", value == nil and 1 or value + 1)
end)

while true do
	wait(0.5)
	local value = data_store:GetAsync("key")
	game.ReplicatedStorage.RemoteEvent:FireAllClients("Value at: " .. os.time() .. " is " .. tostring(value))
end

game.ReplicatedStorage.RemoteEvent.OnClientEvent:Connect(function(message)
	print(message)
end)

Assuming wait() calls are not taking much longer than the cache time for the thread to be resumed (which they shouldn’t be and aren’t in my testing), this code should always print that the value is the initial value, and never get the incremented value since I am resetting the cache cooldown every ~0.5 seconds. However, in my testing, it does receive the new value.

Perhaps I’m missing something obvious, but it sure seems like GlobalDataStore:SetAsync() does indeed set the read cache for the key.

That aside, great thread! Very helpful. Shame that DataStores are so needlessly complex due to the odd caching mechanism.

[buildthomas]

All key writes that you do on a particular server will be immediately available on that server. Perhaps I could describe this more clearly, but with “setting the cache” I mean that it sets the cache time (the value is irrelevant in that description). For example, doing an UpdateAsync and then a GetAsync afterwards makes the GetAsync not consume any budget, since UpdateAsync set the cache time. SetAsync does not set the get cache time, so the GetAsync call after that will consume budget, even though the server should know what the most recent value is because of the recent SetAsync.

[Validark]

Ah, the things we learn when we try to reverse engineer and emulate a black-box system. I know what that’s like. Really great work.

[buildthomas]

DataStore Data Limit Increase

Since launching Data Stores years ago, the service limited storage to a maximum of 256KB data per key. Since then, games and experiences have only grown to be bigger and more immersive, and developers have had to implement some workarounds in order to save an ever-growing amount of player data. To address this, as of today, we are increasing the DataStores per-key data limit from 256KB to 4MB! Note: Request limits for reads and writes are going to stay the same, as is the per-key throughput li…

Updated topic with new max size limit.

[Anaminus]

The limit is actually 2^22-1, or 4,194,303. You forgot to include the encoded double-quotes from the string you used to check the size limit.

[buildthomas]

@Fractality can you confirm? ^

[nezuo]

I noticed an inaccuracy. In this, you state that the documentation is wrong about the 50 character limit for keys. I don’t know if this was true in the past, but it doesn’t seem to be anymore.

In this image, you can see the error happens on line 14 and not 13.

[Fractality]

You can query this directly by the looks of it.

print(settings():GetFVariable("DataStoreMaxValueSize"))
4194304

Which makes sense - that’s 4MB. The real limit is probably that minus one.

[buildthomas]

@Nezuo

I noticed an inaccuracy. In this, you state that the documentation is wrong about the 50 character limit for keys.

Yep looks like they fixed this. Edited the post.

[nezuo]

The write cooldown says it is 6 seconds, however, when I try to use UpdateAsync every 6 seconds, it gives a warning.

Code in question:

while os.clock() - Start < TIME_BEFORE_FORCE_STEAL do
   print('trying')

   local LoadFuture = LoadProfileData(profileStore.DataStore, key)

   wait(6)

   if not LoadFuture:isResolved() then
      LoadFuture:await()
   end
end

[buildthomas]

It’s hard to see what’s happening here because your code sample is not self-contained, but you probably don’t want to wait exactly 6 seconds between requests. The network time taken for requests / for wait(6) will vary a bit so you could be hitting the server a few tens of milliseconds too early if you do it this way.

As far as I understand, the write time is set upon datastore call completion, and read upon datastore call invocation, so you need to wait more than 6 seconds in practice. You’re doing this with a future so that wait(6) is measured between the start times of the requests, not the span between a request ending and a request starting.

[nezuo]

If I waited for the call to be complete + 6 seconds, would that work?

[buildthomas]

I think that should work because then you’re sure the time between the request ending and starting is >=6 (since wait(6) guarantees >=6 seconds wait)

[KovarMachane]

Thanks a lot for this comprehensive material.
One question: Write cooldown 6 sec - it has its own queue? does it ever throw?
BTW, the read cache issue is very important to me. Has it not been fixed since your OP?
Ok, it’s three questions…