Details on DataStoreService for Advanced Developers

[Avigant]

Are you sure that set requests do not set the read cache? I believe they do.

local services = {
	data_store = game:GetService("DataStoreService")
}

--local data_store = services.data_store:GetDataStore("name_0")
--local data_store = services.data_store:GetDataStore("name_1")
local data_store = services.data_store:GetDataStore("name_2")

game.Players.PlayerAdded:Connect(function(player)
	wait(10)
	local value = data_store:GetAsync("key")
	data_store:SetAsync("key", value == nil and 1 or value + 1)
end)

while true do
	wait(0.5)
	local value = data_store:GetAsync("key")
	game.ReplicatedStorage.RemoteEvent:FireAllClients("Value at: " .. os.time() .. " is " .. tostring(value))
end

game.ReplicatedStorage.RemoteEvent.OnClientEvent:Connect(function(message)
	print(message)
end)

Assuming wait() calls are not taking much longer than the cache time for the thread to be resumed (which they shouldn’t be and aren’t in my testing), this code should always print that the value is the initial value, and never get the incremented value since I am resetting the cache cooldown every ~0.5 seconds. However, in my testing, it does receive the new value.

Perhaps I’m missing something obvious, but it sure seems like GlobalDataStore:SetAsync() does indeed set the read cache for the key.

That aside, great thread! Very helpful. Shame that DataStores are so needlessly complex due to the odd caching mechanism.

[buildthomas]

All key writes that you do on a particular server will be immediately available on that server. Perhaps I could describe this more clearly, but with “setting the cache” I mean that it sets the cache time (the value is irrelevant in that description). For example, doing an UpdateAsync and then a GetAsync afterwards makes the GetAsync not consume any budget, since UpdateAsync set the cache time. SetAsync does not set the get cache time, so the GetAsync call after that will consume budget, even though the server should know what the most recent value is because of the recent SetAsync.

[Validark]

Ah, the things we learn when we try to reverse engineer and emulate a black-box system. I know what that’s like. Really great work.

[buildthomas]

DataStore Data Limit Increase

Since launching Data Stores years ago, the service limited storage to a maximum of 256KB data per key. Since then, games and experiences have only grown to be bigger and more immersive, and developers have had to implement some workarounds in order to save an ever-growing amount of player data. To address this, as of today, we are increasing the DataStores per-key data limit from 256KB to 4MB! Note: Request limits for reads and writes are going to stay the same, as is the per-key throughput li…

Updated topic with new max size limit.

[Anaminus]

The limit is actually 2^22-1, or 4,194,303. You forgot to include the encoded double-quotes from the string you used to check the size limit.

[buildthomas]

@Fractality can you confirm? ^

[nezuo]

I noticed an inaccuracy. In this, you state that the documentation is wrong about the 50 character limit for keys. I don’t know if this was true in the past, but it doesn’t seem to be anymore.

In this image, you can see the error happens on line 14 and not 13.

[Fractality]

You can query this directly by the looks of it.

print(settings():GetFVariable("DataStoreMaxValueSize"))
4194304

Which makes sense - that’s 4MB. The real limit is probably that minus one.

[buildthomas]

@Nezuo

I noticed an inaccuracy. In this, you state that the documentation is wrong about the 50 character limit for keys.

Yep looks like they fixed this. Edited the post.

[nezuo]

The write cooldown says it is 6 seconds, however, when I try to use UpdateAsync every 6 seconds, it gives a warning.

Code in question:

while os.clock() - Start < TIME_BEFORE_FORCE_STEAL do
   print('trying')

   local LoadFuture = LoadProfileData(profileStore.DataStore, key)

   wait(6)

   if not LoadFuture:isResolved() then
      LoadFuture:await()
   end
end

[buildthomas]

It’s hard to see what’s happening here because your code sample is not self-contained, but you probably don’t want to wait exactly 6 seconds between requests. The network time taken for requests / for wait(6) will vary a bit so you could be hitting the server a few tens of milliseconds too early if you do it this way.

As far as I understand, the write time is set upon datastore call completion, and read upon datastore call invocation, so you need to wait more than 6 seconds in practice. You’re doing this with a future so that wait(6) is measured between the start times of the requests, not the span between a request ending and a request starting.

[nezuo]

If I waited for the call to be complete + 6 seconds, would that work?

[buildthomas]

I think that should work because then you’re sure the time between the request ending and starting is >=6 (since wait(6) guarantees >=6 seconds wait)

[KovarMachane]

Thanks a lot for this comprehensive material.
One question: Write cooldown 6 sec - it has its own queue? does it ever throw?
BTW, the read cache issue is very important to me. Has it not been fixed since your OP?
Ok, it’s three questions…

[buildthomas]

@KovarMachane

Write cooldown 6 sec - it has its own queue?

I believe there are only queues for every kind of operation. So write cooldown violations go in the same queue as writes out of budget (SetIncrementAsync).

@KovarMachane

does it ever throw?

If your queue is full (30 requests max) it can throw, yes.

@KovarMachane

BTW, the read cache issue is very important to me. Has it not been fixed since your OP?

I believe that was fixed:

Datastores: Caching is fundamentally broken (with details on how to fix logic-wise)

This issue has been resolved. Thanks for bringing this to our attention.

Let me update the thread accordingly

[RoyallyFlushed (Royal)]

I don’t think these limits are correct. With 10 players I am managing 1,380 requests for Set & Get. This means the rate is actually what this user posted. Request Limits on Data Store Errors and Limits page are not accurate

3 * ( 60 + ( 40 * 10 ) ) = 1,380.

[buildthomas]

I’m not fully convinced these rates are intended, see: Request Limits on Data Store Errors and Limits page are not accurate - #5 by buildthomas

If they do end up being intended and the official documentation is updated to reflect it, I will update this post.

The rate calculation is still the same, it’s just the multiplier seems to be 40 instead of 10 currently.

[RoyallyFlushed (Royal)]

It’s possible they might not have reset it back. It might still be good to put a little note somewhere on your post that it is currently 40 per player just so others know. It can always be removed if they decide to reset it back. I will try and get in contact with someone and find out if it’s permanent or not.

[buildthomas]

The note already exists (your reply).

I can update it in the main post here once I know more, I don’t want to accidentally mislead anyone into thinking they can design something that requires many more req/s than they will actually be permitted to long-term.

[UncertainLeo (Leo)]

I came across your post while making a game that’s datastore limit sensitive, and this was very helpful, thank you!
Sorry to revive this topic, but it would be helpful for anyone coming across nowadays it as I did.

And from my testing, I can confirm the limits @RoyallyFlushed mentioned are still in affect till now, therefore it’s probably intentional.

And it appears not only the per-player limit increased, but also the per-player rate in which the budget increases.
From my testing on GetAsync, I found that:
The base rate is still 60 per minute
The per-player rate is 40 per minute