Such a welcome change!! For years this has plagued me and caused so much confusion as to why so many data store requests were being put into a queue, making me think I was pushing the rate limits too hard, yet it turns out after all this time, there has been a 6 second write cooldown for the same key!!!
This occasionally happened since auto-saves and saving on player leave can happen within that same timeframe, so for servers that have been up for a while, it would have quite a few of them. I’ve implemented custom queues for this since, but I’m sure it’d be even more complex if I had to write to the same key from multiple servers.
Excellent improvement to the DataStore Service. I can see this update fixing a plethora of issues across a great number of experiences, especially ones with trading features. Thank you guys for implementing the change.
Data stores became better than before and that’s very great update in my opinion, but does this now mean that data is prevented from being lost from players?
Can we expect to see the same functionality with reading a Data Store key?
Currently a read will cache the result and subsequent reads within the limit interval will return the cache instead of making an actual call to the Data Store. Sometimes data need to be altered based on previously existing data. If the read cache cooldown is still at 5 seconds, then it seems like UpdateAsync() may attempt to update stagnant data within the read cache even if a previous write executed successfully.
Other than that, this is a great update.
[EDIT]
I’m fairly confident there is a read cache for UpdateAsync(), but UpdateAsync() performs a read of the most current version of the key. Didn’t even realize the contradiction until now.
[EDIT 2]
It’s not a contradiction, it does one or the other depending if there is an active read cache cooldown.
This is great! Even though I’ve never had issues with data saving as I use both DataStore/OrderedDataStore to save different versions of the player’s data based on the request time (os.time()) so the player can access different versions of their data, or in case there’s an error they can just go back to an old version of their data.
DataStore2 is a datastore-wrapper. So yes it does apply to that as well. Infact I accidently set my auto-save to 6 seconds and didnt receive a warning.
Currently the :GetAsync() method will retrieve a cached value set for a period of 5 seconds. Are there any plans to change this behaviour as well? It would be valuable to have clarity on this aspect, especially given the recent changes.
While the removal of cooldown for writes to the same key is indeed a substantial improvement, it still necessitates the implementation of pessimistic locking mechanisms such as session locks. This is crucial to prevent players from inadvertently overriding data with their cached versions.
There are also opportunistic locking mechanisms like the MVCC protocol but I don’t know how well they work in practice in this scenario so it would be helpful to know if there are any additional considerations or best practices you could recommend in this regard.
On a related note, I recall our conversation from a few years ago, where you expressed the desire to explore the inclusion of batch writes and transactions. Enabling developers to bundle multiple write operations into atomic transactions but were constrained by limited resources at the time. Given the recent progress made and the positive changes being implemented, I would like to emphasize the value of introducing native support for batch writes and/or transactions. These additions would not only fulfill the aspirations we discussed previously but also serve as a testament to your dedication to continually enhance the Datastore and provide developers with powerful tools to meet their evolving needs for reliability of their experiences.
No offence, but the way this is worded worries me, the fact that you state “(for 5 seconds)” implies to me that this “improvement” will simply be making the cooldown less, however it would be much more preferable to simply not have it at all if possible. It really hurts my experience in it’s current state.
Years ago we added session locking in order to prevent a duping exploit that occurred when a player left a game after trading, and rejoined quickly. Basically they managed to load their date in a new server before the old server saved their data.
I don’t recall how it was related, but back then I concluded this duping method was made possible by the 6s cooldown. Anyway, does this update mean we won’t have to bother session locking anymore?
Sorry my previous message is confusing. Just to be clear, we will not reducing the timeout/cooldown from 5s to say 1s and call it an improvement. On the other hand, we probably won’t be able to simply remove the cache as that could potentially break some experiences.
Are there any plans to establish focus groups regarding DataStore aimed at fostering communication between developers and engineers?
We have a lot of different use cases and it would be highly beneficial to have a place where we can engage in meaningful discussions and provide feedback regarding the API, considering both its usability and practical implications.
Also I forgot to ask, will you improve the read cache in general or just for :GetAsync()? If :IncrementAsync() and :UpdateAsync() still invokes the read cache, how does it work with regards to this update.
For example, if I call :UpdateAsync() twice in lock-step will the second :UpdateAsync() return a cached value, i.e. the first value set by :UpdateAsync()? It sounds pretty counterintuitive that it would write to the datastore but the function itself will return an invalid value…
Is there any way to add a threshold to reads performed in quick succession; Once the threshold is met the read request falls back on the cache?
Maybe Data Stores need to be separated to include data that are only accessed from one server at a time and data that are accessed from multiple servers simultaneously. Seems like this could be a road to promote natural session locking; it may then be applicable to update the read cache with every write request since at that point the most updated version of data are known.
The reason why the GetAsync cache exists is for legacy reasons afaik, it allows you to do this and still only use up 1 request:
local value = DataStore:GetAsync("key")["value"]
local value2 = DataStore:GetAsync("key")["value2"]
That is legacy code, not very clean or readable, nor does it provide much ‘use’ to GetAsync caching, although removing the cache without any extra parameter would cause the above sample to use 2 requests rather than 1, possibly detrimental to the quota.
For years this has plagued me and caused so much confusion as to why so many data store requests were being put into a queue, making me think I was pushing the rate limits too hard, yet it turns out after all this time, there has been a 6 second write cooldown for the same key!!!