Datastore Listing API's Cursor

Hi Creators,

We have updated the Datastore ListKeysAsync() and ListDataStoresAsync() Lua APIs to support an optional cursor parameter so that it’s easier to enumerate all the datastores and keys in an experience.

Previously, you had to go through all the keys or datastores in one path or start over from the beginning, making the iteration process with a large number of keys or datastores vulnerable to disruptions like server shutdown or crashes. Additionally, you couldn’t easily distribute the workload across multiple servers. The cursor now allows you to halt or resume iterations, which means you can enumerate all the keys and datastores across multiple sessions or even servers. Some of the use cases include performing data schema migrations and deleting datastores.

Here are the sample code on how to resume enumerating your keys with a cursor:

local DataStoreService = game:GetService("DataStoreService")

-- Create and Populate Datastores
for count = 1, 70 do

    	local experienceStore = DataStoreService:GetDataStore("Store" .. 
tostring(count), count)
	        local success, errorMessage = pcall(function()
        		experienceStore:SetAsync("Value" .. tostring(count), count)
	  end)
	  if not success then
          	print(errorMessage)
	      end
     end
 
------- 
local listSuccess, pages = pcall(function()
            return DataStoreService:ListDataStoresAsync()
end)
if listSuccess then
       while true do
              local items = pages:GetCurrentPage()
              for _, ds in ipairs(items) do
			          print(ds.DataStoreName, "| Created:", ds.CreatedTime)
		end
		if pages.IsFinished then
			    break
               end
	       print("cursor: ", pages.Cursor)
-- fetches listdatastores but passes in the current cursor to get the next
-- set of datastores
		     pages = DataStoreService:ListDataStoresAsync("", 3, pages.Cursor)
	       end
     end

This code block iterates over all datastores using ListDataStoresAsync. Initially, we pass in the default “” for the cursor, but on subsequent calls, we use the cursor value that we can get from the current page.

As always, your feedback is invaluable for us to keep improving the service. Feel free to post any questions you may have. Our team will monitor the posts regularly and get back to you.

Happy building!
The Roblox Creator Services Team

112 Likes

This topic was automatically opened after 11 minutes.

Great change! One behaviour that is not exactly clear is what happens when keys are added, whilst iterating over the list with the cursor? If using an old cursor, will it keep the old state & not include the new key, and will it be guaranteed that all of the old keys will be included & not skipped?

Lastly, in order to be better able to distribute workload, it would be great if we could skip pages / generate a cursor for X pages in — is that currently possible / being considered?

Thanks.

9 Likes

This seems like a great way to allow us to browse our datastores a lot easier and with better load distribution.

Is this cursor already implemented for external access? e.g. can we make external APIs read from a cursor so that we can have a datastore editor connected from our local computer (or have something like cPanel’s phpMyAdmin that connects to roblox’s datastores)?

another example: If we encounter an issue regarding datastorage, can we send a cursor to the relevant key so that external tools can look into it further, instead of needing to pass potentially several dozen strings in regards to the keys we tried to edit?

1 Like

Honestly, it would be useful if the “prefix” search argument on ListKeysAsync was a regex search instead of a prefix

This would make filtering keys more granular, even if regex is a bit finnicky to grasp

5 Likes

Unfortunately Lua doesn’t use Regex. I made a topic a while back asking about Regex for Roblox, and it had to do with Regex having way more lines of code, complexity, and performance than the current filter or something like that.

2 Likes

cc @metatablecatmaid

The reason Lua doesn’t have regex is defined in the pattern matching article:

Unlike several other scripting languages, Lua does not use POSIX regular expressions (regexp) for pattern matching. The main reason for this is size: A typical implementation of POSIX regexp takes more than 4,000 lines of code. This is bigger than all Lua standard libraries together. In comparison, the implementation of pattern matching in Lua has less than 500 lines.

It would be possible for Roblox to support regex serverside, though, so that could be an option.

For now, you can design around this by prefixing your key with a search prefix.

8 Likes

Good update, but as a developer, its currently hard to get the player rank in a DataStore of millions of players, does this help this problem?

2 Likes

This would be from the C++/Web Server side when it comes to handling it.

regex in Lua probably isn’t needed if its doing a lookup on a dataserver

1 Like

Is it actually possible to “delete” a datastore ? My game has many datastores with no keys at all and it would be nice if there was a way to actually delete the datastore.

1 Like

Wonderful, can we also please get budget Enums for these operations? They are currently missing.

6 Likes

It sounds like you are looking for open cloud datastores API. Both listing datastores and entries have a cursor query parameter in the open cloud API. DataStore API | Roblox Creator Documentation

1 Like

Yeah, sorry, I didn’t mean that part for you, I meant it for Batimus.

The part I did mean for you was:

Should have been more clear, but I was trying to clarify and since your post was related I ccd you. Sorry for the confusion.

1 Like

Does this page tech help pave the way for getting the ranking of an OrderedDataStore? Here is a recent description of what I mean : Determine 1st, 2nd, 3rd, ect based on values sorting system? - #4 by BostonWhaIer

2 Likes

I actually do agree with this. This is certainly a small step in the right direction for being able to quickly attain the numerical position of a key within an ordereddatastore. Through a means of divide and conquer (binary sort/search-esque) type of solutions where you don’t have to worry about your cursors being as unpredictable.

Current solutions that I have seen or applied to this problem are accurate albeit up to a certain point (usually 10.000) if it’s a single numerical stat). So I am curious to play with this idea once again and see what it can do.

Seems like in the very near future the tooling will emerge to make solving this specific problem in an expanded fashion (applying to every element in an ordereddatatore) a lot more convenient. One such idea could be associating a particular cursor with elements in that group but I have yet to test this so I’m probably wrong.

1 Like

Would it be possible to add a method to retrieve how many pages or cursors there are in total?

I’m trying to delete all datas from some of my places and after a few hours I reached 4#global/11111112. I would like to know how much left there is. Like what’s my progress so far.

1 Like

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.