DataStore:ListKeysAsync() does not paginate keys properly

Reproduction Steps

  1. Save multiple key value pair data into your DataStore where the prefix in the key is the same i.e. “flower…”
Saving data; example script.
local DS = game:GetService("DataStoreService")
local KDS = DS:GetDataStore("FlowerLexemes")

local fakeDataImport = {
	["flowerpot"] = 1,
	["flowerface"] = 2,
	["flowermate"] = 3,
	["flowerfind"] = 4,
	["flowerfist"] = 5,
	["flowerman"] = 6,
	["flowertask"] = 7,
	["flowerthrow"] = 8,
	["flowercrude"] = 9,
	["flowerbase"] = 10,
}

for name, weight in pairs(fakeDataImport) do
	local success, errorMessage = pcall(function()
		print(
			string.format(
				'Importing key: %s with weight: %d',
				name,
				weight
			)
		)
		KDS:SetAsync(name, weight)
	end)
	if success then
		print(
			string.format(
				'Imported key: %s successfully!',
				name
			)
		)
	else
		warn(errorMessage)
	end
end
  1. Use ListKeysAsync with a prefix and a limit to retrieve the pages object.
    local myPageObject = KDS:ListKeysAsync("flower", 5)

  2. Iterate over the page object.

Retrieving data; example script.
local myPageObject = KDS:ListKeysAsync("flower", 5)


-- Reformat pages as tables
local function pagesToTable(pages)
	local items = {}
	while true do
		table.insert(items, pages:GetCurrentPage())
		if pages.IsFinished then
			break
		end
		print("---- AdvanceToNextPageAsync ----")
		pages:AdvanceToNextPageAsync()
	end
	return items
end

local function iterPageItems(pages)
	local contents = pagesToTable(pages)
	-- Track the current page number starting at 1
	local pageNum = 1
	-- Get last page number so we don't iterate over it
	local lastPageNum = #contents

	-- for will resume this coroutine until there's nothing to go through
	return coroutine.wrap(function ()
		-- Loop until page number is greater than last page number
		while pageNum <= lastPageNum do
			-- Go through all the entries of the current page
			for _, item in ipairs(contents[pageNum]) do
				-- Pause loop to let developer handle entry and page number
				coroutine.yield(item, pageNum)
			end
			pageNum += 1
		end
	end)
end


for item, pageNo in iterPageItems(myPageObject) do
	print(
		string.format(
			'Page: %d has key: %s',
			pageNo,
			item.KeyName
		)
	)
end

Full script right here:

local DS = game:GetService("DataStoreService")
local KDS = DS:GetDataStore("FlowerLexemes")

local fakeDataImport = {
	["flowerpot"] = 1,
	["flowerface"] = 2,
	["flowermate"] = 3,
	["flowerfind"] = 4,
	["flowerfist"] = 5,
	["flowerman"] = 6,
	["flowertask"] = 7,
	["flowerthrow"] = 8,
	["flowercrude"] = 9,
	["flowerbase"] = 10,
}

for name, weight in pairs(fakeDataImport) do
	local success, errorMessage = pcall(function()
		print(
			string.format(
				'Importing key: %s with weight: %d',
				name,
				weight
			)
		)
		KDS:SetAsync(name, weight)
	end)
	if success then
		print(
			string.format(
				'Imported key: %s successfully!',
				name
			)
		)
	else
		warn(errorMessage)
	end
end


local myPageObject = KDS:ListKeysAsync("flower", 5)


-- Reformat pages as tables
local function pagesToTable(pages)
	local items = {}
	while true do
		table.insert(items, pages:GetCurrentPage())
		if pages.IsFinished then
			break
		end
		print("---- AdvanceToNextPageAsync ----")
		pages:AdvanceToNextPageAsync()
	end
	return items
end

local function iterPageItems(pages)
	local contents = pagesToTable(pages)
	-- Track the current page number starting at 1
	local pageNum = 1
	-- Get last page number so we don't iterate over it
	local lastPageNum = #contents

	-- for will resume this coroutine until there's nothing to go through
	return coroutine.wrap(function ()
		-- Loop until page number is greater than last page number
		while pageNum <= lastPageNum do
			-- Go through all the entries of the current page
			for _, item in ipairs(contents[pageNum]) do
				-- Pause loop to let developer handle entry and page number
				coroutine.yield(item, pageNum)
			end
			pageNum += 1
		end
	end)
end


for item, pageNo in iterPageItems(myPageObject) do
	print(
		string.format(
			'Page: %d has key: %s',
			pageNo,
			item.KeyName
		)
	)
end
  1. Observe that the output incorrectly paginates keys:
    image

Expected Behavior

ListKeysAsync should be able to return all pages with the keys matching the prefix within the limit. If I have 10 “flower…” keys with the “flower” prefix set and a limit of 5 as my query parameter, upon iterating it should find only 2 pages with 5 keys.

Actual Behavior
ListKeysAsync incorrectly returns the keys in separate pages instead of following the query parameter “pageSize” rule. Occasionally it puts only 2 keys into a single page but most of the time 1 key per page is returned. Full thread where I first noticed the bug: ListKeysAsync does not work? - Help and Feedback / Scripting Support - DevForum | Roblox

Workaround
None

Issue Area: Engine
Issue Type: Other
Impact: High
Frequency: Constantly
Date First Experienced: 2023-01-25 21:00:00 (+02:00)

5 Likes

We’ve filed a ticket to our internal database, and will follow up when we have news!

Thanks for flagging!

1 Like

I believe an engineer previously said that this unfortunately has to be intended behaviour as sometimes keys are stored on a different partition and cannot be retrieved via a single call.

1 Like

If that’s the case, that needs to be properly documented and not left in an obscure response somewhere on the forum. Even if keys do partition, I’m not particularly sure it makes sense to need to split 8-10 keys among several partitions, but I’m not going to do any backseat engineering or pretend I know how they’re interfacing with the database platform.

It would be nice if you could find that reply and link it.

5 Likes

@DrBrainHurt @Abcreator @colbert2677
Hi, I work on the Creator Services team on Data Stores. @Abcreator is correct that the way ListKeysAsync works is that it fetches up to the number of keys requests, but not necessarily that max number, ie up to 5 keys in the flowers example, but doesn’t fetch across partitions in the Data Stores. You will need a loop to accomplish cross-partition listing of keys. This is a bigger problem with small data sets but for very large data sets, each partition will have many objects the cross partition issue will only come up occasionally during a listing of objects when a partition gets exhausted.

We are working on correcting the documentation.

We have added cross-partition listing of keys as a backlog item to work on in the future as well, so will fix this inconvenience down the road.

2 Likes

Appreciate the follow up. Like I mentioned in my previous reply, is there any way you could reach out to IX to get this partition behaviour documented for listing APIs? This behaviour might confuse more developers down the road who’ve yet to run into this problem but eventually will when needing to perform listing.

Now that I’m aware of this behaviour, I’m more aware that I need to create my own interface for pagination and validation of listed contents pulled from a DataStore so I can better tailor my systems around that interface rather than using the literal contents of a DataStore fetch. This is for cases where a page and its content size are important to me.

The main piece from me is having this documented. Happy to hear cross-partition is on the backlog though!

2 Likes
  • Do the right thing. Do what’s best for Creators.
  • Done!

GDay everyone, I have the right to believe it’s fixed now. Thanks for the patience, and sorry it took so long!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.