ListKeysAsync listing keys of removed data

Bug Reports Engine Bugs
#1

Reproduction Steps
Links: Not relevant, if there is really a need to it please send me a PM as the place this was found in is not meant to be displayed publicly yet.
System Information: Not relevant, purely an API call that returns odd info
Reproduction: When calling ListKeysAsync (only tested with prefix and scope usage on the datastore). it lists all keys with said prefix however when one of these datas are removed using RemoveAsync it rids of the data but the key will remain present in ListKeysAsync despite it not fetching any data other than versions

Expected Behavior
I expect that ListKeysAsync returns the keys of any present data where the keys of deleted storages are no longer given as they no longer contain any info or somewhat be listed seperately from a list of entirely active data as it’s confusing to have all the currently present keys together with keys that could potentially access previous versions.

Alternatively having a seperate API call or Enum as the parameters for these type of keys works too and might allow for a more structural usage of the key listing api feature.

Actual Behavior
The key of the removed datastore is still present in the list which is extremely confusing and can hardly be identified on if it’s an removed key or not as it just gets marked as a “new data” through GetAsync, etc.

https://gyazo.com/9f7f0a959fc71e6df4dfc4eb63286c8a

Workaround
the only work around i could come up with is identifying the keys with a GUID (what’s shown as .1 and .2 in the example code) and marking said GUID as “removed” under a general storage of said user. Though this could fill up excessively while i really shouldn’t need to store said key.

Issue Area: Engine
Issue Type: Other
Impact: Moderate
Frequency: Constantly
Date First Experienced: 2022-04-01 16:04:00 (+02:00)
Date Last Experienced: 2022-04-01 00:04:00 (+02:00)

5 Likes
#2

Thanks for reporting. We will look into it , we will likely expose this as a property of DataStoreKeyInfo which says a key is deleted when you do get async.

Some additional background , we are returning the deleted keys so that you could do list versions and potentially restore the key later. But we should provide a way to differentiate deleted keys.

4 Likes
#3

I feel like this might cause rate limiting issues, considering the limit is 60 + playerCount * 10 and assuming these keys are wiped after 30 days. Is there a way where devs could manually wipe all the data including previous versions and said key present in the ListKeysAsync as im unsure if manually removing all versions of said key would do that or not. if not is there potential of said feature to be added?

1 Like
#4

The keys will be automatically removed after 30 days , but the key deletion is not guaranteed to happen after 30 days. We will look into providing an API to immediately remove the key permanently. In general where possible, we suggest overwriting objects instead of creating and deleting keys.

2 Likes
#5

Any news on this?
I was going to open a Bug Report, but I came across this one, which is 5 months old…
I have this test:

local DataStoreService = game:GetService("DataStoreService")
JogoStore 	= DataStoreService:GetDataStore("Jogo")

for i = 1, 3 do -- create 3 keys
	JogoStore:SetAsync(i, i*10)
end

function ListAllKeys(DS, Operation)
	print("\n***ListAllKeys(DS, Operation): ", DS, Operation)
	local DSoptions = Instance.new("DataStoreOptions")
	DSoptions.AllScopes = true
	local DataStore = DataStoreService:GetDataStore(DS, "", DSoptions)

	local Success, Pages = pcall(function()
		return DataStore:ListKeysAsync('global')
	end)
	if Success then
		while true do
			local Itens = Pages:GetCurrentPage()
			for _, Key in ipairs(Itens) do
				if Operation == 'Remove' then
					local Success, Erro = pcall(function()
						local Result = DataStore:RemoveAsync(Key.KeyName) -- remove conteĂşdo da chave
						print('Remove', Key.KeyName)
					end)

					if not Success then
						warn(Erro)
					end
				else
					local Value, KeyInfo = DataStore:GetAsync(Key.KeyName)
					print('Key.KeyName, Value, KeyInfo', Key.KeyName, Value, KeyInfo)
				end
			end
			if Pages.IsFinished then
				break
			end

			repeat
				local Success, Errormessage = pcall(function()
					Pages:AdvanceToNextPageAsync()
				end)
				if not Success then
					warn('Error')
					warn(Errormessage)
					wait(1)
				end
			until Success

		end
	end
end


ListAllKeys('Jogo')
ListAllKeys('Jogo', 'Remove')
ListAllKeys('Jogo')
  1. First, it will create 3 keys inside DS “Jogo”
  2. The keys are listed
  3. The keys are deleted
  4. The keys are listed again (with no values)
***ListAllKeys(DS, Operation):  Jogo nil
  Key.KeyName, Value, KeyInfo global/3 30 Instance
  Key.KeyName, Value, KeyInfo global/2 20 Instance
  Key.KeyName, Value, KeyInfo global/1 10 Instance
  
***ListAllKeys(DS, Operation):  Jogo Remove
  Remove global/3
  Remove global/2
  Remove global/1
  
***ListAllKeys(DS, Operation):  Jogo nil
  Key.KeyName, Value, KeyInfo global/3 nil nil
  Key.KeyName, Value, KeyInfo global/2 nil nil
  Key.KeyName, Value, KeyInfo global/1 nil nil

The keys should not be listed after being removed. :point_left:t2:

New "excludeDeleted" parameter for "ListKeysAsync" doesn't work
Release Notes for 565
#6

ListKeysAsync not listing keys of removed data would help me immensely complying with GDPR requests. I have a few PlaceIds that are no longer active and would like to delete all data from them so I would no longer have to check them.

Currently, I use ListKeysAsync to loop through all available keys to delete them. But because this takes a very long time, I might not be able to do this in 1 session. Problem with ListKeysAsync is that I can’t pause or continue in another session because I would have to go through all the keys again, even if I have removed them.

1 Like
#7

Hi, We are still looking into providing a way to exclude deleted keys. But we have added support to resume list keys operation that could be helpful for your use case. Please see Datastore Listing API's Cursor for more details.

1 Like
#8

The problem continues until today and, in my case, every day it gets worse, because in some cases I need to delete all the records from my DataStore, but as every day there are more records, every day it takes longer to delete them.
In short, each removal of a key has taken around 1 second. This means that to remove 600 keys, I have to wait around 10 minutes.
None of this would happen if ListKeysAsync wasn’t incorrectly listing keys that have already been deleted.

#9

You can fire off multiple concurrent requests to datastores in separate Lua threads, no? Or is there a budgeting issue here?

1 Like
#10

In the documentation, ListKeysAsync has an excludeDeleted parameter. I have tested it and it seems to not do anything.

1 Like
#11

Please try it now :partying_face:

It wasn’t enabled until today so the docs page was definitely a spoiler. We will add some formal documentation to that link shortly.

With excludeDeleted enabled ListKeysAsync will not return any deleted keys. However this will not parse through all your keys. For now it will check 512 keys, if all 512 keys are marked as deleted it will return an empty list with a nextPageToken for the user to keep parsing.

This parameter defaults to false so it must be explicitly set.

Let us know if this is working and/or if it fulfills your use case. Thank you!

2 Likes
#12

I just opened a bug report about this:

1 Like
#13

FYI we are temporarily turning this parameter off to investigate a bug report. Sorry about the inconvenience.

4 Likes
#14

excludeDeleted has been re-enabled. Thank you for your patience :slight_smile:

4 Likes
#15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.