Datastore transfer issue

[TheGreatFurymax (TheRatman)]

Hello there,

I am trying to move from DataStoreService toward an external database.

Now, the first issue is that there’s one datastore for each player:
game.DataStoreService:GetDataStore('DATA/1295727161') → This is the datastore with my stats

So there are probably more than 100k datastores with only player stats .

For that, I got this pretty simple script that gather all those datastores:

local DataStoreService = game:GetService("DataStoreService")

local listSuccess, pages = pcall(function()
	return DataStoreService:ListDataStoresAsync()
end)


if listSuccess then
	while task.wait(1) do
		local items = pages:GetCurrentPage()
		for _, v in ipairs(items) do
			
			local naem = v.DataStoreName

			if string.sub(naem, 1, 5) == "DATA/" then
				print("DATASTORE FOUND: " .. naem)
			end
		end
		if pages.IsFinished then
			break
		end
		pages:AdvanceToNextPageAsync()
	end
end

Now the true issue is that each datastore has a key that starts from 1, and each time a new version of the datastore is being made, the key get increased by one. So if someone has got their datastore saved 200 times, their key is 200.

I only need the latest key.

So I’ve tried this script which is a combination with another one found on the forum, I think.

local dataService = game:GetService("DataStoreService")


local function help(DataStore)

	local listSuccess, pages = pcall(function()
		return DataStore:ListKeysAsync()
	end)
	if listSuccess then
		while true do
			local items = pages:GetCurrentPage()
			for _, v in ipairs(items) do
				local value = DataStore:GetAsync(v.KeyName)
				print(value)
			end
			if pages.IsFinished then
				break
			end
			pages:AdvanceToNextPageAsync()
		end
	end

end

local listSuccess, pages = pcall(function()
	return dataService:ListDataStoresAsync()
end)


if listSuccess then
	while task.wait(1) do
	local items = pages:GetCurrentPage()
	for _, v in ipairs(items) do
		local naem = v.DataStoreName
		if string.sub(naem, 1, 5) == "DATA/" then
			help(dataService:GetDataStore(naem))
		end

	end
	if pages.IsFinished then
		break
	end
	pages:AdvanceToNextPageAsync()
	end
end

This will most likely saturate the datastore rate limit very quickly.

How can I proceed this operation relatively quickly?

[player356377 (Haroon)]

If you only need the last key you should avoid calling :GetAsnyc() on each key.
To only get the last keys values you can modify your function like this:

local function fetchLastKeyInDataStore(DataStore)

	local listSuccess, pages = pcall(function()
		return DataStore:ListKeysAsync()
	end)

	if not listSuccess then 
        warn("Failed to fetch keys for DataStore!")
        return
	end

    local lastKeyIndex = 0 --We count the amount of keys in the pages. The last keys name is the amount of keys in the DataStore
    repeat --Keep advancing the page pointer adding the amount of keys per page until we arrive at the last one

        currentMaxIndex += #pages:GetCurrentPage()

        if not pages.IsFinished then
            pages:AdvanceToNextPageAsync()
        end

    until Pages.IsFinished  

    local success, data = pcall(function()
        return DataStore:GetAsync(tostring(lastKeyIndex))
    end)

    if not success then
        warn("Failed to fetch data!")
        return
    end

    print(data) 

end

Keep in mind you will exceptionally quickly hit the rate limits in place for the DataStores.
You will have to implement logic to automatically respect the limits set in place by Roblox in addition to resending requests if a DataStore query fails.

If your DataStores actually store 100k entries you can expect to wait multiple days for the data transfer to finish. And that is without considering potentially failed requests that you will have to resend. You also have to close your game during the transfer as data written to the DataStores during the transfer might not carry over resulting in data loss.