Best way to store multiple pieces of data?

so what im trying to do is store a lot (i mean thousands to millions) of pieces of data without exhausting my datastore. Lets take this hypothetical situation here:

local Table = {
"1",
"2",
"3",
--etc etc
}

for i, v in pairs(Table) do
dataStore:UpdateAsync("ds", function(ov)
ov[i] = v
return ov
end)
end

my knowledge is pretty small on this, as I haven’t messed with updateasync so much so correct me if I’m wrong. But anyways im trying to store a lot of data all at once with out updating it multiple times (exhaust) i know you may be asking why not just save the whole table. For the thing I’m doing that’s not possible.

All answers appreciated.

2 Likes

Just an idea, but maybe combine all the data into a single string and put a space between each piece of data then when loading the data you could split it / convert it back.

You can try using external databases for storing data

1 Like

Datastores are primarily meant for storing single-user data, and occasionally for some unique instances, single-server data. For that reason if you want to store millions of pieces of data, you might consider using a database hosted on an online service such as google cloud for something like this. If you still want to use a database, though, here’s what I recommend:

Create a table with index:value pairs for whatever updates you need. So for instance:

local updates = {
  [1] = "New_1";
  [20] = "New_20";
  [5000] = "New_5000";
}

Then call update async once and set ov[index] = key for all updates in the table. So it’d look something like:

dataStore:UpdateAsync("ds", function(ov)
  for i,v in pairs(updates) do
    ov[i] = v
  end
  return ov
end)

This way you only have to use one datastore call, and it handles all the updates you need.

Side note: I know you’re not doing this, but make sure you don’t use ipairs for this, and instead use pairs. Because of the way we set this up, it treats the index numbers like a Map/Dictionary, not an Array.

1 Like

I’d personally recommend DataStore2 and the :Combine() function.

tableOfValues = {"a", "b", "c", "d"}
stringTable = table.concat(tableOfValues, "~") --use a unique delimiter
datastore:SetAsync(somekey, stringTable)

This will work for your use case, whenever you use GetAsync to get the data back you can simply use string.split() to split the data up again.

i had tried this once this exact thing but it didn’t seem to work as far as i saw cause i thought it would exhaust the data store. so i decided to re-asses this. i’ll check it out when i get on my pc.

haven’t learned ds2 yet. i’ve been meaning to check it out

wrong reply my bad… but i’ll check this method out too. efficiency is a key factor since my datastore will be storing thousands of large tables.

example? I don’t have my own server to store data

alright this works! thank you. It turns out the issue I had was that the return ov was inside the loop causing it not to work. Much appreciated. UpdateAsync() was something I never really got to fully learn so this is a great lesson on understanding it better. Thank you!

1 Like