What is best way of saving/indexing data to/from large datastore - clan system attempt

[caviarbro]

Hello everybody, I have been trying to think what is the best way of saving data to very large datastore, I expect the datastore size to be near the datastore limit.

So I was currently thinking about two methods.

1. Method

• Keys will be the clans UUID and I can just index the clan easily by doing data[UUID]

local data = {}
data["UUID"] = {
  Name = "",
  Members = {},
}

2. Method

• The whole datastore is going to be an array - indexing of the clan is going to be done by pairs loop

local data = {}
table.insert(data,{
  clanId = "UUID"
  Name = "",
  Members = {},
})

Please let me know if there’s better approach to do so or which method should I use.

Thanks in advance,
caviarbro.

[colbert2677 (ImagineerColbert)]

How willing are you to wait for DataStore V2 to face stable release or completely set up your DataStores with experimental features with a way to handle cases of potential failure? Additionally, are you willing to not query DataStores frequently and only if you need them?

If you’re willing to run these assumptions then save clan data in separate keys and use ListKeysAsync if you want to get a list of all existing guilds in your experience. As for how you should display them, you’ll probably want to save on request budget and only display them if the user searches for them or you have, for example, a guild recommendation feature.

Any static information should be cached on the server. If you’re caching any changing data, you may want to let the player know when you’re stalling for more request budget or when the cache was last updated so they understand if they’re working with updated information or not.

Generally you should not collate data like this into a single key. Keys should be unique to a data set.

[caviarbro]

@colbert2677

How willing are you to wait for DataStore V2 to face stable release or completely set up your DataStores with experimental features with a way to handle cases of potential failure? Additionally, are you willing to not query DataStores frequently and only if you need them?

My project is far from release so I am willing to wait for a bit.

@colbert2677

If you’re willing to run these assumptions then save clan data in separate keys

I just realized it after I wrote this topic that saving under one key won’t be a good idea.

@colbert2677

As for how you should display them, you’ll probably want to save on request budget and only display them if the user searches for them or you have, for example, a guild recommendation feature.

My idea is to update the info about the clan on change so for example I want to change clan’s name and then I will call updateasync. The thing with this is the probability of abuse because more than one player are going to be able to change clan’s info but this can be probably prevented by storing clan’s UUID to table and set change info cooldown.

@colbert2677

Any static information should be cached on the server. If you’re caching any changing data, you may want to let the player know when you’re stalling for more request budget or when the cache was last updated so they understand if they’re working with updated information or not.

All active clans in the server will be cached in table and Info replicated via folders in replicated storage with attributes.

Additional question:

Am I going to be able prevent “duplicated names” of clans with the ListKeysAsync? Because this is probably my biggest concern right now.

Anyways thanks a bunch for your answer ;).

[colbert2677 (ImagineerColbert)]

@caviarbro

Am I going to be able prevent “duplicated names” of clans with the ListKeysAsync? Because this is probably my biggest concern right now.

My concern as well since I’m working on a guild system as well: you probably might have difficulty here unless you actually fetch clan data since ListKeysAsync only returns keys in your DataStore at the moment and additional querying options are still in development. In my case, I intend to use guild names as the identifier and it can’t be changed without creating a new guild. Not the best user experience but I’ve got to work with what I have access to and this seemed the most sound.

[caviarbro]

That’s unfortunate but I don’t really want to use guild names as a key but as I am seeing your reply there’s no probably no other efficient way to do it.

My original idea was to make second key in datastore with the name of the clan that will be holding the clan’s UUID and upon rename I can just Remove the key from datastore and create new one.

Do you think that this is a good idea?

[Abcreator]

@caviarbro

2. Method

• The whole datastore is going to be an array - indexing of the clan is going to be done by pairs loop

You will quickly hit the datastore key max length limit with this method.

[caviarbro]

@caviarbro

I just realized it after I wrote this topic that saving under one key won’t be a good idea.

I’ve already mentioned it here.

[Abcreator]

@caviarbro

that will be holding the clan’s UUID

You can’t get the value of a key in a datastore without making a datastore request for every guild.

[caviarbro]

Let me explain how it would work.

Check for if the guild name exists would work like that:

local guildName = "somethingx1"
local guildNameRequest = dataStore:GetAsync("@"..guildName) --// the @ is there to prevent conflict with randomly generated UUID
if (not guildNameRequest) then
    return true
else
    return false
end

[Abcreator]

In this case this should work as long as you don’t have two users trying to create the same guild name at the same time.

[caviarbro]

You are right but there’s nothing much I can do about it and I don’t think I would be able to do it even with ListKeysAsync.

[Abcreator]

Messaging Service could help with most of the cases.

[caviarbro]

I don’t really get what benefit would this have, because creating new key in datastore is probably faster than caching received data from MessagingService to table and check if anybody isn’t trying to make guild with same name.

But when I am thinking about it now it can be a good idea, but I don’t really know if MessagingService is fast enough for it to be even good.

[Abcreator]

@caviarbro

but I don’t really know if MessagingService is fast enough for it to be even good.

Messaging service is really fast (<1 second receival time), while keys in a datastore have a cool down of 6 seconds.

[caviarbro]

That’s true but with current limitations of Messaging Service this would be a big problem

MessagingService 'Messages received for entire game' limit is incorrect

@Seranok The wiki states the current limit for ‘Messages received for entire game’ is (100 + 50 * number of servers) per minute. Instead, it appears the game does not increase this budget for each additional server, so the budget instead remains at (100 + 0) per minute You can replicate the bug with the following code for one-player servers. With two players it runs fine, however as soon as you go above it begins throttling and throwing the error The rate of publish requests exceeds the allowe…

[Abcreator]

Actually, this was incorrect (they explained the limits incorrectly). You can receive 50 messages (across all servers) per minute safely.

[Abcreator]

@caviarbro In order to get away with bigger limits you should only subscribe the servers that are currently in the create a guild screen.

[caviarbro]

@Abcreator

while keys in a datastore have a cool down of 6 seconds.

I don’t really see the issue here since you are going to call the GetAsync and UpdateAsync here only once and the updates are going to be done only per certain time and on change.

@Abcreator

@caviarbro In order to get away with bigger limits you should only subscribe the servers that are currently in the create a guild screen.

Well that’s a good idea but how I would go about it?

Something like this?

local currentlyAddedNames = {}

local function getValidName(guildName)
  local guildNameRequest = dataStore:GetAsync("@"..guildName) --// the @ is there to prevent conflict with randomly generated UUID

 --// Messaging Service implementation
  local MessagingConnection = MessagingService:SubscribeAsync("guildNames",function(message)
        table.insert(currentlyAddedNames,HttpService:JSONDecode(message.Data))
  end

  if (not guildNameRequest and not table.find(currentlyAddedNames,guildName)) then
      MessagingService:PublishAsync("guildNames",guildName)
      MessagingConnection:Disconnect()
      return true
  else
      MessagingConnection:Disconnect()
      return false
  end
end

[Abcreator]

@caviarbro

  local MessagingConnection = MessagingService:SubscribeAsync("guildNames",function(message)
        table.insert(currentlyAddedNames,HttpService:JSONDecode(message.Data))
  end

You should connect this when the create Guild menu is opened.

@caviarbro

I don’t really see the issue here since you are going to call the GetAsync and UpdateAsync here only once

Multiple servers contribute to an experience limit (time between request never mentioned) meaning if two clients created a Guild with the same name, two Get requests would be issued possibly causing one to throttle.

[caviarbro]

@Abcreator

You should connect this when the create Guild menu is opened.

It sounds useless because I can just add small wait amount (something such as 1 second) to make sure I get all currently created guildNames.

@Abcreator

Multiple servers contribute to an experience limit (time between request never mentioned) meaning if two clients created a Guild with the same name, two Get requests would be issued possibly causing one to throttle.

Yeah now the MessagingService implementation sounds more useful.