Help With Data Storage And Replication Architecture

Hello everyone,
For the last few projects of mine I have been relying on IntValues and StringValue instances to keep track of the player’s stats. However, this has limitations because storing an Inventory as a StringValue means it is extremely hard to give attributes to the items.

So, I changed my architecture to a ModuleScript in the Server, and with every update to the player’s stats sends a remote event to a localscript in StarterCharacterScripts with the updated stats and a bindableEvent is used to communicate these stats with different scripts.

But I am very paranoid that what I am doing is inefficient or not how people would usually go about storing data. And also sending a RemoteEvent every stat update for each player could activate some sort of rate limit.

Is what I am doing correct?

1 Like

You could batch updates.

while task.wait(1) do
    for _, players in Players:GetPlayers() do
        local vt = get_data_version_time(player)
        if vt == version_times[player] then
            continue
        end

        version_times[player] = vt

        --- dealing with confidential data
        RemoteEvent:FireClient(player, get_player_data(player))

        --- dealing with public data or something
        player:SetAttribute("Data", to_json(get_player_data(player)))

        --task.wait()
    end
end
1 Like

Yes but then the currency UI / inventory UI updates would be delayed.
How have you done this in the past?

I didn’t use any RemoteEvent / RemoteFunctions, I just used attributes like my previous answer:

You would wrap it on your OnDataUpdated.

But then couldn’t I just use stringvalue for inventory but just JSON.encode/JSON.decode it instead of doing like string.split?

Can you show me like how you have done it in your previous games?

It could drop the request sending errors, but only if you continuosly try to fire remote events.
So if you update the data every 5 seconds or every time a player data is changed you should be fine, as long as you don’t use one remote event for many stats every second.

1 Like

I don’t have any previous games, but I’m working on one.
I think I didn’t use one singular attribute for all.
I would store them at the ReplicatedStorage like this:

1 Like

By “Architecture” I still don’t know if you mean data replication or a method to send data efficently.

Like what method would you use to store player data and transfer to client, just using Instances or using modulescripts etc

Attributes are “heavy” and alot of them can cause alot of lag.
Finding another way to use them I think it’s alot better, to save or send data.

I think you could just use JSON.encode and decode to put all attributes in 1 stringvalue for each item

Module scripts seem the better option.
I recently created my own replication system, using buffers which help you to reduce the paycheck of remotes too.
If you are intrested I can show you my module (it’s not as good as other community resources).

2 Likes

What’s a buffer?
If you could show me some resources that would be nice because I’ve looked across the whole internet and can’t find any :joy:

To replicate data I would consider using “Replica” or “ByteNet”, two very fast modules to replicate your data.
And for data saving I would recommend either “ProfileStore” or either “Suphi’s Data Store Module”, again two of the best data saving modules in the forum.
All the resources are free.

A DataType created for compressing strings and numbers, you can compress other objects too but you will have to learn serialization.

2 Likes

Is there a quicker way then to have a seperate remote event for each stat? I have like 15 stats so I might want it to be a bit more organised

I mean you can just send a table with information from a single table, just make sure it’s not gigantic.

1 Like

What do you mean? (charlimit–)

You can create a single remote and send data throught it.
Example:

local Remote = path.to.remote
local Players = game:GetService("Players")
local stats = {} --your stats here

local function send(player: Player, stats: {any})
   -- code here

   remote:FireServer(player, "StatUpdater", stats) --or FireClient
end

Players.PlayerAdded:Connect(function(player: Player)
    task.spawn(send, player, stats)
end


2 Likes

In your case, optimization of stress is useless, because players will have HUD and inventory opened very often, this mean that gain from possible: Update only when value is visible gives you nothing

The most important thing is what you send, you should always have One Remote per task, otherwise it will be very inefficient, if you have already remote called ReplicateData, you should think about where you need to send data

For instance, if you have HUD data, like coins or level or smth, this is possible solution:

  1. Create HUDReplicator
  2. Assign numerical IDs to every HUD component (strings aren’t that efficient, you can pack numbers into buffers to gain advantage)
  3. Send ID and Value to the client, by packing them into 3 byte buffer you’ll reduce their size from 16 bytes, which is a lot
  4. Decompress value and use maps to find correct elements & perform functions
local function onClientEvent(Data: buffer)
    local ElementID = buffer.readu8(Data, 0)
    local Value = buffer.readu16(Data, 1)
    local Gui = ElementIDToGuiObject[ElementID]
    if not Gui then
        return
    end

    Gui.Text = Value
end

ReplicateHUD.OnClientEvent:Connect(onClientEvent)
2 Likes

This is very bad idea, as network load will increase making game unplayable