Replica - an easily controllable Server/Client data replication library

EDIT: I have not been updating this library much as of late, and no longer use it in my own projects. I plan on making a new library in the future to supercede Replica, once typed Luau comes out.


Get it here: https://www.roblox.com/library/2977749802/Replica
Documentation: https://github.com/headjoe3/Replica

Data replication is probably one of the trickier issues with roblox game development. Once you start creating table-based data, it becomes more difficult to replicate certain portions at a time, or to listen to changes in only parts of the data. In previous games, I’ve resigned to using ValueBase and Folder instances to represent player state, as it’s usually the easiest way of getting data replicated. But even that has problems, namely that there is no good way to coordinate local changes, or bundle data changes together. Another important downside with ValueBase representation is that all data must be public. For most cases, this isn’t a problem, but it’s definitely a limiting factor.

So I recently created a new package called Replica, which contains a set of helper classes called “Replicants” with some basic Getter/Setter functions for changing and replicating data. It also has functions Collate and Local which control how data is replicated, or whether it will be replicated at all. Finally, it includes helper signal classes for listening to changes in certain portions of data.


Replicants

There are currently five Replicant classes that can be used (PM me suggestions if you want to see more):

  • Replica.Array - A table of items with sequential integer keys
  • Replica.Map - A table of items with string keys
  • Replica.FactoredOr - A table with string-key factors with a “true” or “false” state; iff at least one factor is true, the FactoredOr object will resolve to “true”
  • Replica.FactoredNor - Similar to FactoredOr; iff at least one factor is true, the FactoredOr object will resolve to “false”
  • Replica.FactoredSum - A table with string-key factors and numeric values. The FactoredSum object will resolve to the sum of all factors.

Replicant objects are replicated tables whose values can be accessed, changed, and replicated using the :Get() and :Set() functions

Example:

local Replica = require(path.to.Replica)

game:GetService("Players").PlayerAdded:Connect(function(player)
    -- Set up initial playerData state as a Replicant object
    local playerData = Replica.Map.new({
        Coins = 0,
        -- Nested replicant objects
        Inventory = Replica.Array.new({
            "Foo",
            Replica.Array.new({}),
            "Bar",
        })
    })
    
    -- Replicant objects can be registered and de-registered to any key.
    -- Once they are registered, they will be replicated to all subscribed clients
    Replica.Register(player, playerData)
end)

This will create a new Replicant map called “playerData” with nested Replicant objects.
To change a player’s coins and replicate the change, simple use:

playerData:Set("Coins", playerData:Get("Coins") + 10)

If you have a lot of changes you would like to make at once, you can use the Collate method to freeze replication until a series of changes have been made at once

playerData:Collate(function()
    playerData:Set("Coins", playerData:Get("Coins") - 10)
    playerData:Set("Power", playerData:Get("Power") + 10)
end)

This will send a single replication update to all subscribed clients, so that the client will see Coins and Power change at the same time.

Client-side code can also set predictive changes to a Replicant’s state, which will be overwritten on the next server update. This is done using the :Predict() method

local playerData = Replica.WaitForRegistered(game.Players.LocalPlayer)
playerData.OnUpdate:Connect(function(isLocal)
	print("You have", playerData:Get("Coins"), "coins")
end)

path.to.humanoid.Jumping:Connect(function(isJumping)
	if isJumping then
		playerData:Predict("Coins", 1000)
	end
end)

The above code will temporarily set the player’s coins to “1000” when the player jumps (client-side only), which will reset back to the actual value if a server update overwrites it

Finally, there are a number of helper events that you can listen to whenever an update occurs in a replicant’s data. OnUpdate fires just after the data has changed, and WillUpdate fires just before. There is also GetValueWillUpdateSignal and GetValueOnUpdateSignal which acts similarly to the GetPropertyChangedSignal method if Instances:

playerData.WillUpdate:Connect(function(isLocal)
	print("You had", playerData:Get("Coins"), "coins")
end)
playerData.OnUpdate:Connect(function(isLocal)
	print("You have", playerData:Get("Coins"), "coins")
end)

Alternatively (will only fire when Coins changes, rather than the whole player data):

playerData:GetValueWillUpdateSignal("Coins"):Connect(function(isLocal)
	print("You had", playerData:Get("Coins"), "coins")
end)
playerData.GetValueOnUpdateSignal:Connect("Coins"):Connect(function(isLocal)
	print("You have", playerData:Get("Coins"), "coins")
end)

Here’s a demo putting it all together:

Server

local Replica = require(game.ReplicatedStorage.Replica)

game:GetService("Players").PlayerAdded:Connect(function(player)
    -- Set up initial playerData state
    local playerData = Replica.Map.new({
        Coins = 0,
        -- Replicant objects
        Inventory = Replica.Array.new({
            "Foo",
            Replica.Array.new({}),
            "Bar",
        })
    })
    
    -- Replicant objects can be registered and de-registered to any key.
    -- Once they are registered, they will be replicated to all subscribed clients
    Replica.Register(player, playerData)
end)

game:GetService("Players").PlayerRemoving:Connect(function(player)
    Replica.Unregister(player)
end)

-- The server can make changes to the replicated data using setter functions, and it
-- will automatically be replicated to subscribed clients
while wait(1) do
    for _, player in pairs(game.Players:GetPlayers()) do
        local playerData = Replica.GetRegistered(player)
        if playerData ~= nil then
            -- Collated data changes will be sent to the client in one single update
            playerData:Collate(function()
                -- Getters and setters automatically replicate changes to Replicant data
                playerData:Set("Coins", playerData:Get("Coins") + 1)
                
                -- The Array object allows items to be inserted and removed at any
                -- position. Though I would recommend using a map instead,
                -- all array changes will be buffered and replicated to the clients,
                -- maintaining Replicant objects even when their position changes.
                -- within the playerData tree.
                local quxIndex = playerData:Get("Inventory"):IndexOf("Qux")
                if quxIndex == nil then
                    playerData:Get("Inventory"):Insert(1, "Qux")
                else
                    playerData:Get("Inventory"):Remove(quxIndex)
                end
            end)
        end
    end
end

Client

local Replica = require(game.ReplicatedStorage.Replica)

-- You can guarantee the existance of a registered Replicant using WaitForRegistered
local playerData = Replica.WaitForRegistered(game.Players.LocalPlayer)

print("Player data loaded")

-- The client will receive data replication updates from the server
playerData.OnUpdate:Connect(function(isLocal)
    print("You have", playerData:Get("Coins"), "coins")
    print("You have", playerData:Get("Inventory"):Size(), "items in your inventory")
end)

-- Wait for character to load
local char = game.Players.LocalPlayer.Character or game.Players.LocalPlayer.CharacterAdded:Wait()
local hum = char:FindFirstChildOfClass("Humanoid"); while not hum do char.ChildAdded:Wait() hum = char:FindFirstChildOfClass("Humanoid") end

-- The client can also make predictive state updates which will be overridden by
-- the server on the next update if the prediction does not match the replicated value
hum.Jumping:Connect(function(isJumping)
    if isJumping then
        -- The Local function allows changes to be made without any replication
        -- implication. This is useful for things like UI, where you want to display
        -- changes predictively, even though you have not confirmed them with the
        -- server.
        playerData:Predict("Coins", 1000)
    end
end)

Finally, Replica lets you choose which players are able to see what portions of your data. Most Replicant objects have a “Config” argument as their second parameter, which subscribes all clients to changes to the object by default.

This can be changed using a partial configuration (see the documentation for more information https://github.com/headjoe3/Replica/blob/master/docs/Config.md):

local playerData = Replica.Map.new({
    Public = Replica.Map.new({}, {
        SubscribeAll = true,
    }),
    Private = Replica.Map.new({}, {
        SubscribeAll = false,
        Whitelist = { player },
    }),
})

playerData:Get("Public"):Set("Coins", 10) -- Replicated to all players
playerData:Get("Private"):Set("Secret", "I watch MLP") -- Replicated to one player only

Replica.Register(player, playerData)

In the above code, any updates within the “Public” tree will be visible to all players; however, updates within the “Private” tree will only be visible to the player that the data belongs to.

For more information on how to use the library, or the different Replicant objects such as Array, Map or FactoredSum, see the documentation on github: https://github.com/headjoe3/Replica

Here’s another demo using the FactoredOr, FactoredNor, and FactoredSum objects

Server

local Replica = require(game.ReplicatedStorage.Replica)

game:GetService("Players").PlayerAdded:Connect(function(player)
    local playerData = Replica.Map.new({
        Or = Replica.FactoredOr.new(),
        Nor = Replica.FactoredNor.new(),
        -- This configuration makes the Sum only visible to the player that the
        -- playerData belongs to
        Sum = Replica.FactoredSum.new({}, { SubscribeAll = false, Whitelist = {player} }),
    })
    
    -- These should fire on the server output only when the state has changed value
    playerData:Get("Or").StateChanged:Connect(function(...)
        print("Or state changed", ...)
    end)
    playerData:Get("Nor").StateChanged:Connect(function(...)
        print("Nor state changed", ...)
    end)
    playerData:Get("Sum").StateChanged:Connect(function(...)
        print("Sum state changed", ...)
    end)
    
    Replica.Register(player, playerData)
end)

game:GetService("Players").PlayerRemoving:Connect(function(player)
    Replica.Unregister(player)
end)

local keys = {"a", "b", "c"}

while wait(2) do
    for _,player in pairs(game.Players:GetPlayers()) do
        local playerData = Replica.GetRegistered(player)
        if playerData then
            playerData:Collate(function()
                local key = keys[math.random(1, #keys)]
                -- The Or and Nor states should always resolve to the opposite
                -- of each other, since they always toggle the same keys.
                playerData:Get("Or"):Toggle(key)
                playerData:Get("Nor"):Toggle(key)
                -- The Sum state resolve to a random value between 0 and 3
                playerData:Get("Sum"):Set(key, math.random())
            end)
        end
    end
end

Client

local Replica = require(game.ReplicatedStorage.Replica)

-- Note that we are reading Player1's data, so if Player2 joins, they will
-- see the same data with factors in Sum omitted
local playerData = Replica.WaitForRegistered(game.Players.Player1)

print("Player data loaded")

playerData.OnUpdate:Connect(function()
    print("UPDATE")
    for key, factorMap in playerData:Pairs() do
        print("   ", key .. ":", factorMap:ResolveState())
    end
end)

If you have any questions or suggestions for improvements, let me know!


Roblox-TS typings (for those of you that use TypeScript): https://github.com/headjoe3/rbxts-replica/blob/master/README.md

42 Likes

Nice! I will definitely be using this in my games. This will make everything a lot easier.

2 Likes

Cool! Expect updates in the future, as it is undertested, and I am still in the process of porting it to my game’s code

3 Likes

I have now finished implementing Replica into my own game, and have made some quality-of-life additions (Replicant:MergeSerialized() and Replicant:Inspect(), documented here), as well as some important bugfixes.

If you are using the initial release in your game, make sure to update it with the most recent model (or through RoStrap)

1 Like

This is a very cool module, thanks for making in public!

I’m currently halfway in to my game that uses value objects for similar reasons, but I look forward to using this in the future!

1 Like

Just published some more updates! I’m getting closer and closer to being able to phase out ValueBase instances to represent replicated state in my own game.

Some key changes:

  • Replica.Register now maps Replicants to actual instances (before they would just convert the instance input to a string by the instance’s name) using CollectionService and some other tricks. This means that you can now give any Instance Replicant data, regardless of what it is named (or whether it has fully replicated to other clients yet).
  • Local is deprecated and no longer required for setting values on the client. This means that Set now has the overloaded behavior of locally setting and replicating, depending on whether the calling network side (Server or Client) has permissions to replicate this value. I decided this would be better and more intuitive overall, since Local was mostly pointless based on context.
  • Added a swifty new function Replicant:Predict which allows you to predict the next states of data before receiving server confirmation. Predictions are buffered, and, if the predictions match up (even when multiple predictions are made in a row), then the server updates will NOT override the client’s predictions, since these state changes would be redundant. If, however, the predictions are inaccurate, then the client’s state will be rubber-banded to the actual value.
  • Fixed a bug with visibility (didn’t think it was a bug at first, but it was) where the initial state would be replicated to clients that recently join, even if updates are not supposed to be visible to that player.
  • Added a second argument to Replicant:Serialize(key, forClient) which will serialize a Replicant with visibility for another player in mind

A use case for :Predict() would be a weapon system where you want it to appear like your weapon immediately damages another player, but health changes occur so rapidly that you cannot rely on overriding server updates alone (since you might have damaged a player multiple times before you ever receive confirmation from the server about the first hit).

2 Likes

This is a really neat idea! I tried integrating it into one of my projects and I couldn’t get the most basic example working until I spent a bit of time debugging Replica initialization. Here’s the simple example I’m testing.

--[[ Server ]] --
local testData = Replica.Map.new({
    Coins = 0,
})

Replica.Register("foo", testData)

testData:Set("Coins", 111)

--[[ Client ]]--
local testData = Replica.WaitForRegistered("foo")

testData.OnUpdate:Connect(function()
    local coins = testData:Get("Coins")
    print(string.format("Player has %d coins!", coins))
end)

My client was not receiving the update for this Replicant mutation on the server until I adjusted this line in Replica.lua (line 347 on latest version).


Changing line 347 to check not sentInitialReplication[client] (underlined in yellow), as this stood out to me as what was preventing the client from receiving the replication. After this the example works as expected. Is this a bug as suspected? If not, what else might be going wrong in such a simple use case?

3 Likes

That was definitely a bug! Looks like it was working fine for me since all of the Replicants in my games were registered before any player joined (so that particular event wouldn’t fire), so I’m glad you caught it. I updated the model with your change.

1 Like

After looking into Replica more in depth today, I noticed this issue is actually due to a more subtle bug, and the code pointed out previously was actually fine.

The issue is that it’s possible for the Replica module to be required on the server after players have already joined the game, in which case these players will not get initial replication, as this is only happening on the PlayerAdded event callback - which is only registered once this module is run.

To fix this I added an iteration over all existing players to sent initial replication of registered Replicants which resolved the issue. (Relevant snippet posted below).

local function sendInitReplicationToClient(client)
    for key, replicant in pairs(registry) do
        if replicant:VisibleToClient(client) then
            baseReplicantEvent:FireClient(client, key, replicant:Serialize(key, client), replicant.config)
        end
    end
    sentInitialReplication[client] = true
end

game.Players.PlayerAdded:Connect(function(client)
    sendInitReplicationToClient(client)
end)

-- For any players already on the server at the time this module runs (as this can be required after players have joined)
-- invoke initial replication of replicants
for _, client in pairs(game.Players:GetPlayers()) do
    sendInitReplicationToClient(client)
end

I’ve submitted a PR to the main repo https://github.com/headjoe3/Replica/pull/2

2 Likes

Currently it doesn’t appear that there is any direct support for detecting additions/removals to Replica.Map or Replica.Array. A primary example of where this is extremely useful is when the server adds a new weapon to the player’s inventory (let’s say it’s represented via a Replica.Map) where each item has a unique string key. Upon server adding this new item to the Map there’s no direct way for the client to respond to this new item being added, as it appears there is only support for registering a callback for OnUpdate or a change to an existing key (through GetValueOnUpdateSignal). This would require listening for a change via the Map’s “OnUpdate” signal, and then having to iterate the entire Map and searching for any keys that weren’t there before just to determine that a new item has been added.

Is there any elegant way of handling this scenario in the current state of Replica that I’m not seeing?

1 Like

Yep, those are definitely limitations that could probably be fixed by adding specialized functions to the Map and Array classes.

On a side note, I no longer use Replica in any of my production projects, so I will still try to maintain it when bugs are found, but I am not eating my own dog food in this scenario unfortunately. Right now I use a similar system of key-based prediction/replication of data, but it doesn’t have the same protections against directly mutating data that Replica has, and just uses a single table with a wrapper to replicate changes to the table.

I have stopped using Replica in my case because of the overhead and verbosity of code for the most part. Replica was a step in the right direction for me, but I have iterated through better solutions to data replication over time. Unfortunately, most of it is highly coupled with my game systems, so I haven’t really documented it well or made it user friendly, but I’ve bundled what I currently use together into a public Free Model on my profile.

1 Like

Thanks for the heads up! We’ll most likely implement something bespoke in that case that is highly influenced by the design of Replica. It’s really nice to see a well thought out approach to data replication on the platform, you’ve definitely created a valuable resource even if it’s primarily to learn from at this point :slight_smile:

How do the arrays work? Incrementally?

Are you sending indexes to the client, or keeping UUIDs? I would strongly recommend the latter for deletions, so just in case the client makes a small mistake, it won’t screw up everything else.

Small mistakes shouldn’t happen unless someone has a weird network connection or is exploiting, but it’s always a consideration I hope.

Would this work with DataStore2?

Have you considered making roblox-ts typings for this? I’d love to use it, but I use roblox-ts.

Have you? roblox-ts is nice and all, but not everybody uses it :stuck_out_tongue:

I never said everybody uses it. I’m just not fluent enough in typefu to write typings for a module like this, and iirc he’s active in the roblox-ts community and think he’s written a package or two.

Oh, no that’s not what I mean at all. Personally, I wouldn’t expect everyone who has written a Lua library that I use to make roblox-ts typings, though.

Okay, that makes sense. If they’re involved in roblox-ts then it probably wouldn’t hurt as much to ask.

Sorry, I just misunderstood.

Thanks, this is very useful. I just have one question: can exploiters take advantage of this on my own game and wreak havoc? Nonetheless, this’ll come handy (especially for my game).

Exploiters can do the exact same thing they usually do: send any data they want to the server. As long as you don’t trust whatever data you receive from the client, this library should not make your game any more vulnerable than it would be if you did trust the client.