RemoteTable-Light - Replicate tables over the network!

RemoteTable-Light - Replicate tables over the network!
Download | Source | Github Download

RemoteTable-Light is a special type of network library that allows you to replicate tables from server to client with ease and minimal cost. RemoteTable-Light is meant to be a replacement to Replica. It uses a slightly modified version of Packet as the networking library to optimize data usage via buffers.


Features
  • Automatically replicate tables to connected listeners
  • Selective replication
  • Table ids to reduce network and cpu usage
  • Buffer based batching
  • Support dynamic and nested tables
  • Provide all table mutation functionality from standard table library
  • Compatible with ProfileStore
  • Strictly Typed
  • Multi script support
  • Multiple listeners per remote table

What makes RemoteTable-Light better than RemoteTable
  • Very efficient with table operations
  • Selective replication to hide internal data from players
  • 1.3x faster initial write, 3x faster write, 20x faster read
  • Pure tables, references can be reused for mutation

What makes RemoteTable-Light better than Replica
  • Uses ids and references to identify tables instead of string arrays (O(1) lookup)
  • Serializes and batches events into buffers
  • Uses buffers to take advantage of roblox compression
  • More reliable data signal behaviour (ChildAdded, ChildRemoved, Changed)
  • Cleaner API (only 5 core server functions)
  • Faster operations, no need to deal with strings, pure references
  • Super easy to setup, plug and play compared to replica which requires you to call :RequestData yourself (RemoteTable Light handles connection logic internally)
  • Timeouts for wait operations
  • Clean modular code for devs to read and customize to their needs

Quick Start
Setup
  1. Get the module here.
  2. Put it under replicated storage.
Server.lua
local RemoteTable = require(game.ReplicatedStorage.RemoteTableLight).Server

local SelectiveReplicationData = {
	Progression = {
		Private = false
	},
}

local PlayerDataTemplate = {
	Cash = 1,
	Experience = 0,
	Progression = {
		Private = {
			PrivateValue = "This won't replicate",
		}
	},
	Inventory = {"Phone", "Keys", "Wallet"},
	Perks = {},
}

game.Players.PlayerRemoving:Connect(function(player: Player)
	RemoteTable.Destroy("PlayerData"..player.UserId)
end)

game.Players.PlayerAdded:Connect(function(player: Player)
	local token = "PlayerData"..player.UserId
	local player_data = RemoteTable.Create(token, PlayerDataTemplate, SelectiveReplicationData)
	RemoteTable.AddClient(token, player)

	--[[
		Give client a small delay to connect
		This is only for demonstration purposes
		In production it doesn't matter when client connects
		as long as server does .AddClient() for the client
	]]
	task.wait(2)
	
	RemoteTable.Set(player_data, "Cash", 1000)
	task.wait(1)
	
	RemoteTable.Set(player_data.Progression.Private, "PrivateValue", "This change won't be replicated.")
	task.wait(1)
	
	local perk = {Type = "Additive", Target = "WalkSpeed", Value = 5}
	RemoteTable.Set(player_data.Perks, "SpeedPerk", perk)
	task.wait(1)
	
	-- We can reuse the perk reference to edit the perk value
	RemoteTable.Increment(perk, "Value", 3)
	task.wait(1)
	
	RemoteTable.Increment(player_data, "Experience", 100)
	task.wait(1)
	
	-- {"Coffee", "Phone", "Keys", "Wallet"}
	RemoteTable.Insert(player_data.Inventory, 1, "Coffee")
	task.wait(1)
	
	-- {"Coffee", "Phone", "Wallet"}
	RemoteTable.Remove(player_data.Inventory, 3)
	task.wait(1)

	-- {"Wallet", "Phone"}
	RemoteTable.SwapRemove(player_data.Inventory, 1)
	task.wait(1)

	-- {}
	RemoteTable.Clear(player_data.Inventory)
	task.wait(1)
	
	RemoteTable.RemoveClient(token, player)
	
	-- This won't replicate as the client is removed from listeners
	RemoteTable.Set(player_data, "Cash", 99999)
	task.wait(1)
end)
Client.lua
local Player = game:GetService("Players").LocalPlayer
local RemoteTable = require(game.ReplicatedStorage.RemoteTableLight).Client

local token = "PlayerData"..Player.UserId
local data = RemoteTable.WaitForTable(token)

RemoteTable.GetSignal(token, "ChildAdded", {"Inventory"}):Connect(function(child, key)
	warn("Inventory added: ", child)
end)

RemoteTable.GetSignal(token, "ChildRemoved", {"Inventory"}):Connect(function(child, key)
	warn("Inventory removed: ", child)
end)

RemoteTable.GetSignal(token, "Changed", {"Cash"}):Connect(function(new, old)
	warn("Cash changed: ", new)
end)


--TIP: Enable "Show Tables Expanded by Default" for a better visual feedback
while task.wait(0.2) do
	--Data changes automatically
	print(data)
end
Bonus: Example with ProfileStore
DataService
local DataService = {}

local Players = game:GetService("Players")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local ServerScriptService = game:GetService("ServerScriptService")

local ProfileStore = require(ServerScriptService.ProfileStore)
local RemoteTable = require(ReplicatedStorage.RemoteTableLight).Server
local Signal = require(ReplicatedStorage.RemoteTableLight.Shared.Zignal)

local PROFILE_TEMPLATE = {
	Cash = 0,
	Items = {},
}
local PlayerStore = ProfileStore.New("PlayerStore", PROFILE_TEMPLATE)

type PlayerProfile = typeof(PlayerStore:StartSessionAsync())
type ProfileData = typeof(PROFILE_TEMPLATE)
type PlayerData = {
	Profile: PlayerProfile,
	Data: ProfileData,
}

local PlayerDatas = {} :: {[Player]: PlayerData}

DataService.PlayerDatas = PlayerDatas
DataService.DataLoaded = Signal.new()

function DataService.GetData(player: Player): PlayerData?
	return RemoteTable.Get(`PlayerData{player.UserId}`)
end

local function PlayerAdded(player: Player)
	local data_token = `PlayerData{player.UserId}`
	
	local profile = PlayerStore:StartSessionAsync(`{player.UserId}`, {
		Cancel = function()
			return player.Parent ~= Players
		end,
	})
	
	if not profile then
		player:Kick(`[{script.Name}]: Profile load fail - Please rejoin`)
		return
	end
	
	profile:AddUserId(player.UserId)
	profile:Reconcile()

	profile.OnSessionEnd:Connect(function()
		PlayerDatas[player] = nil
		player:Kick(`[{script.Name}]: Profile session end - Please rejoin`)

		--Release the token
		RemoteTable.Destroy(data_token)
	end)
	
	if not player:IsDescendantOf(Players) then
		profile:EndSession()
		return
	end
	
	-- Connect and add client right after player data loads
	local player_data = RemoteTable.Create(data_token, profile.Data)
	RemoteTable.AddClient(data_token, player)

	-- Override profile.Data with the replicated remote table
	profile.Data = player_data
	PlayerDatas[player] = {
		Profile = profile,
		Data = player_data
	}
	print(`[{script.Name}]: Profile loaded for {player.DisplayName}!`)
	DataService.DataLoaded:Fire(player, player_data)
end

for _, player in Players:GetPlayers() do
	task.defer(PlayerAdded, player)
end

Players.PlayerAdded:Connect(PlayerAdded)
Players.PlayerRemoving:Connect(function(player)
	local profile = PlayerDatas[player].Profile
	if profile ~= nil then
		profile:EndSession()
	end
end)

-- Edit functions
DataService.Set = RemoteTable.Set
DataService.Increment = RemoteTable.Increment
DataService.Insert = RemoteTable.Insert
DataService.InsertAt = RemoteTable.InsertAt
DataService.Remove = RemoteTable.Remove
DataService.SwapRemove = RemoteTable.SwapRemove
DataService.Clear = RemoteTable.Clear

return DataService
Server.lua
local ServerScriptService = game:GetService("ServerScriptService")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local Players = game:GetService("Players")

local DataService = require(ReplicatedStorage.RemoteTableLight.Examples.ProfileStoreExample.DataService)

DataService.DataLoaded:Once(function(player: Player, data)
	-- Add 100 cash each join
	DataService.Increment(data, "Cash", 100)
	
	-- Grant a ticket to player everytime they join
	DataService.Insert(data.Items, {Name = "Ticket", Value = 10})
end)
Client.lua
local Player = game:GetService("Players").LocalPlayer
local ReplicatedStorage = game:GetService("ReplicatedStorage")

local RemoteTable = require(ReplicatedStorage.RemoteTableLight).Client

local TOKEN = `PlayerData{Player.UserId}`
local data = RemoteTable.WaitForTable(TOKEN)

--TIP: Enable "Show Tables Expanded by Default" for a better visual feedback
while task.wait(0.2) do
	--Data changes automatically
	print(data)
end

TIP: Private fields can be edited directly like regular tables since they are not replicated.


Documentation
Server
--Requiring the module
local RemoteTable = require(RemoteTable).Server -- or require(RemoteTable.Server)
RemoteTable.Create<T>(token_name: string, template: T, selective_replication: any): T
	- Creates a new remote table with given selective replication data
RemoteTable.AddClient(token_name: TokenName, clients: Player | {Player})
	- Permits client(s) to listen to a token
RemoteTable.RemoveClient(token_name: TokenName, clients: Player | {Player})
	- Revokes client(s) listening permission(s) and disconnects them
RemoteTable.Destroy<T>(token_name: string)
	- Releases the token and disconnects the remote table associated with the token.
RemoteTable.Get(token_name: string): any?
	- Gets the remote table from token name
RemoteTable.Set(parent: any, key: Key, value: any)
	- Set a value to the table with given key
RemoteTable.Increment(parent: any, key: Key, value: number)
	- Changes the key by value
RemoteTable.InsertAt<V>(tbl: {V}, pos: number?, value: V)
	- table.insert with a pos argument
RemoteTable.Insert<V>(tbl: {V}, value: V)
	- table.insert without a pos argument
RemoteTable.Remove<V>(tbl: {V}, pos: number?): V?
	- table.remove
RemoteTable.Clear(tbl: any)
	- table.clear

Client
-- Requiring the module
local RemoteTable = require(RemoteTable).Client -- or require(RemoteTable.Client)
RemoteTable.IsRemoteTableReady
	- Checks if a remote table with a specific token is ready
	@param token_name: String alias of the token
	@return boolean
RemoteTable.WaitForTable
	- Returns the table if available, waits for it if not.
	@param token_name: String alias of the token
	@param timeout: Timeout in seconds. Returns nil after timing out
	@return data: Ready-only replicated table.
RemoteTable.GetSignal
	- Gets the apropriate data signal
	@param token_name: String alias of the token
	@param signal_type: "Changed" | "ChildAdded" | "ChildRemoved"
	@param path_list: A string array representing the desired path
	@return Signal: Signal that fires (new, old) data or (value, key) for child signals
RemoteTable.DestroySignal
	- Cleans up the specified signal
	@param token_name: String alias of the token
	@param signal_type: "Changed" | "ChildAdded" | "ChildRemoved"
	@param path_list: A string array representing the desired path

Versions

1.0

  • Initial release.

1.1

  • Fixed: Unregister logic on client
  • Small correction to Changed signal behaviour

1.2

  • Fixed: Token registry not registering all tokens on initialize. Thanks to @calinpj04
  • Destroying remote table now cleans up everything and releases all references.
  • Improvement to listener logic
  • Removing a client from server cleans up everything on client
  • .Get now returns the root table of the remote table
  • Fixed: Insert inserting at wrong index. Thanks to @phantasmal_deathrayz
  • Added: Seperate opcode for Insert and InsertAt
  • Fixed: Player on leave error
36 Likes

Once again, he cooked another masterpiece!

2 Likes

This looks promising. I will check it out. It really is a lot more convenient, simpler, and intuitive.

Also, Download link isn’t working, only Source.

1 Like

any chance this’ll be uploaded as a wally package?


Roblox messing with me

Very much sorry for the inconvenience. You can always get the latest release here

No, personally I will not maintain wally. But if you want to do it feel free to upload it to wally

1 Like

Also a question, is it necessary to use DestroySignal? Does the connection stay connected after the script that created it is destroyed?

Unfortunately this is an issue on roblox end. Script destruction is very much doomed from start. I would not connect from script that can be destroyed. If you mind explaining your use case I can suggest a better pattern. (There is currently no way to reliably detect script destruction from outside like loleris mentioned.)

I would like to make a genuine question about this then.

I personally have a module I call IndependentFunction, which runs functions separately from the scripts they were originally called on. This made it possible for me to check when a script itself is destroyed through a connection that’s connected in the script itself.

This is done by passing functions between BindableEvents. There is a client “processor” and a server “processor” in StarterPlayerScripts and ServerScriptService respectively that listen to when the BindableEvent is fired and call the passed function in their own contexts.

I’m not really experienced with how signal modules work, but perhaps this could be a way to detect when the script is destroyed and then call clean up? I can send the module’s link if you want to check it. It’s really simple and I made it because I like to work with scripts placed directly under tools.

This fixed several issues for me like 2 years ago when I had instances being created in a script under a tool that took long to be destroyed through task.delay (I used this because I wanted them to have a special effect before being destroyed). The instances would never be destroyed when the character that casted the skill that created the instance died or left because threads are cancelled when a script is destroyed. After I implemented this module, I was able fix it, and I’ve never had any issues so far.

What you can do is make a bindable interface that fires the bindable from the RemoteTable signal callback.

My personal suggestion would be to take a step back and realize you’re overengineering a simple problem just to use an unecessarily painful tool.

Fyi the example of problems I would have with this are local scripts placed under ScreenGuis that have ResetOnSpawn enabled and somehow rely on the character, such as popping an ui when a specific stat in their data is changed. I believe there can be more examples as well, such as in combat games checking when a skill is added or when the player levels up in a script under the character.

Imagine having to use a more complex organization for every single script that could suffer of this problem in a way such that it considers when the player’s character is respawned, disconnect connections, and then reload? I believe that this is overengineering, not a simple solution that takes just some lines.

Anyway, I appreciate the conversation.

1 Like

Glad you’re appreciating it. I find it very valuable for scripters to exchange their techniques.

For the problem. I always have a script that requires the ui modules. These ui modules are not directly under uis but rather in ReplicatedStorage. I never have .ResetOnSpawn on any of my guis. I always connect once and reset stuff OnCharacterAdded rather than relying on roblox to do it for me. This way I can have precise control over what happens. And never face issues with script lifetimes and temporary runners

1 Like

You mention that you’re using a slightly modified version of Packet, what changes have you made to Packet?

A single function was added to packets to allow for contiguous deserialization.
Here’s the function under the main module itself:

function Packet:DeserializeReturnOffset(serializeBuffer, offset)
	Import({Buffer = serializeBuffer, BufferLength = buffer.len(serializeBuffer), BufferOffset = offset or 0, Instances = {}, InstancesOffset = 0})
	local values = table.create(#self.Reads)
	for index, func in self.Reads do values[index] = func() end
	return Export().BufferOffset, values
end

and the type of it:

DeserializeReturnOffset:(self: Packet<A..., B...>, serializeBuffer: buffer, offset: number?) -> (number, {any})
1 Like

Sorry, wrong reply. But is there any way to make all the replication private by default? Also, can changes be detected to values inside of dictionaries? For example changes made to items in an inventory like below;

local PlayerDataTemplate = {
	Inventory = {
		["Phone"] = {
			Amount = 1,
		},
		["Keys"] = {
			Amount = 2,
		},
		["Wallet"] = {
			Amount = 3,
		},
	},
}

If I change the amount of an item, I wan’t to be able to detect that on the client and respond accordingly.

Making all replication private by default removes the entire point of the library. Replication is already private per client added. Unathorized clients can’t listent to a table.

For the second question you can do this:

local data = RemoteTable.WaitForTable(TOKEN)
for item, _ in data.Inventory do
  RemoteTable.GetSignal(TOKEN, {"Inventory", item, "Amount"}):Connect(function(new, old)
    print(`{item} amount changed from {old} to {new}`)
  end)
end

Okay thank you, but what I meant by private default is the selective replication data. It would be easier to whitelist certain keys to replicate rather than blacklisting keys.

You can easily reconcile all keys to be false by default with a simple loop.

local template = ...
local selective_replication = ...
for k, v in template do
  if selective_replication[k] == nil then
    selective_replication[k] = false
  end
end

Hi, I’ve got a couple of questions here about how replication works with your module and I’m curious on your input.

When a RemoteTable is destroyed on the server-side through RemoteTableServer.Destroy(token), does this automatically clean up any internal events and threads pertaining to this table on both the server and client, so that it’s pretty much only remaining as a table that the developer must clean up on their end (usually by t = nil)?

So if I had a custom player list on the client that listened to player joins to display everyone’s Level, which is received from their data table, could we expect for this to be cleaned up properly?

--!strict

-- No matter how many players join and leave, we can expect
--  for all RemoteTables to be cleaned up.

local conn: {[number]: any} = {}

local function UpdateText(obj: GuiComponent, data: any)
    obj.Text = `Level {data.Level}`
end

local function OnPlayerAdded(Player: Player)
    local dataToken = `userData/{Player.UserId}`
    local data = RemoteTableClient.WaitForTable(dataToken, 5)
    
    if not data then
        return
    end

    local obj = CreateGuiComponent(Player)
    local changedSignal = RemoteTableClient.GetSignal(`user/{Player.UserId}`, "Changed", {"Level"})
    
    conn[Player.UserId] = changedSignal:Connect(function(newValue: number)
        UpdateText(obj, data)
    end)
end

local function OnPlayerRemoving(Player: Player)
    local con = conn[Player.UserId]

    if not con then
        return
    end

    conn:Disconnect()
    conn[Player.UserId] = nil
    -- Imagine we also clean-up the GuiComponent
end

Players.PlayerAdded:Connect(OnPlayerAdded)
Players.PlayerRemoving:Connect(OnPlayerRemoving)
for _, Player in Players:GetPlayers() do
    OnPlayerAdded(Player)
end

Additionally, I noticed that there’s a DestroySignal function on the client. When this gets called with a path being passed in that was created through GetSignal earlier, is this just disconnecting all of its existing connections and leaving the actual clean-up of the Signal object itself to the Luau garbage collector?

If this is indeed the case, I assume this means that DestroySignal never actually has to be called as long as we ensure all connections generated through this Signal interface is properly cleaned up when we don’t need them anymore?

I ask this because I often listen to the same path in the table for changes across several connections/signals across scripts, and if I had one connection that was short-lived, I would like to rest easy knowing I can disconnect this connection without having all of my other connections break. This behavior doesn’t sound possible if we’re required to call DestroySignal.

Finally, I’m assuming that all Signals pertaining to each table get cleaned up regardless when it gets destroyed on the server-side.

When a RemoteTable is destroyed on the server-side through RemoteTableServer.Destroy(token), does this automatically clean up any internal events and threads pertaining to this table on both the server and client

Yes internal threads do get cleared but signals stay intact just in case remote table is recreated with the same token. So disconnecting is a choice left to the user. For server side when a RemoteTable is destroyed it only releases the internal references maintained by RemoteTable. The table stays intact.

Additionally, I noticed that there’s a DestroySignal function on the client. When this gets called with a path being passed in that was created through GetSignal earlier, is this just disconnecting all of its existing connections and leaving the actual clean-up of the Signal object itself to the Luau garbage collector?

Unfortunately destroy signal is literally destroying the signal and getting rid of it completely with the given path. You can still store connections on your own and disconnect from each individual system.

Note that if you listen to a temporary remote table that might be destroyed during runtime you are expected to disconnect on your own. Although If there is enough demand for it I might add a way to detect when a remote table is destroyed.