RemoteTable - Replicate tables over the network!

RemoteTable - Replicate tables over the network!
Download

RemoteTable is a special type of network library which uses proxy tables to detect changes to a table and send only the changes over the network to reduce network usage. It uses Packet as the networking library to optimize data usage via buffers.


Features
  • Automatically replicate changes via proxy tables
  • Hash paths to reduce network and cpu usage
  • Batch updates and rate limit
  • Handle and optimize dynamic indexes created at runtime
  • Compatible with ProfileStore
  • Strictly Typed

Planned Features
  • Subscribe to state changes
  • Multi script support

Limitations
  • Not very efficient for relatively large (20-30) item arrays that needs to preserve order
  • Keys for tables can only be number or string and values can only be types Packet.Any supports
  • Can’t use table.insert and table.remove or other standard table library functionality [Instead use RemoteTable.Insert, RemoteTable.Remove, RemoteTable.FastRemove]
  • Only 255 tokens can be registered at the same time. [This can be changed by editing Packets module]
  • Currently a client can’t listen to 2 tokens that share an identical path. [This will be fixed in 1.1]

Quick Start
Setup
  1. Get the module here.
  2. Put it under replicated storage.

TIP: You can link your own Packet module to further benefit from batching.

Advanced Setup

You can edit you config file to further customize RemoteTable.
Config file is just a module that returns a table and is located at RemoteTable.Shared.Config.
Documentation for each table property:

Packet -> require(Path.To.Your.Packet)
UpdatesPerSecond -> How often changes are sent.
HashTableSize -> The hash table size for RemoteTable to operate on.

INFO: Each path gets a unique id from the hash table. Default value for the hash table space is 2000 and it is safe to use up to %70 (1400) of this hash table space. For most use cases 1400 paths will be enough for something like user data but incase there is an inventory system with a lot of items you can change this value using this formula max_paths * 1.4.

WARNING: This unique id is unfortunately per RemoteTable module NOT per created remote table. Fortunately players usually get the same user data template so each identical path can use the same id not adding to the total unique path count.

Server.lua

local RemoteTable = require(game.ReplicatedStorage.RemoteTable).Server
-- or alternatively require(game.ReplicatedStorage.RemoteTable.Server)

local ExampleTable = {
	Table = {
		Key1 = "Path of this value is 'Table\Key1' ",
	},
	Array = {"A","B","C"} -- number indexes
}

game.Players.PlayerAdded:Connect(function(player: Player)
	--Create a unique token for each player
	local token = "PlayerDataToken"..player.UserId

	--Initialize a new remote table with the given token
	local remote_table = RemoteTable.ConnectTable(ExampleTable, token)

	--Give client permission to listen to the remote table
	remote_table:AddClient(player) --or RemoteTable.AddClient(token, player)

	task.wait(2) -- wait for client to test if changes replicate
	remote_table.Data.Table.Key1 = player.UserId -- should replicate
end

game.Players.PlayerRemoving:Connect(function(player: Player)
	--Release token for future players
	local token = "PlayerDataToken"..player.UserId
	RemoteTable.ReleaseToken(token)
end)

Client.lua

local RemoteTable = require(game.ReplicatedStorage.RemoteTable).Client
-- or alternatively require(game.ReplicatedStorage.RemoteTable.Client)

local token = "PlayerDataToken"..player.UserId
local ready_signal = RemoteTable.GetRemoteTableFromToken(token)

-- wait until table is ready to replicate
local registered, token, data = ready_signal:Wait()
if not registered then
	warn("Token haven't been registered by server!")
end

--TIP: Enable "Show Tables Expanded by Default" for a better visual feedback
while task.wait(0.2) do
	--Data changes automatically
	print(data)
end

TIP: It is usually recommended to have 1 remote table token per player but the library is designed in a way that is more flexible. Multiple players can listen to the same token.

Bonus: Example with ProfileStore

Coming soon gimme a min gang


Documentation
Server
--Requiring the module
local RemoteTable = require(RemoteTable.Server) -- or require(RemoteTable).Server
RemoteTable.ConnectTable
    - Creates a new remote table and initializes it.
    @param tbl: Table to be tracked
    @param token_alias: String token_alias for the token
    @return RemoteTable<T>: Newly created remote table object
RemoteTable.AddClient
    - Authorizes a client to listen to a token
    @param player: Client to be added
    @param token_alias: String token_alias for the token
RemoteTable.RemoveClient
    - Disconnects a client and removes permissions to listen for changes.
    @param player: Client to be added
    @param token_alias: String token_alias for the token
RemoteTable.ReleaseToken
    - Releases the token and disconnects the remote table associated with the token.
    @param token_alias: String token_alias for the token
RemoteTable.GetRemoteTable
    - Gets the remote table from token alias.
    @param token_alias: String alias of the token
    @return RemoteTable<T>?: returns nil if remote table does not exist

Client
-- Requiring the module
local RemoteTable = require(RemoteTable.Client) -- or require(RemoteTable).Client
RemoteTable.GetRemoteTableFromToken
    - Starts listening to the remote table with the specified token
    @param token_alias: String token_alias of the token
    @return Signal: A signal that fires when the remote table is ready to be used by client
]]

Versions

1.0

  1. Initial Release
  2. Added XXH32 as the main hashing algorithm. Thanks to @XoifailTheGod
9 Likes

:fire:
will use in my next projects

Hold on, does it support instances? (Obviously the server won’t see client-sided instances but how does the module act?)
Also, except those and some data-types since I don’t know how it works with them, how different is it from sending a JSON encoded string over remotes?

Instances can be set as a value but keys don’t support instances. As I said

Keys for tables can only be number or string and values can only be types Packet.Any supports

Second thing. This module optimizes data usage up to 100x or more compared to sending over the json each time. Module only sends changes with 3 bytes to identify the token and the path. The values network cost is optimized by packet and can vary.

1 Like

This is extremely cool, but I would consider it the _G equivalent of networking where it is convenient, but is not a good habit to use for everything.

if this is being used for special datatypes (a chess board game maybe?) this could make programming 100x more convenient.

Great module :+1:

If you want a fast hash :eyes:

1 Like

This was originally made as an alternative to replica. I wanted something that was more automatic than manually entering path. It will be a lot better after v1.1 which will introduce subcription to paths. So you can use it directly in ui code. Very excited for it!

Thank you for the resource Ill look into it! I’m not really advanced with hashing and cryptography so I just chose the simplest one.

XXHash32 is the fastest hash because its not meant for security, think a facebook engineer made it so its perfect for hash tables or checksums
XXH32(Message: buffer, Seed?: number) -> number

  1. Added XXH32 as the main hashing algorithm. Thanks to @XoifailTheGod

Thank you again! Very cool library you got.

1 Like