I’ve heard people talking about only updating data when it’s changed, but how would you manage to do this?
I have an RPG based game with a ton of data, I now send a remote to the client whenever something changes, which is obviously not the best way to do it…
Would you send the key and it’s new value and if so how would this work for nested tables? And how do you even detect the change?
Any tips would be appreciated!
I usually just cram data into objects like StringValues, and then connect to their property change events to listen for changes to the underlying data. This makes data management simple for me, as I’m using Roblox’s replication system instead of implementing my own.
This is not ideal for my games since it contains hundreds of properties etc, this is a method I want to avoid.
For nested tables, you could use arrays or .-separated strings as indexes like below.
local myData = {nest = {nestnest = {entry = "woo!"}}}
local stringkey = "nest.nestnest.entry"
local arraykey = stringkey:split(".") -- {"nest", "nestnest", "entry"}
-- Reading a value:
function readNested(tab, key)
while #key > 0 do
tab = tab[table.remove(key, 1)] -- Remove first key element and descend
if tab == nil then return end -- If nil, return here. Otherwise next iteration will error
end
return tab -- Return value
end
print(readNested(myData, arraykey)) -- "woo!"
-- Writing a value
function writeNested(tab, key, value)
while #key > 1 do -- Because we need to preserve the last key
local index = table.remove(key, 1)
if not tab[index] then tab[index] = {} end -- Create nested table if not yet present
tab = tab[index] -- Descend
end
tab[key[1]] = value -- Write new value
end
writeNested(myData, {"nest", "othernest", "entry"}, "wee!")
print(game.HttpService:JSONEncode(myData))
-- {"nest":{"nestnest":{"entry":"woo!"},"othernest":{"entry":"wee!"}}}
Use metatables and __newindex? Or have functions which changes current data or adds new data plus confirms data has been updated?
For stuff like this, I created my own “hybrid data” module that uses tables, remotes and ValueObjects to manipulate data. Originally, the module was not intended to use ValueObjects, but after realizing I was firing a remote too often, I decided to use ValueObjects which is where the “hybrid” comes. I only used ValueObjects to store much simpler data and data that was getting sent to the client too often. Some things I stored in them were Coins, Exp, and Level. The rest of my data(items, effects, etc) was passed to the client via remotes.
Maybe give this method a try. It worked very well for me.
Another suggestion is update things/retrieve data when a user needs it. Such as opening an inventory. When a player opens an inventory, fire a remote function and return the data from the server. Using that data, create an inventory for them.
I have a system sending at only 8hz clients’ and a platform cube’s positions.
Results on a powerful system (intended):
For someone with a normal machine (average user experience, oh no!):
On another post a reference to a destroyed Wiki Article:
The latency of the connected clients can be adversely affected when using remote events and functions too often or when sending too much data
I have no idea why my packets are 8kbits (and thus 64kbps). I have yet to optimize, but still it’s insanely large.
Here are a few things I’ll be looking into today:
Multiple events, or the vague term “localized networking” (Me: wait what?). I have used one event object for all of my network exchanges called repEvent. I realize now this likely means everyone listens to everyone and are thus dropping a lot of unintentionally replicated packets. But we can go further apparently by not just having separated server and client Events but also unique Events for each client.
The “highly optimized fit as much data as you can into each request” strategy:
Supposedly by sending the data as a string (from 90 to 40kbps for some other forum-er), or a stream of numbers, you can dimish kbps.
JSONEncode doesn’t support Vector3s, CFrames, or Color3s, all of which are almost all of what I am sending. So perhaps I’ll go on tangent to the above bullet with this stuff?
‘Checksum’ and ‘Sequential Counter for DataSet Changes’ for the case of lost packets.
Stripping arrays of their hierarchies for distribution.
RE’s and RF’s are promised so they may be sandwiched and immediately overridden but eventually they’ll reach their destination, unlike ObjectValues where only the latest value-on-access are found. But that’s all I need. So maybe those?
Replica
It seems you’re pretty comfortable writing serious code?
I’ve done a bunch of work with this, (replicating large tables) and the best solution I’ve found is using delta compression.
--Produces a table that is old + result = new
--nil'ing an already set value is ignored
--handles nesting
function DiffTable(old, new)
local res = {}
local count = 0
for var,data in pairs(new) do
if (old[var] == nil) then
res[var] = data
else
if (type(new[var])=="table") then
--its a table, recurse
local newtable, num = DiffTable(old[var],new[var])
if (num > 0) then
count = count + 1
res[var] = newtable
end
else
local a = new[var]
local b = old[var]
if (a ~= b) then
count = count + 1
res[var] = a
end
end
end
end
return res,count
end
--Produces a table that is result = old + new
function MergeTable(old, new)
local newTable = DeepCopy(old)
if (newTable ==nil) then
newTable = {}
end
for var,data in pairs(new) do
if (type(new[var])=="table") then
newTable[var] = MergeTable(old[var], new[var])
else
newTable[var] = new[var]
end
end
return newTable
end
--Standard lua deep copy routine
function DeepCopy(orig)
local copy
if type(orig) == 'table' then
copy = {}
for orig_key, orig_value in next, orig, nil do
copy[DeepCopy(orig_key)] = DeepCopy(orig_value)
end
setmetatable(copy, DeepCopy(getmetatable(orig)))
else --by value
copy = orig
end
return copy
end
So those are battle tested in a bunch of my projects.
How to use them:
Lets say you have a big fat table that you want to replicate to all clients, and will make random changes to it, and update it pretty frequently (10hz or more!)
--Produce a table to send to all clients
function module:WriteSnapshot(playerData)
local SendFull = false
if (playerData.prevWorldVarsState==nil) then
SendFull = true
end
if (playerData.firstSnap == true) then
SendFull = true
playerData.firstSnap = false
end
local snap = {}
snap.full = SendFull
--Send the worldVars
if (SendFull == true) then
snap.worldVars = DeepCopy(self.worldVars)
else
snap.worldVars = DiffTable(playerData.prevWorldVarsState, self.worldVars)
end
playerData.prevWorldVarsState = DeepCopy(self.worldVars)
return snap --send me to a client!
end
Then on the client once you have the snap table…
function module:ReadSnapshot(snap)
if (snap.full == true) then
self.prevWorldVarsState = nil
self.prevStates = nil
self.prevStates = {}
print("Got full snapshot")
end
--Read/Undelta the worldvars
if (self.prevWorldVarsState == nil) then
self.worldVars = DeepCopy(snap.worldVars)
else
local newState = DeepCopy(snap.worldVars)
self.worldVars = MergeTable(self.prevWorldVarsState,newState)
end
self.prevWorldVarsState = DeepCopy(self.worldVars)
end
And that’s it. It’ll only send down whats changed.
A caveat is that if you nil out something in a table you’re replicating this way, it wont get nil’d on the clients - it’ll stay set.
It also won’t work with anything that can’t be replicated as part of a table in an event.
If you have any success with this, let me know!
HTH
Although it is my own library, and it works fine, I can’t say I recommend Replica as much anymore for data replication; if you read my latest post on that thread, I mentioned that I use something else now
The best way to replicate data, in my experience, is to always “buffer” changes to your data rather than directly mutating that data (I would NOT recommend __index and __newindex for this though, since it’s easy to mess up).
For my current game, instead of changing a player’s data like this:
playerData.coins = 50
I use a wrapper API and do something like this:
playerData:Set('Coins', 50)
That’s essentially what Replica does, but I find Replica’s structure is a bit too verbose. Instead, I would
recommend making your own specialized data replication system system that uses a keypath (a table of nested keys) for setting changes to data like this:
playerData:Set({'Backpack', 'Slot1'}, myItemToPutInSlot1)
or (in the case of what I currently use)
playerData:Set('Backpack', 'Slot1', myItemToPutInSlot1)
This will indirectly mutate the player’s data equivalently to the statement data.Backpack.Slot1 = myItemToPutInSlot1; however, using the indirect :Set() method, we can do things like fire events when data changes, and replicate these changes (and only these changes) to the client.
Although this is ultimately more verbose than player.Coins = 50, it’s a lot simpler in my experience. You also have to be responsible with this kind of system, since you could just call playerData:Get().Coins = 50, and it will set your coins to 50 on the server, but never replicate this to the client. However, I think it’s really easy to be responsible, and you have to actually try to mess it up.
This also makes it easy to subscribe to changes to your data, since all changes are indirectly made through the :Set call.
e.g.
playerData:Subscribe({'Coins'}, function(oldCoins, newCoins)
print("Coins changed from", oldCoins, "to", newCoins, "!")
end)
Thank you all for the suggestions! I marked DataBrain’s post as solved.