Networking module 'metadata' - custom attributes?

Help and Feedback Creations Feedback
#1

I spent a few hours working on a custom attribute module for my game Corautmors. I call it ‘metadata’
I will never open source it to the public
But I did find it really cool and lightweight (~400 LoC) so I wanted to post about it

But how it works is:
First you require the module

local metadataServer = require('server@metadataServer')

Then you create a part to assign the metadata to. This is the part you’re adding ‘attributes’ to, like with roblox’s attribute system

local part = Instance.new('Part', workspace.Terrain)

Next, you create something called a metareference. In this example I create 100 of them, named i, with a bitsize of 4 (max is 0b1111) and a default value of 15 (0b1111). I insert them into a table stuff which is the ‘metareferences’ you pass into a partition, which ill get to in a sec

local stuff = {}
for i = 1, 100 do
	table.insert(stuff, {name=`{i}`, bitSize=4, default=15})
end

Here I’m making the metadata and then partitioning it.

local pt = metadataServer.New(part, stuff):Partition()

A partition is how the metareferences is divided. The metarefs table ‘stuff’ contains all the metarefs, and with a partition you can select which metarefs you wish to replicate as the first argument: {string}, containing the names of the metarefs for that partition. In this example I’m passing no arguments, because I want to use all the metarefs in that partition. However this is how you’d generally use partitions:

local md = metadataServer.New(part, {
{name=`apples`, bitSize=4, default=15}
{name=`bananas`, bitSize=4, default=15}
{name=`broccoli`, bitSize=4, default=15}
{name=`asparagus`, bitSize=4, default=15}
})
local fruits = md:Partition({'apples', 'bananas'}) -- this partition can only replicate apples and bananas
local vegetables = md:Partition({'broccoli', 'asparagus'}) -- this partition can only replicate broccoli and asparagus

-- as a replication example
fruits:PushReplicator(somePlayer, somePlayer2) -- let us replicate the fruits to some player. note: this doesnt mean replicate
fruits:Set('apples', 5) -- set apples to 5
fruit:Replicate() -- replicate it to someplayer and someplayer2

Here I’m configuring a bunch of settings

pt:SetDefaultBufferRate(1)
pt:SetThrottleOnDefault(true)
pt:PushReplicator(game.Players:WaitForChild('Ilucere'))

SetDefaultBufferRate basically throttles the amount of replications per second. If it’s 1, then only 1 replication per second can be done with that partition. If its 1/16, then 16 replications can be done in 1 second.

And finally, we replicate. In the for _, v loop I’m just setting each value to something random (within its 4 bit constraint). After that, I’m replicating all the changes to the client.

for _ = 1, 1000 do
	for _, v in next, stuff do
		local x = math.random(0, 15)
		print(v.name, x)
		pt:Set(v.name, x)
	end
	
	pt:Replicate()
	task.wait(1)
end

That was a short runthrough of a server-side example. As you expect, the client recieves the data and updates it for a specific object. However, we’re still pretty surface level. So I’m gonna go into more depth

The module itself fundamentally works by combining binary numbers into one bitstream, with a max size of 31 bits. With the fruits and veggies example, they’re all conveniently combined into a 16 bit bitstream. However, what if we had 2 more fruits and 2 more vegetables, each with a bitSize of 4. Then that would be 32 bits, and we cant have that. So instead I create a new bitstream and put the extra fruit/veggie that’s overflowing the bitstream, in the new bitstream. And this process is repeated up to 15 times, as I only want a maximum number of 15 bitstreams in the system, which is reasonable for a pretty big game.
When a new partition is made, its metareferences are replicated to any new replicant, along with each metarefs bitstream (called groups) and its shift. When the client receives data, it uses the shift, group, and the bitSize of the number, and conveniently puts it into a dictionary with the key being the metarefs name, and the data extracted from the bitstream (using the group, shift, and bitSize together) being the value.

pushMetadataChange = function(instance, ...)
		local metadata = totalPartitions[instance]
		
		local bitStreams = {...}
		local partitionStartingGroupMix = bitStreams[1]
		local partitionNumber, initialGroup = bit32.extract(partitionStartingGroupMix, 0, 4), bit32.extract(partitionStartingGroupMix, 4, 4)
		
		local targetPartition;
		for _, v in ipairs(metadata.partitions) do
			if (v.number == partitionNumber) then
				targetPartition = v
				break
			end
		end
		
		local maxGroup = initialGroup + #bitStreams
		for _, metaRef in ipairs(targetPartition.metadataReferences) do
			if (metaRef.group < initialGroup or metaRef.group > maxGroup) then
				continue
			end
			local value = bit32.extract(bitStreams[metaRef.group], metaRef.shift, metaRef.bitSize)
			if (type(metaRef.default) == 'boolean') then
				value = (value == 1) and (true) or (false)
			else
				if (metaRef.minMax) then
					if (metaRef.minMax.Min < 0) then
						value += metaRef.minMax.Min
					end
				end
				
				if (metaRef.precision) then
					value /= 10^metaRef.precision
				end
			end
			targetPartition.localData[metaRef.name] = value
			print(metaRef.name, value)
		end
	end,

That’s the clientside code for receiving data.
You may also notice this line:

local partitionNumber, initialGroup = bit32.extract(partitionStartingGroupMix, 0, 4), bit32.extract(partitionStartingGroupMix, 4, 4)

In the system, when determining the group and shift of each metaref, I start at shift 8. This is to accomodate for the partition number and the initial group (ill get to both in a sec) being the first 8 bits (0b11111111). The partition number is just an ID representing which partition is being modified or whatever in the remote. A metaref table is unique to each partition. The initial group is which group (bitstream) we’re starting on when we receive the bitstreams. This is the minimum modified bitstream. Say our bitstreams looks like this on the server: 18 632 0 0. We trunucate the extra 0’s to save some data, and are left with 18 632. The first 8 bits of that 632 represents the initial group and partition number keep in mind.

I don’t know what else to talk about or go more in depth about, so AMA if you’re curious.

1 Like
#2

Oh yeah I forgot to mention a couple things

  1. It’s more efficient than roblox’s attributes because of its bulk replication. Setting 1000 different attributes bumps my network recv to like 90kb/s, but with this module adn just a singular :Replicate it is only 5kb/s.
for _ = 1, 1000 do
	for _, v in next, stuff do
		local x = math.random(0, 15)
		print(v.name, x)
		pt:Set(v.name, x)
	end
	
	pt:Replicate()
	task.wait(.01)
end

^ this was like 3kb/s

local plr = game.Players:WaitForChild('Ilucere')
for _ = 1, 1000 do
	for _, v in next, stuff do
		plr:SetAttribute(v.name, math.random(0, 15))
	end
	task.wait(.01)
end

this was over 30x that
And paired with my throttling system i can get an even lower recv. Partitions also allow selective replication, as you know, using the :PushReplicator function, so you can do a chunk-only replication type of thing between clients, where clients can only recieve data from nearby clients.

  1. Here’s the type things
type metadataSingularReference = {
	name:string,
	bitSize:number,
	default:number|boolean,
	minMax: NumberRange?,
	precision: number?,
	startingBit:number,
	group:number,
	shift:number,
	private:boolean?,
}
type metadataReferences = {metadataSingularReference}

You can set privacy, to prevent a metaRefs replciation altogether. You can set the decimal precision, etc. It’s a simple but elegant module.

1 Like