Pack - A packing/compression utility for entities

Resources Community Resources
, , ,
#1

WARNING
This module relies on the new buffer type which is not yet available for use in live places!

Introduction

Hello! I am releasing my compression utility which is inspired by the Source engine’s network data tables. I wrote this module to handle compression & decompression of entity states sent over the network. It has built in support for delta compression, bytes, shorts, integers and vectors.

API

Pack.Types

Contains all supported data types, both signed and unsigned.

  • Float
  • Short
  • Double
  • Vector
  • String
  • Integer
  • Boolean
  • UnsignedByte
  • UnsignedShort
  • UnsignedInteger
type BaseValues = number | Vector3 | boolean | string

A type which accepts all values supported by the utility.

function Always (BaseValues) -> (() -> (BaseValues))

Acts as a wrapper for values that will always be included into the packed result regardless of whether they changed or not.

function Pack.new(Layout: {[string]: CompressionType) -> Packer

WARNING
Layout can only contain a maximum of 53 keys

Takes in a dictionary Layout where each key is a string and each value is one of the types available in Pack.Types

function Packer.Pack(Values: {[string]: BaseValues}, PreviousValues: {[string]: BaseValues}?) -> buffer

Takes in Values and an optional PreviousValues in the layout of Layout and returns a buffer used for unpacking. If no PreviousValues is provided it will assume that all values have changed.

function Packer.Unpack(Stream: buffer, Destination: {[string]: BaseValues}) -> ()

Takes in the result of Packer.Pack and unpacks it into the Destination table, switching old values with the new ones.

Performance

Layout used for benchmarks 
local Packer = Pack.new({
	Always = Types.UnsignedByte,
	String = Types.String,
	UnsignedShort = Types.UnsignedShort,
	Boolean = Types.Boolean,
	Vector = Types.Vector
})

Benchmark       | Pack    | Pack x100 | Unpack  | Unpack x100
--------------------------------------------------------------
Fastest Time    | 1 μs    | 94 μs     | 1.29 μs | 136 μs
Average Time    | 1.21 μs | 115 μs    | 1.63 μs | 157 μs
50th Percentile | 1.09 μs | 110 μs    | 1.39 μs | 155 μs
90th Percentile | 1.59 μs | 144 μs    | 2.40 μs | 183 μs
95th Percentile | 4.50 μs | 190 μs    | 7.49 μs | 230 μs

Download

Example.rbxl (60.5 KB)
Pack.lua (13.2 KB)

License

License 
Copyright 2023 Axen

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
23 Likes
BridgeNet2, v1.0.0 | A blazing fast networking library for Roblox
How to make a client sided part that replicates to the server
#2

UPDATE 1

  • Cleaned up some of the code.
  • Added support for booleans and strings.
  • Added support for dictionary formatting, take a look at the example file to get an idea of how this works.

NOTE: It is suggested to use the Source Code under Downloads as the source of the modules latest version as the example file might not be updated with every change.

2 Likes
#3

Is there a way to use an Instance in the CompressionTable? e. g. I need to send the Workspace instance to the players using remote events

#4

There isn’t, you just have to send it outside the compression table.

#5

So I should send list {compressionTable, Workspace}?

Also I’ve go an idea to send encoded instances using numbers / strings / vector2int16s. For example one value in the encoded list is vector2int16 and the server knows that vector2int16 with values (0; 0) is a workspace, so once we get this vector with this values, we just set a workspace as an arugment in the decoded data

What do you think about this idea?

#6

VERSION 2.0.0

THIS VERSION IS NOT BACKWARDS COMPATIBLE WITH PREVIOUS RELEASES

  • Almost full rewrite of the internals
  • Switched string.pack/unpack to the new buffer type
  • Removed support for array based layouts, now only dictionary ones are supported
#7

Looks very useful! However I have one question, how did you do the benchmarking tests for this module?

#8

Here is the code I used for the benchmarks. Benchmark is a utility module which I use to do all my benchmarks since it formats the run times and it has a few extra bells and whistles which I find useful, though the main portion of it is just a simple os.clock() time comparison.

local Packer = Pack.new({
	Always = Types.UnsignedByte,
	String = Types.String,
	UnsignedShort = Types.UnsignedShort,
	Boolean = Types.Boolean,
	Vector = Types.Vector
})
	
local Values = {
	Always = Pack.Always(0),
	String = "Hello, World!",
	UnsignedShort = 65535,
	Boolean = true,
	Vector = Vector3.zero,
}
	
local Stream = Packer.Pack(Values)
	
Benchmark.Benchmark("Pack", 10_000, function()
	Packer.Pack(Values)
end)
	
Benchmark.Benchmark("Pack 100", 10_000, function()
	for _ = 1, 100 do
		Packer.Pack(Values)
	end
end)
	
Benchmark.Benchmark("Unpack", 10_000, function()
	Packer.Unpack(Stream, {})
end)

Benchmark.Benchmark("Unpack 100", 10_000, function()
	for _ = 1, 100 do
		Packer.Unpack(Stream, {})
	end
end)
1 Like
#9

Is it from Benchmarker module?
If not, can I have the module?

#10

It’s my own module, you can find the source code for it here
It has some stability issues around script exhaust times which is why you see task.wait and Rest