IEEE 754 Compression (floating-point array Compression)

Resources Community Resources
,
#1

I created a compression module very easy to use. So basically it converts an Array of float numbers (decimal numbers) to a string of characters.

Thought I shared this with yall so if yall need floating point number compression here it is. I’ll make ALP compression in the future. (ALP: Adaptive Lossless floating-Point Compression)

Demo code

local IEEE754 = require(script.IEEE754)

local myLongDecimalArray = {}
for i = 1, 10 do -- put 10 random numbers in array
	myLongDecimalArray[i] = math.random() -- give random long number
end

local compressedString = IEEE754.compress(myLongDecimalArray)
print(compressedString) -- 3DAD36EA3F460ED13F083EF23F06217E3F4959C43EF51C583F472DE23EABDD553F450B8F3EEBE4B9
-- that whole string stores a long character of numbers

-- this is that same array but as a json
-- [0.0845773994558259,0.7736636010536396,0.5322105229007587,0.5239485665571584,0.7865260121008202,0.4787318785114881,0.7780438929507889,0.33567303171728737,0.7697076657440565,0.46072941264401809]
-- it is very long. and it got compressed into a string.

local myOtherSameLongDecimalArray = IEEE754.decompress(compressedString)
-- IEEE754 has some uh problems that are known, on decompression it has a 0.0000000001 number difference. It doesnt really matter even on big scale projects.
print(myOtherSameLongDecimalArray) -- its back hello array!

-- thats basically it

IEEE754 Download
IEEE754.lua (1.6 KB)

Full Module Code

local Compression = {}

local abs = math.abs
local frexp = math.frexp
local ldexp = math.ldexp

local bor = bit32.bor

local tableConcat = table.concat

local stringPack = string.pack
local stringUnpack = string.unpack

local char = string.char
local format = string.format
local byte = string.byte

local extract = bit32.extract
local insert = table.insert

function Compression.floatToBinary(num)
	local sign = num < 0 and 1 or 0
	num = abs(num)
	local mantissa, exponent = frexp(num)
	mantissa = (mantissa * 2 - 1) * ldexp(0.5, 24)
	exponent = exponent + 126
	local raw = bor(sign * 0x80000000, exponent * 0x800000, mantissa)
	return stringPack(">I4", raw)
end

local floatToBinary = Compression.floatToBinary

function Compression.compress(array: { number })
	local compressed = {}
	for arrayIndex = 1, #array do
		compressed[arrayIndex] = floatToBinary(array[arrayIndex])
	end
	
	local compressedString = tableConcat(compressed)
	local compressedASCII = compressedString:gsub(".", function(c) return format("%02X", byte(c)) end)
	return compressedASCII
end

function Compression.decompress(compressedASCII: string)
	local binary = compressedASCII:gsub("..", function(hex) return char(tonumber(hex, 16)) end)
	local decompressed = {}
	for i = 1, #binary, 4 do
		local raw = stringUnpack(">I4", binary:sub(i, i + 3))
		local sign = extract(raw, 31, 1)
		local exponent = extract(raw, 23, 8) - 126
		local mantissa = (extract(raw, 0, 23) + 0x800000) / 0x1000000
		local number = ldexp(mantissa, exponent) * (sign == 1 and -1 or 1)
		insert(decompressed, number)
	end
	
	return decompressed
end

return Compression

This is used on my Machine Learning module to compress very large networks 250% while keeping its parameters the same.

You can see the Machine Learning module here.

1 Like
ALP, ALP UTF-8 and IEEE 754 Compression (floating-point precision array Compression)