Do you perhaps have a more accurate release date for this? I am planning to implement this into my game and would plan to have it out with this feature by end of January / mid February
BINARY DATASTORES!
The next step is being able to serialize Instances into and from an RBXM buffer, so we can save Instances in game to datastores, that would be amazing, or if thats too low-level, just the ability to directly save them (and have it silently convert under the hood).
Would bit-level access be possible at some point in the future?
1 Like
Do you have a use case for a value when the attribute option is being discussed? I’ve moved away from values in favor of attributes because require less code to set values (4 for object creation, or more with checking for existing instances vs SetAttribute) and getting values (1-3 lines with nil check vs GetAttribute).
1 Like
I thought i could store voxel data in it but i forgot that attributes existed. Althought it would be cool if this worked differently to attributes. Maybe it could store compressed data the same way that it compresses data before sending it to client/server.
For a quaternion you probably want to normalize it, then you can get the maximum amount of precision by encoding the components as integers. Using a float16 would waste a lot of potential precision.
Thanks for this info @tnavarts, this wasn’t something I had considered
For anyone else wanting to do the same:
P.S. Please do note that I’m not completely certain this is correct, but based on what Stravant suggested I’m somewhat certain the method would look something like the below
local FP_EPSILON = 1e-6
local I16_PRECISION = 32767 -- int16 range { -32,786, 32,767 }
local BUFF_CFRAME_SIZE = (3*4) + (1 + 3*2) -- i.e. 3x f32, 1x u8 and 3x i16
local function getNormalisedQuaternion(cframe)
local axis, angle = cframe:ToAxisAngle()
axis = axis.Magnitude > FP_EPSILON and axis.Unit or Vector3.xAxis
local ha = angle / 2
local sha = math.sin(ha)
local x = sha*axis.X
local y = sha*axis.Y
local z = sha*axis.Z
local w = math.cos(ha)
local length = math.sqrt(x*x + y*y + z*z + w*w)
if length < FP_EPSILON then
return 0, 0, 0, 1
end
return x / length,
y / length,
z / length,
w / length
end
local function compressQuaternion(cframe)
local qx, qy, qz, qw = getNormalisedQuaternion(cframe)
local index = -1
local value = -math.huge
local sign
for i = 1, 4, 1 do
local val = select(i, qx, qy, qz, qw)
local abs = math.abs(val)
if abs > value then
index = i
value = abs
sign = val
end
end
sign = sign >= 0 and 1 or -1
local v0, v1, v2
if index == 1 then
v0 = math.floor(qy * sign * I16_PRECISION + 0.5)
v1 = math.floor(qz * sign * I16_PRECISION + 0.5)
v2 = math.floor(qw * sign * I16_PRECISION + 0.5)
elseif index == 2 then
v0 = math.floor(qx * sign * I16_PRECISION + 0.5)
v1 = math.floor(qz * sign * I16_PRECISION + 0.5)
v2 = math.floor(qw * sign * I16_PRECISION + 0.5)
elseif index == 3 then
v0 = math.floor(qx * sign * I16_PRECISION + 0.5)
v1 = math.floor(qy * sign * I16_PRECISION + 0.5)
v2 = math.floor(qw * sign * I16_PRECISION + 0.5)
elseif index == 4 then
v0 = math.floor(qx * sign * I16_PRECISION + 0.5)
v1 = math.floor(qy * sign * I16_PRECISION + 0.5)
v2 = math.floor(qz * sign * I16_PRECISION + 0.5)
end
return index, v0, v1, v2
end
local function decompressQuaternion(index, v0, v1, v2)
v0 /= I16_PRECISION
v1 /= I16_PRECISION
v2 /= I16_PRECISION
local d = math.sqrt(1 - (v0*v0 + v1*v1 + v2*v2))
if index == 1 then
return d, v0, v1, v2
elseif index == 2 then
return v0, d, v1, v2
elseif index == 3 then
return v0, v1, d, v2
end
return v0, v1, v2, d
end
local function write(buf, offset, input)
buffer.writef32(buf, offset + 0, input.X)
buffer.writef32(buf, offset + 4, input.Y)
buffer.writef32(buf, offset + 8, input.Z)
local qi, q0, q1, q2 = compressQuaternion(input)
buffer.writeu8(buf, offset + 12, qi)
buffer.writei16(buf, offset + 13, q0)
buffer.writei16(buf, offset + 15, q1)
buffer.writei16(buf, offset + 17, q2)
return BUFF_CFRAME_SIZE
end
local function read(buf, offset)
local x = buffer.readf32(buf, offset + 0)
local y = buffer.readf32(buf, offset + 4)
local z = buffer.readf32(buf, offset + 8)
local qi = buffer.readu8(buf, offset + 12)
local q0 = buffer.readi16(buf, offset + 13)
local q1 = buffer.readi16(buf, offset + 15)
local q2 = buffer.readi16(buf, offset + 17)
local qx, qy, qz, qw = decompressQuaternion(qi, q0, q1, q2)
return CFrame.new(x, y, z, qx, qy, qz, qw),
BUFF_CFRAME_SIZE
end
return {
readCFrame = read,
writeCFrame = write,
}
4 Likes
Yippeee! Data compression les go!
Have they been tested and debugged enough for them to be safe and reliable to use for replicating data?
What’s the current compression ratio for big buffers?
Sorry if this has been asked before but how efficient is this “on the wire” compression of buffers and how does it work?
If I write a 32-bit number to a buffer that looks like 00000000 00000000 00001010 00000111 does it like recognize the zeroes and compress them?
1 Like
I could think of a few ways to do this with buffers actually.
I would probably manually serialize instances into buffers instead since it gives more control over what you want to store.
Storing every single property of an instance would likely be… perhaps a bit impractical.
I don’t think you want to store what the name of an instance was (1 byte per character) or all of it’s physical properties.
That is, if you want to keep the data small and as efficiently compressed as possible.
RBXM already compresses quite well and minus its implementation quirk hell, its a decent format for what it needs to do
Ports of this aren’t practical in Lua because there’s a lot of stuff in the file that it cant access without Roblox throwing security errors, or there being no feasible API to utilise (meshparts)
1 Like
Returning to here. Would Buffers be supported to write and read pixels in EditableImages or EditableMesh (IF THATS EVEN POSSIBLE)? Unless there’s an issue somewhere regarding it, I’m pretty sure you can use it to get better performance when writing/reading pixels to the buffer and passing/getting it to/from a method. The only issue is this might not be familiar to those who have never used a buffer.
1 Like
It’s been mentioned here by Stravant, but you can actually pack the data even tighter than that.
Right now, you’re selecting the largest of the 4 numbers in the quaternion, dropping it and reconstructing it using the other three since it’s normalized, which is great. But that means that since we already know the largest of the squared numbers has been subtracted, we can actually also know for sure that none of the remaining numbers will have a square larger than 1/2 (since if it was greater than 1/2, then it would be the largest number, not the number we already removed). Hence why we can actually assume all values to be between -1/sqrt(2) and 1/sqrt(2). (instead of the -1 to 1 range you currently use)
The other optimization you could make (which is probably not necessary unless you really needed to squeeze out every bit or encode a lot of data) would be to encode on the bit-level, not byte level. This way, you could encode your index in 2 bits, not 8. And, you could have more freedom to do odd bit counts like 15 for the 3 numbers, instead of sticking to multiples of 8.
If anyone’s reading this and wants to do bit-level packing, one way you could do it is to write to a table of booleans (representing your bits), and then once you’ve “encoded” all of your data, you would go through the table in 8-sized chunks and encode/decode with read/writeu8 (unsigned 8 bit). This also has the benefit that if you know the layout of your data (i.e serializing an object with known parameters) on the sending and receiving end, you can stuff a bunch of dissimilar data types back to back in one giant “bitstring” without having to waste space on padding.
1 Like
How likely would it be for EditableImages to support buffers? I’d appreciate if you guys would add such functionality.
Buffers would only need to store 1 byte for each component, giving really large memory savings, specially for transferring large image buffers over the internet.
Also, considering the new “–!native” feature, I’m sure EditableImages would take advantage of buffers even more, as the code is likely closer to C code.
2 Likes
Are those 25 extra bytes really that big of a deal though? Are there any statistics or data on how many extra bytes will actually start to impact performance?
If it’s only a little bit of data it’s not a big deal really but if you have a building game where you have to save every wall, floor, door, window, etc then size starts to matter A LOT.
Smaller data is also much faster to save and load from datastores and uses less bandwidth when send over a network.
You’re not expected to always keep data as small as possible but it’s a good practice to learn efficient data storage and structuring.
Keeping things small and compact has many benefits.
3 Likes
Can you please check again?
We’ve made improvements to the amount of extra data that was sent in some cases.
1 Like
Small update!
Luau buffer type support is now enabled in DataStore, MemoryStoreService, MessagingService, TeleportService (TeleportData) and HttpService (JSONEncode/JSONDecode functions).
Documentation updates for those services will follow soon.
13 Likes
DataStore buffer support? As in, we can write binary data to a DataStore now without issues?
1 Like
Are there plans for MemoryStoreService to support buffers for its data structures?
No, the new feature is that you can have ‘buffer’ objects in the value field of SetAsync (directly or inside a table).
And you will get ‘buffer’ objects back from GetAsync etc.
You still can’t have non-utf8 strings as values.
Yes, it is also supported in MemoryStoreService (MemoryStoreHashMap).
I forgot to put it in the list.