I’ve come back to this because I was curious if this limit will ever change.
I’d really love to be able to use buffers for our data serialization, as we have a very complex structure & item system in our game, which uses a ton of data. I know there’s ways we could make our data more efficient with strings as is, but I would love to be able to knock it all out with buffers. The 3MB limit is the only thing holding me back, as it’s not worthwhile at that point.
I’m not aware of any plans to increase the DataStore value limit.
Is there plans to make the buffer type available for the SharedTable type?
+1 on this, shared memory buffers would be very useful, even if we just get simple, not thread safe, read and write
Is there any plans on a terrain :WriteVoxelChannelBuffer() function? Could be very useful.
Are there any plans on implementing some sort of array support like:
local bfr = buffer.create(4)
buffer.writei8(0,{1,5,8,4})
where each entry of the array is stored in different offsets automatically:
buffer.readi8(bfr,2) – returns 5
buffer.readi8(bfr,3) – returns 8
Idk this might be none sensical but this reminds me of storing things with memory addresses in c where we can directly input items in the later indexes past the initial one.
hey WheretlB really appreciate all the work you’re doing here. it is awesome to see these improvements to the buffer feature.
i totally agree that adding more flexibility with bit widths like using writeuint would be really useful. it would give developers more control over serialized data which is a big help for handling complex systems with lots of data.
that said, i can see the concern with performance when it comes to things like bounds checking. developers who need to optimize for speed might still prefer manual bit packing with libraries like bit32. but having a more native option would definitely make things easier for those who don’t need to go that deep into performance.
it would also be cool to see expanded buffer features for specific use cases. things like sharedtable support or a writevoxelchannelbuffer for terrain data would make working with large datasets or complex terrain systems way smoother.
really excited to see where this goes. adding more options for buffers would definitely make workflows more efficient and help developers build better games.
Is there a better and faster way to do this without using a big for loop?
for i = 0, Width * Height * 4 - 1, 4 do
buffer.writeu8(PixelBuffer, i, R)
buffer.writeu8(PixelBuffer, i + 1, G)
buffer.writeu8(PixelBuffer, i + 2, B)
buffer.writeu8(PixelBuffer, i + 3, A)
end
this is very slow at 1024x1024.
it would be very useful if buffer.fill or something similar to be able to input multiple values instead of just one.
That’s exactly what I’ve been investigating. I don’t know how to optimize this:
It is an attempt (with the intention of improvement) of a normal map reader for the interface, and the only way for it not to consume much is by rescaling my images to 120x120, from that number onwards it causes lag, because in more exaggerated cases such as 1000x1000 would be 1,000,000 million turns in a loop 
500x500
bump for this, could be very useful in my project.
I discovered that you can get quite a boost in performance if you do:
local ColourBuffer = buffer.create(4)
buffer.writeu8(ColourBuffer, 0, R)
buffer.writeu8(ColourBuffer, 1, G)
buffer.writeu8(ColourBuffer, 2, B)
buffer.writeu8(ColourBuffer, 3, A)
local ColourU32 = buffer.readu32(ColourBuffer, 0) -- Convert to single number value
for i = 0, Width * Height * 4 - 1, 4 do
-- Now we've gone from 4 to 1 buffer calls!
buffer.writeu32(PixelBuffer, i, ColourU32)
end
This effectively reduces the amount of buffer calls in the iteration from 4 to 1!
Spectacular, let me try this change. thank you so much; The other day I made some improvements that reduced the speed by 0.17sec on average. Let’s see how this goes for us. Thank you 
HAHA, it lasted 2 hours and I didn’t know how to apply it. In fact, I think my code didn’t use the multiplier of 4, so thank you, either I’m bad at this or I already had it without realizing it (it’s very laggy).
local function GetColor(Buffer:buffer,Offset:number,ColorID:number)
local Color = ColorMaps[ColorID][Offset]
if Color then
return Color[1],Color[2],Color[3],Color[4]
end
local r = buffer.readu8(Buffer, Offset) / 255
local g = buffer.readu8(Buffer, Offset + 1) / 255
local b = buffer.readu8(Buffer, Offset + 2) / 255
local a = buffer.readu8(Buffer, Offset + 3)
if a == 0 then
a = false
end
ColorMaps[ColorID][Offset] = {r,g,b,a}
return r,g,b,a
end
local function GetNormal(NormalMap:buffer,offset:number, DephthEffect:number,NormalID:number)
local getNormal = NormalMaps[NormalID][offset]
if getNormal then
return (getNormal*DephthEffect).Unit
end
local r = (buffer.readu8(NormalMap, offset) / 255 * 2 - 1)
local g = (buffer.readu8(NormalMap, offset + 1) / 255 * 2 - 1)
local b = (buffer.readu8(NormalMap, offset + 2) / 255 * 2 - 1)
local NewNormal = Vector3.new(r , g , b )
NormalMaps[NormalID][offset] = NewNormal
return (NewNormal*DephthEffect).Unit
end
local function CalculateLighting(normal: Vector3, lightDir: Vector2, brightness: number, distance: number, range: number, depthEffect: number)
local dotProduct = math.max(normal:Dot(lightDir), 0)
local attenuation = math.clamp(1 - (distance / range), 0, 1)
local depthFactor = math.clamp(attenuation ^ depthEffect, 0, 1)
local adjustedLighting = math.clamp(dotProduct * depthEffect, 0, 1)
return adjustedLighting * brightness * depthFactor
end
function UPDATETEXTURE(ID)
local DepthEffect = Depths[ID]
local Image = EditablesImages[ID]
local Buffer, BufferNormal = BufferColors[ID], BufferNomals[ID]
if not Image then
warn("Error, ID is not valid")
return false
end
local DataLights = Lights[ID]
local ImageSize = Image.Size
local NormalDir = NormalDirections[ID]
for y = 0, ImageSize.Y - 1 do
for x = 0, ImageSize.X - 1 do
local offset = (y * ImageSize.X + x) * 4
local normal = GetNormal(BufferNormal, offset, DepthEffect, ID)
local r, g, b, a = GetColor(Buffer, offset, ID)
if a then
local pixelPos = Vector2.new(x / ImageSize.X, y / ImageSize.Y)
local totalLightIntensity = 0
for _, light in pairs(DataLights) do
local lightPos = light.Position or Vector2.zero
local brightness = light.Brightness or 1
local range = light.Range or 1
local distance = (pixelPos - lightPos).Magnitude
local lightDir = Vector3.new((lightPos.X- pixelPos.X)*NormalDir.X, (lightPos.Y - pixelPos.Y)*NormalDir.Y,1).Unit
local lightIntensity = CalculateLighting(normal, lightDir, brightness, distance, range, DepthEffect)
totalLightIntensity += lightIntensity
end
local depthDarkness = math.clamp((1 - normal.Z) * DepthEffect, 0, 1)
r = math.clamp((r * (1 - depthDarkness)) + totalLightIntensity, 0, 1)
g = math.clamp((g * (1 - depthDarkness)) + totalLightIntensity, 0, 1)
b = math.clamp((b * (1 - depthDarkness)) + totalLightIntensity, 0, 1)
buffer.writeu8(Buffer, offset, r * 255)
buffer.writeu8(Buffer, offset + 1, g * 255)
buffer.writeu8(Buffer, offset + 2, b * 255)
buffer.writeu8(Buffer, offset + 3, a)
end
end
end
Image:WritePixelsBuffer(Vector2.zero, ImageSize, Buffer)
return true
end
If you are free and consider helping me, I would appreciate it (I don’t demand it, okay).
In case you are interested in reading it more in depth:
LightEffect.lua (7.4 KB)
given the release or read/write bits to the luau language recently, will there be an option to have the offset in bits for buffers?
example would be
buffer.writeu8(buffer: buffer, offset: number, value: number, useBitOffset: boolean)
You can just calculate the ColourU32 manually if you have the RGBA separately by doing this:
ColourU32 = R + G * 2^8 + B * 2^16 + A * 2^24
We have explored this idea a bit, but I have to disappoint that we do not have current plans to extend existing methods in this way.
We are planning to enable buffer.readbits and buffer.writebits functions in the nearest future (once all clients update to support it).
But note that those functions are added for convenience, but bit32 bit-packing before a single buffer.write* will remain more performant.
Terrain team keeps in mind the possibility to use buffers for more performant use cases, but they are busy with an overall redesign so can’t share any timelines.
We might add the ability to pass a buffer in and out, but the way SharedTable operates will not allow different threads to modify different/overlapping parts of the buffer, it will be a full buffer data replacement. This limit might make buffer placement in SharedTables not as interesting to developers.
No, we are not going to provide bit offsets outside of buffer.readbits and buffer.writebits functions. Bit offsets go against how CPUs are designed to operate and have to be emulated using byte operations with extra overhead.
What is the maximum size a buffer should be when saving to the data store?
I found that 4*2^20 * 3/4 - 27 = 3145701 is the maximum size, but this could be wrong.
[note] Roblox sometimes optimizes data when doing JSONEncode , so that limit is surpassed depending on the data.
EDIT: What I really want to know here:
Is the buffer length or the JSONEncode of the buffer counted against the data size limit?
The result of JSONEncode on the buffer is used. Roblox uses zstd (sometimes) and base64 (always) to represent the data as json. The json is then counted against the data size limit.
without zstd:
{"m":null,"t":"buffer","base64":"..."}
with zstd (zstd compressed and then base64 encoded):
{"m":null,"t":"buffer","zbase64":"..."}
By the way, you can abuse the same example in JSONDecode to decode zstd compressed data.