If you’re only reading the data and not writing it at runtime, you may actually be even served better by using a string for this. That will give you 100% memory efficiency packing your data into memory, vs even with Vector3s I think you’ll still “only” get ~50% memory efficiency.
For instance, to extract the data with str:sub(index*4, index*4 + 3) and then use string.byte to turn the data back into numbers.
Isn’t this what string.unpack is for?
string.unpack("f",str,(index-1)*4+1) would be an example for 32 bit floats
It also seems weird to get a substring to just use string.byte on that substring, as string.byte accepts bounds to get several bytes at once.
Storing numbers as a string can take advantage of specific properties to reduce the space taken, like if the number is an integer and within a small range then it could be represented with a byte or two.
Indeed you can, my bad. I was going to suggest them at first but in my head they hadn’t been shipped yet, where they were actually shipped quite a while ago.
It’s not just that though. The table includes extra bookkeeping space alongside each value that isn’t being used for the value itself (to remember what type the value is), vs with a string you can spend 100% of the space on the values themselves.
On top of what you mention there, which is that you can use a smaller number of bytes for the value itself if your data can nicely be represented with something like a 16-bit integer.
Are you sure that is the case? I’ve tested this before, and I concluded that strings in lua reserve some extra space when created. Of course, packing all of the data into massive 200k char strings would be peak efficiency, but unfortunately for my application I need a nested table structure I can index (bvh structure), so I need individual strings that I can index.
When I ran the following code, which tests 30000 single char values representing 0-255, versus 1000 native Vector3’s containing 3 0-255 values, the Vector3s were always significantly better:
local n = 10000
do
local start = collectgarbage(“count”)
local s = {}
for i = 1, n * 3 do
s[i] = string.char(math.random(0, 255))
end
print(collectgarbage(“count”) - start)
end
wait(10)
do
local start = collectgarbage(“count”)
local s = {}
for i = 1, n do
s[i] = Vector3.new(math.random(0, 255), math.random(0, 255), math.random(0, 255))
end
print(collectgarbage(“count”) - start)
end
If the following code shows that strings only representing one byte are still more inefficient than Vector3’s which represent entire floats, then I doubt strings will become better at any reasonable use case for me. At most, I could probably pack about 13 floats into one string, but even then, I don’t have much reason to believe I’ll be making any gains there. Plus it costs me a little when reading back the data.
[Edit]: I did some extensive testing, and in the most extreme case where I squeeze out every byte that I can by lowering precisions here and there, I get a marginal 11% improvement which dwindles to about 4% on larger quantities for some reason. This is with about 16 numbers stored per table entry. Vector3s are hard to beat.
Oh, no, I don’t mean storing individual strings in an array… I mean having one big string which has all the array data and having no Lua table at all. So, s = "... really long ...", not s = {}. That will work if you never write new entries after startup, because reading entries can be acceptably cheap, but modifying that string will be very expensive. You’ll also have to be careful about how you construct it initially, using table.concat rather than repetitive uses of the .. operator to avoid O(n^2) behavior.
Because yes, there is some string overhead (more than some, it’s quite a few bytes of overhead), but if your data is one big string that overhead will be completely negligible.
Vector3Int16 and Vector2Int16 too
its not like as if anyone uses them but y’know might as well ask what would happen to them aswell
I use them tho, but only for storing numbers in 16-bit integers to be sent to the player.
You should use string.pack for that which is both faster and more memory efficient while not using a hacky method to achieve the same goal.
Well, I know virtually nothing about string.pack. I only know about string.char but that only works for 8-bit integers (unless you use two characters, multiply the first one by 256 and add them up to get 16-bit integers)
string.pack/unpack serializes/deserializes data according to a format specified by you into binary strings.
It is not documented on the wiki but here is an overview how to use it:
https://www.lua.org/manual/5.3/manual.html#6.4.2
For example:
local unsigned16bit = 450 --2 bytes
local float = 4.25 --4 bytes
local unsigned48bit = 2^48-1 --6 bytes
local format = "H f I6"
local binaryString = string.pack(format, unsigned16bit, float, unsigned48bit)
print(#binaryString) --will print 12
print(string.unpack(format,binaryString))
Another example let’s say we have a table consisting only of unsigned 16 bit integers:
local tbl16bit = {15,25,312,5555,9}
It can be packed into a binary string like this:
local binaryString = string.pack(string.rep("H",#tbl16bit),table.unpack(tbl16bit))
print(#binaryString) --this will print 10, 2 bytes per value
And to unpack the binary string into the original table:
local unpackedTbl16bit = table.pack(string.unpack(string.rep("H",#binaryString/2),binaryString))
print(unpackedTbl16bit)
This has been enabled in Studio as of June 10. We are planning to enable it on live clients and servers on June 14th.
This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.