I’m trying to decode an 8 bit string into numbers so that it looks something like this
I’m wanting to normalize them also after as they’re unsigned after I would decode them.
I’ve done this in python, but I’m trying to do it in lua so I don’t have to deal with http overload issues like I’ve done in py here,
samples = np.frombuffer(bytes, dtype=np.uint8)
normalized_samples = (samples.astype(np.float32) - 128) / 127.0
This looks like simple hexadecimal encoding, using a slight modification of this stack overflow answer I was able to achieve similar looking results to what you asked for:
local format = string.format
local function encode(str)
return (str:gsub(".", function(char) return "\\x"..format("%02x", char:byte()) end))
end
Apologise for my lack of knowledge, I rarely work with this sort of stuff. Is hexadecimal the same as bytes? These bytes are returned from the frames of a wave file:
wav_file = wave.open('', 'rb')
sample_rate = wav_file.getframerate()
bytes = wav_file.readframes(wav_file.getnframes())
I also need to normalize them after which should return floats, how would I do this in conjunction with your codes?
Thanks.
Hexadecimal is another way of displaying byte data, which you can read up more on here..
As for converting the bytes to floats, it depends on the size (4 or 8 bytes)and endianness (big or small) of the floats.
Using this byte array to float function in conjunction with a hex to string reader, I was able to make up this:
local char = string.char
local insert = table.insert
function byte_to_float(bin)
local sig = bin:byte(2) % 0x80 * 0x10000 + bin:byte(3) * 0x100 + bin:byte(4)
local exp = bin:byte(1) % 0x80 * 2 + math.floor(bin:byte(2) / 0x80) - 0x7F
if exp == 0x7F then return 0 end
return math.ldexp(math.ldexp(sig, -23) + 1, exp) * (bin:byte(1) < 0x80 and 1 or -1)
end
local function decode(hex)
local str = hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits, 16)) end)
local floats = {}
for i=1,str:len(),4 do
insert(floats, byte_to_float(str:sub(i, i + 3)))
end
return floats
end
(FYI, this works with big-endian floats)
Thank you very much. Is bin the entire byte string or just per byte? If it is, how would I split the entire string into individual bytes? By normalize I meant this:
(sample - 128) / 127.0
Each byte should be normalized and to look something like this:
also note they do not have to be floats, the .0’s are just there so they can be floats and have fft performed on them
The decode function takes in the entire string. As for the byte_to_float function, it takes in chunks of 4 bytes in the form of a string and converts them to the corresponding big-endian float.
Also, any script output would help in the debugging/solving in this case.
Perhaps you mean converting the hex string back to byte data, as it would seem from the integer numbers each being less than 255. As for that it could be:
local char = string.char
local function decode(hex)
return {hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits, 16)) end):byte(1, -1)}
end
print(table.concat(decode("your hex string here"), "\n"))
oh sorry for no output, I forgot to put it in.
local RunService = game:GetService("RunService")
local HttpService = game:GetService("HttpService")
local byteString = HttpService:GetAsync('')
local char = string.char
local insert = table.insert
function byte_to_float(bin)
local sig = bin:byte(2) % 0x80 * 0x10000 + bin:byte(3) * 0x100 + bin:byte(4)
local exp = bin:byte(1) % 0x80 * 2 + math.floor(bin:byte(2) / 0x80) - 0x7F
if exp == 0x7F then return 0 end
return math.ldexp(math.ldexp(sig, -23) + 1, exp) * (bin:byte(1) < 0x80 and 1 or -1)
end
print(byte_to_float(byteString:sub(1, 100)))
I only wanted 100 characters of the result since the actual string is way larger than that.
anyway i got this
771817333989547400000
Using the aforementioned decode function, it should return 128 (100 times for the first 100 bytes), or if you want them to be in the float format the above,above function returns -1.1801040622505e-38 (6 times for the first 96 bytes).
Okay so I did this but I’m getting an error
local RunService = game:GetService("RunService")
local HttpService = game:GetService("HttpService")
local byteString = HttpService:GetAsync('')
local char = string.char
local insert = table.insert
function byte_to_float(bin)
local sig = bin:byte(2) % 0x80 * 0x10000 + bin:byte(3) * 0x100 + bin:byte(4)
local exp = bin:byte(1) % 0x80 * 2 + math.floor(bin:byte(2) / 0x80) - 0x7F
if exp == 0x7F then return 0 end
return math.ldexp(math.ldexp(sig, -23) + 1, exp) * (bin:byte(1) < 0x80 and 1 or -1)
end
local function decode(hex)
local str = hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits, 16)) end)
local floats = {}
for i=1,str:len(),4 do
insert(floats, byte_to_float(str:sub(i, i + 3)))
end
return floats
end
print(decode(byteString)[1])
output:
invalid argument #1 to 'char' (number expected, got nil) <-- local str = hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits, 16)) end)
local str = hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits, 16)) end)
Should be
local str = hex:gsub("\\x%x%x", function(digits) return char(tonumber(digits:sub(3), 16)) end)
Which should remove the hex indicator from the start of each match.
Also, it sounds like your best bet would be sending the raw byte data to a file online, then requesting it in Roblox, instead of doing any hexadecimal conversions, as that introduces overhead and unnecessary storage usage.
That fixed that part, but there was also an issue of the first line in your byte to float function
attempt to perform arithmetic (add) on number and
In regards to the first line, I believe it’d be better to write to the file using the wb (write binary) tag instead of the w (write text) one as you are working with byte data. This’d allow you to skip over the converting process later on.
(Sixteenth line would become: with open('scripts/wav/wav_bytes.txt', 'wb') as f:)
Actually that is a good point, but how would i get the normalized values from here in roblox?
Not talking about the question marks I believe that is a rendering issue but with the other characters
Simply use the string.byte function,
local str = "some string data here"
local bytes = {str:byte(1, -1)}
print(table.concat(bytes, "\n"))
Nevermind about the questionmarks, they actually do appear as they are
Oh this works. I will mark this as the solution. Thank you very much for the effort.
stack overflow (string slice too long)
here I would just split the string up then right?
Yea, probably into chunks of 1024 would be a reasonable size.
How would I do that in roblox considering the entire thing is just full of random characters
local rep = string.rep
local insert = table.insert
local function split(str, csize)
local out = {}
local pattern = "."..rep(".?", csize - 1)
for m in str:gmatch(pattern) do
insert(out, m)
end
return out
end
Where str is the input string, and csize is the size of the chunks (1024 for the example number), and it outputs a table of strings with that length.