Trying to make string.pack work with dataStores

[Famous5000]

1. What do you want to achieve? I’m trying to make string.pack work well with the data stores for the save system I’m working on for a different project.

2. What is the issue? When using string.pack, if the character goes above 127, the JSON encoding fails due to, what I believe, is UTF-8 not supporting those characters. This wouldn’t be a problem if the values I’m trying to store is at max 256, but it’s much larger.

3. What solutions have you tried so far? I tried encoding the strings using the JSONEncode from HTTPService but when I tried that for things that went over 127 when using string.pack("B",val), the encoding failed.

I hope I don’t need to provide too much code from the script; I can provide output tables if necessary though.

What I’m asking is this: Does anyone have a good replacement to string.pack and string.unpack that is both compact and allows variable layouts? Or does anyone have a good save encoder that they can recommend to me? Any help would be greatly appreciated.

[spearmint98]

Can you make a table of smaller substrings and send it through like that? It may be that your string is too many bytes.

[Harishxx (Windy)]

Have you tried using any of the other string.pack formats?

[Tylerisawsome113 (Tyler)]

This other post may help you answer your string.pack question:

What is string.pack() and string.packsize()?

Datastores use JSON, so using a binary format doesn’t work with them. (must be a valid utf8 string, and control characters except 7F as well as \ and " get expanded to multiple bytes) It would be useful for something like passing large quantities of data across the network.

As for already made encoders, I am not entirely sure, but you could always make your own.

[Famous5000]

This does not have anything that may be helpful; as they all basically use the same packing method.

[Famous5000]

Try HTTPService:JSONEncode(string.pack("B",200)), you’ll find that it fails. And not only that but the save service also does not like any characters that aren’t alphanumeric.

[Famous5000]

A coding friend of mine did provide a library that seems to contain what would be recommended for reformatting the string.pack that was used. Hopefully I will be able to use the program soon to re-write the string that string.pack uses, if I have any good news from that I’ll provide the solution in case anyone else has the same problem.

[Famous5000]

a few days later and I have OK news

So I made my own character encoder/decoder which works to make the string fully saveable with only double the overhead in comparison to string.pack. I’m assuming you all have heard of hexadecimal. The converter basically converts every character into 2 hexadecimal numbers. For example: string.char(255) becomes FF.

If you want the code for this feel free to ask but otherwise the issue is resolved.