Enforce ordering with HttpService:JSONEncode()

Feature Requests Engine Features
#1

As a Roblox developer, it is currently too hard to enforce an order in the returned value of HttpService:JSONEncode(). Consider this example:

I have two nesting dictionaries, originating from different places, but who are deeply equal to one another. Putting both tables through JSONEncode() spits out these two values:

Dictionary 1:
{"Assets":[{"Id":27112025,"Name":"Roblox 2.0 Torso","AssetType":{"Id":27,"Name":"Torso"}} …

Dictionary 2:
{"BodyColors":{"HeadColor":"0.80,0.56,0.41","TorsoColor":"0.80,0.56,0.41","LeftArmColor":"0.80,0.56,0.41","RightLegColor":"0.80,0.56,0.41" …

We can already see that there is conflicting ordering in the two dictionaries. This behavior propagates further into any nested dictionaries, making this problem recursively more difficult to solve. If the dictionaries are equals (the data in these dictionaries contains only primitive data), then they should always return the same encoded string.

The idea here is to leverage the encoded unique JSON to then create an MD5 hash that describes an outfit created by a player in our game. As far as I know, Roblox does this on the website by creating MD5 hashes that describe the Players’ outfits for thumbnails for a number of reasons, one of them being storage efficiency.

If Roblox is able to address this issue, it would improve my development experience, because I wouldn’t have to cut up, parse, and reconstruct the nested JSON alphabetically.

I’m not suggesting that JSONEncode enforce alphabetical ordering necessarily, but some expectable pattern would be very valuable.

1 Like
#2

I think there’s a couple issues here:

  1. Object keys are specified as unordered in the JSON spec. So there’s no enforcement on the JSON side to keep keys in any order.
  2. Similarly, Lua tables don’t have an enforced ordering of keys (except for the array portion of them).

So relying on key ordering is not something that should be done.

Typically, using an MD5 hash is a common practice to ensure data integrity. So you might hash your data, and then compare that hash on the other end of the wire with the incoming data. This happens before the JSON would be parsed again (and usually isn’t even JSON, but something like a base64 string).

If you’re trying to use a hash to describe uniqueness of your data, I might recommend using the newer buffers instead, which will allow you to control ordering of your data. Then take the hash of the outputted string value of the buffer.

local buf = buffer.create(2)
buffer.writeu8(buf, 0, 10)
buffer.writeu8(buf, 1, 32)

local hash = yourMd5Func(buffer.tostring(buf))
2 Likes
#3

Hey @sleitnick,

Thanks for your response. While yes, JSON’s spec is unordered, I would personally be surprised if Roblox actually follows JSON spec (being that JSONEncode is so old). Secondly, Luau tables’ ordering is mostly irrelevant in this manner, since JSONEncode’s current ordering is actually a side-product of how it does it’s iteration internally. The iteration part is really what I’m looking to have changed, I think.

If we look at generalized iteration in tables in Luau right now:
image

Regardless of order of initiation, generalized iteration works, first, numerically, then alphabetically, in Luau. The output of JSONEncode suggests to me that it doesn’t use generalized iteration. If there’s a specific reason why that an engineer could explain to me, maybe back-compat, internal standard, or if it’s actually load-bearing, then so be my fate!

side-note; I would like to use buffers for this, too, actually. But this is data going into a DataStore. So until buffers find their way into DataStores, that is unfortunately not a path I can take at the moment.

EDIT:
I unpacked my dictionary into a deeper nested set of { key, value } arrays, and then put that into a buffer and the hash reads equal now. I appreciate the help! Though, I would still prefer the suggestions I’ve made for various reasons, namely performance, standard, et cetera.

#4

You can actually send buffers to DataStores now. See this post!

1 Like