This equation is getting the wrong answers from Roblox Lua, why?

It can be solved if you convert the number to a string and back to a number:

print(tonumber(tostring(-200 * 2.55)) + (255 * 2)) -- 0

Don’t know why this is the case but it works.

2 Likes

How strange, it seems to be incredibly close to zero though. math.floor would fix it, but it’s still strange.

1 Like

The reason for this is because “2.55” can`t actually be written in binary with a finite amount of digits so it has to be rounded resulting in a rounding error, it’s like trying to write 1/7 as a decimal number and trying to do calculations with that.
An example for this in decimal would be expecting this to be true:
0.111111111 * 9 = 1

You (should) get the same exact rounding error with every single programing language that exists when using double-precision floating-point numbers as it is a problem that is inherent to the floating point system itself rather than the programing languages that make use of it.

4 Likes

This is exactly what I’m looking for, I knew I was missing something really basic. And this isn’t even the first time I accidentally computed a value outside of binary bounds, we live and we learn, thank you thank you :grinning_face_with_smiling_eyes:

This is just what is known as a precision error, binary which is the 2-base number system is incapable of representing 2.55.

1 Like

You mean math.floor or math.ceil

no I mean math.round
try it yourself it works