Im subtracting an interger from a decimal, (1.05-1 for example), and I get a strange number like this
I also need exact numbers in my scenario, is there any way to work around this? Or possibly fix?
P.S. It also says they’re inequivalent
I also need exact numbers in my scenario, is there any way to work around this? Or possibly fix?
That’s just floating-point rounding precision error or something like that
Basically, numbers can only get so precise because there’s a limit on how much bits are used to represent them
You can read more about it here: language agnostic - Why are floating point numbers inaccurate? - Stack Overflow
There is no direct fix for it, the best that you can do to mitigate it is to round your number by some precise factor
local function preciseRound(n, i)
return math.floor(n * i + .5) / i
end
--round to the nearest 0.0001
print(preciseRound(1.05 - 1, 1e4) == .05) --> true
4 Likes
Fixed! If anyone was wondering, I converted it into a string, split it so I get an integer and decimal, and did any arithmetic on the integer, then I just put the number back together
1 Like
Just so you’re aware the math library has a function named “modf” which splits a floating point number into its integer & decimal components.
local number = 1.05
local integer, decimal = math.modf(number)
local number = 1.05
local splitNumber = tostring(1.05):split(".")
local integer = tonumber(splitNumber[1])
local decimal = tonumber(splitNumber[2])
As you can see the first version is much cleaner.