Hello. I have scavenged all around for a way to fix floating point errors. I have found one super helpful post, but it does not account for numbers being limited in precision negatively. Here’s an example.
3.000002 → 3
2.994584 → 2.994 (Expected: 3)
math.round does help out to an extent as well, but it does not seem to be as useful. Does anybody know what could help out with this other scenario?
Here’s an example of my problem.
In the bad case is the expected value also 3?
Yes. I apologize for not pointing that out initially.
What function are you currently using to get this output?
The function is shown in answer 4 of the attached post. Here it is directly though.
local function FormatChance(chance: number): string
local formatted = string.format("%.3f", chance) --> omit to three decimals
formatted = formatted:gsub("%.?0+$", "") --> remove the dot and trailing zeroes
return formatted
end
Just something to keep in mind, the return value does not have to be a number. I plan on only using this function for displaying numbers to the user.
Is your goal to have the output as an integer when its bigger than 1 and to look clean when its below 1 (Ex. 0.005)?
I am trying to make sure precision errors do not happen while still allowing decimals up to three places away can show.
Here are some examples.
1.00000001 → 1
1.1234567890 → 1.123
1.099988798 → 1.01
The only thing I can think of is to check if the rounded number is very close to a whole number and if so return that else return the formatted string.
Below is the code from the mentioned post but edited to do the above:
local function FormatChance(chance: number): string
local nearest = math.round(number)
local formatted
if math.abs(nearest - chance) > 0.01 then
formatted = string.format("%.3f", chance)
formatted = formatted:gsub("%.?0+$", "")
else
formatted = nearest
end
return formatted
end
You can play around with the threshold.
Thank you very much for your suggestion. The only problem is that the floating point issue can happen on decimals, and the nearest variable would have to support that too.
Try this:
local function FormatChance(chance: number): string
local formatted
local nearest = math.round(chance)
if math.abs(nearest - chance) > 0.01 or chance < 1 then
formatted = string.format("%.3f", chance)
formatted = formatted:gsub("%.?0+$", "")
else
formatted = nearest
end
return formatted
end
It seems like the code sample contains the same method to improve the floating point precision error as the original post. The method involving the formatting and gsubbing doesn’t solve the issue for rounding up numbers like this:
1.99995858234 → 2
0.25999999999 → 0.26
I think there is a function on the math library called math.round, why not use it?
That method rounds to the nearest whole number and not to the nearest decimals. Take a look at some of the posts I wrote above.
I know math.round(x * grid) / grid (or something along that nature) works, but it is also a target of precision errors. I’m looking for a solution that fixes a decimal number so it can be displayed to the user.