Hello. I have scavenged all around for a way to fix floating point errors. I have found one super helpful post, but it does not account for numbers being limited in precision negatively. Here’s an example.

3.000002 → 3
2.994584 → 2.994 (Expected: 3)

math.round does help out to an extent as well, but it does not seem to be as useful. Does anybody know what could help out with this other scenario?

The function is shown in answer 4 of the attached post. Here it is directly though.

local function FormatChance(chance: number): string
local formatted = string.format("%.3f", chance) --> omit to three decimals
formatted = formatted:gsub("%.?0+$", "") --> remove the dot and trailing zeroes
return formatted
end

Just something to keep in mind, the return value does not have to be a number. I plan on only using this function for displaying numbers to the user.

The only thing I can think of is to check if the rounded number is very close to a whole number and if so return that else return the formatted string.

Below is the code from the mentioned post but edited to do the above:

local function FormatChance(chance: number): string
local nearest = math.round(number)
local formatted
if math.abs(nearest - chance) > 0.01 then
formatted = string.format("%.3f", chance)
formatted = formatted:gsub("%.?0+$", "")
else
formatted = nearest
end
return formatted
end

Thank you very much for your suggestion. The only problem is that the floating point issue can happen on decimals, and the nearest variable would have to support that too.

local function FormatChance(chance: number): string
local formatted
local nearest = math.round(chance)
if math.abs(nearest - chance) > 0.01 or chance < 1 then
formatted = string.format("%.3f", chance)
formatted = formatted:gsub("%.?0+$", "")
else
formatted = nearest
end
return formatted
end

It seems like the code sample contains the same method to improve the floating point precision error as the original post. The method involving the formatting and gsubbing doesn’t solve the issue for rounding up numbers like this:

That method rounds to the nearest whole number and not to the nearest decimals. Take a look at some of the posts I wrote above.

I know math.round(x * grid) / grid (or something along that nature) works, but it is also a target of precision errors. I’m looking for a solution that fixes a decimal number so it can be displayed to the user.