Small decimals problem

Help and Feedback Scripting Support
#1

For a section in my script, where it formats a kilogram input into a more readable output, when it takes a number lower than 1e-15, which is below a nanogram, it doesn’t go to any lower levels and just outputs: “0ng”

Source code:

KG = function(kilos)
			local log = math.log(kilos, 10)
			if log >= 3 then
			return rnd(kilos / 1e3, .1).."t"
		elseif log >= -0.005 then
			return rnd(kilos, .1).."kg"
		elseif log >= -3 then
			return rnd(kilos * 1e3, .1).."g"
		elseif log >= -6 then
			return rnd(kilos * 1e6, .1).."mg"
		elseif log >= -9 then
			return rnd(kilos * 1e9, .01).."µg"
		elseif log >= -12 then
			return rnd(kilos * 1e12, .01).."ng"
		elseif log >= -15 then
			return rnd(kilos * 1e15, .01).."pg"
		else
			return rnd(kilos * 1e18, .01).."fg"
		end
		end,

(the rnd() function is just this:)
image

Also another, slightly unrelated issue, how can I prevent long irrelivant decimals such as these:

image
image880×24 2.35 KB

1 Like
#2

There isn’t much you can do to stop that other than rounding the number to a certain decimal point.

1 Like
#3

There is a way, which is splitting the number (as a string) with “.” being the seperator, and then subbing the float part of the string. I would usually do that but im certain theres a better way

2 Likes
#5

string.format lets you specify number of digits.

n = 3.000000000002

print(n)
print(string.format("%.3f", n))
1 Like
#6

alr, but how can I make the units “reach” pg or fg, since I put a low number that should be 12pg, but i get 0ng as an output

#7

I ran your code, and I actually got “1pg” rather than “0ng” like you said.
I’m still looking to see if there is anything fishy.

EDIT: Ok, self correction! You said lower than 1e-15. One moment…
So I tested:

print(KG(1e-15))
print(KG(1e-16))
print(KG(1e-17))
print(KG(1e-18))
print(KG(1e-19))
print(KG(1e-20))
print(KG(1e-21))
print(KG(1e-22))
print(KG(1e-23))

and got:

1pg
100fg
10fg
1fg
0.1fg
0.01fg
0fg
0fg
0fg

So at a certain point, it just gave up and could not give an answer.
I did some breakpoint debugging to confirm what I am suspecting, but it’s clear to me that the issue is partialy due to working with very small numbers.

numbers in Lua are either 32 bit or 64 bit (I keep forgetting which). So this means that you have a limited amount of space to represent values.
So as you try to feed smaller numbers into you KG function, you are feeding in a value that is less and less precise.

So for example, when I did "print(KG(1e-21))"
It would get down to the line rnd(kilos * 1e18, .01).."fg"
At this point, you are multiplying a SUPER SMALL number by a SUPER BIG number. Both of those values probably have significant precision loss due to the amount of digits required to represent both numbers.
So this is what that number was when it was passed into “rnd”:
image

And once that gets divided by 0.01 and passed into the round function, the result is zero.

Though… Now that I look at this more, it looks like it’s not so much about precision (though you should keep that in mind). And it may be about the fact that femto grams is the last thing you are checking for. So smaller and smaller numbers are just being rounded down to zero, even if they shoudn’t.

Ok, so here is a modified version of your code, with a potential fix to demonstrate the issue.

function rnd(n,d)
	return math.round(n/d)*d
end

function KG(kilos)
	local log = math.log(kilos, 10)
	if log >= 3 then
		return rnd(kilos / 1e3, .1).."t"
	elseif log >= -0.005 then
		return rnd(kilos, .1).."kg"
	elseif log >= -3 then
		return rnd(kilos * 1e3, .1).."g"
	elseif log >= -6 then
		return rnd(kilos * 1e6, .1).."mg"
	elseif log >= -9 then
		return rnd(kilos * 1e9, .01).."µg"
	elseif log >= -12 then
		return rnd(kilos * 1e12, .01).."ng"
	elseif log >= -15 then
		return rnd(kilos * 1e15, .01).."pg"
	else
		return string.format("%.6f",kilos * 1e18).."fg"
	end
end

print(KG(1e-20))
print(KG(1e-21))
print(KG(1e-22))
print(KG(1e-23))

Because you didn’t continue your tree of if-statements further, it meant that you were making smaller and smaller numbers be used in the last “fg” case.
So when you rounded your number, you were rounding somthing so small that it became zero.
So I made the last “fg case” just do the multiplication and that’s it, and then used string.format to shorten the value. Because as we get smaller and smaller numbers, you lose more precision and may have issues due to that. So just keep that in mind, you won’t be able to have arbitrarily small numbers.

But with this code, I get:
0.010000fg
0.001000fg
0.000100fg
0.000010fg

#8

alr, btw the rest of the module is here: Visual unit conversion

I’ll try your fix and will see if it helps

(also for my use, its rare that objects would be lighter than 0.1 fg, so although that fix is nice, isn’t really necisary)

1 Like
#9

Ok cool.
One final remark, because I did want to mention this as well.
You also don’t necesarily need to use your custom “rnd” function.

As @azqjanna mentioned, you can use string.format to limit how many digits after the decimal you have, which is what rnd is also doing, but it probably does it in a different way with its own pros and cons.
(One of them being that the output is a string and not a number, but that’s what you want in the end anyway)

So for the other cases in your if statement tree, it would be:
string.format("%.2f",kilos * 1e15).."pg"

2 Likes
#10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.