The obvious approach would be to check if the absolute value of the number is between 0 and 999, then 1000 and 99999, and so on. However you can determine what power of a thousand the number is via logarithms.
Here’s an example of the latter:
local SuffixList = { "", "K", "M", "B", "T", "Q" }
local function Format(value, idp)
local exp = math.floor(math.log(math.max(1, math.abs(value)), 1000))
local suffix = SuffixList[1 + exp] or ("e+" .. exp)
local norm = math.floor(value * ((10 ^ idp) / (1000 ^ exp))) / (10 ^ idp)
return ("%." .. idp .. "f%s"):format(norm, suffix)
end