Is this a problem with the math.float() call?

Questions
1

Documentation for math.float states:

This function returns the largest integer smaller than or equal to value. This rounds a number down to the nearest integer. For example, math.floor(5.7) returns 5.

Below is my code for rounding a number to a nearest decimal place. The code works fine when I pass in a float that needs to be rounded. When I pass in a number that doesn’t need rounding my call to math.floor causes this code to round down. In this case my number goes from 1.1219 to 1.2189 when I try to round to four digits.

function setup()
    f=Formatter(5)
    f:set(1.1219)
    print(f:get())
end
--===================================================
Formatter = class()

function Formatter:init(places)
    -- you can accept and set parameters here
    if places==nil or places<0 then
        places=0
    elseif places >8 then --arbitrary limit.
        places=8
    end
    self.incr = places  -- save number of decimal places to preserve
    self.num=0 -- initialize as a numeric value
end

function Formatter:set(float)
    print("In set, float = " .. float)

    round_num=float * 10^(self.incr+1)
    round=round_num % 10
    print ("In set, round_num = " .. round_num)
    print ("In set, round = " .. round)

    self.num=math.floor(float*10^self.incr) -- turn the number into an integer with self.incr decimal places
    
    print("In set, calc = " .. float * (10^self.incr))
    print("In set, calc with math.floor = " .. math.floor(float * (10^self.incr))) -- WHY IS THIS NUMBER different? 
    print("In set self.num = " .. self.num)
    

    if round > 5 then      -- round up if trailing decimal place is greater than 5
        self.num=self.num+1
    end

    self.num=self.num/10^self.incr -- return rounded decimal
end

function Formatter:get()
  retval="" .. self.num
  return retval
end
2

@nfgdayton If all you’re doing is rounding a number to X decimal places, then the string.format command does it for you. Google string.format for all the options.

function setup()
    v=1.1219
    print(string.format("5 places %.5f",v))
    print(string.format("4 places %.4f",v))
    print(string.format("3 places %.3f",v))
    print(string.format("2 places %.2f",v))
    print(string.format("1 place  %.1f",v))
end
3

@nfgdayton math.floor is working the way it should. The problem is the number 1.1219 isn’t 1.1219 . Some floating point numbers can’t be saved as exact. They might be just a little bit larger or smaller than what is printed. When 1.1219 is multiplied by 10000, it’s really 11218.999999999 . It just prints as 11219, so math.floor(1.1219*10000) prints as 11218.

4

I like this rounding function:

function math.round(number, places) --use -ve places to round to tens, hundreds etc
    local mult = 10^(places or 0)
    return math.floor(number * mult + 0.5) / mult
end
5

Thanks to you both.

For anyone that stumbles on this thread:

I looked up the Lua users group NumbersTutorial after Dave’s answer. And then wrote some code to show another unexpected behavior with floating point numbers:

--This prints 11 as expected:
    num= 1.1
    num=num * 10
    num2=string.format("%i",num)
    print(num2)
-- This prints 110 as expected
    num= 1.1
    num=math.floor(num * 100) 
    num2=string.format("%i",num)
    print(num)
-- But this raises an error when calling string.format:
    num= 1.1
    num=num * 100
    num2=string.format("%i",num)
    print(num)

The lesson I’m taking away from this is twofold: first quit thinking like a Visual Basic circa 1990 programmer, and second to not count on having an integer without passing through a math.floor function.

6

@nfgdayton An easy way to get an integer is to use the integer divide, // . That fixes the error, see below.

    num=(num * 100) // 1     -- use // 1 to force an integer

or

    num2=string.format("%i",num // 1)     -- use // 1 to force an integer.
7

That is also by far the fastest way to get an integer