Ruby BigDecimal rounding error -


this question has answer here:

doing basic decimal maths in ruby (x = 1/b * c).

a = bigdecimal.new("1");  b = bigdecimal.new("0.8903");  c = bigdecimal.new("0.8903");  x = / b * c;  puts x 

output: 0.9999999998379

to rule out problems i've simplified test case 1 decimal (x = 1/b * b)

a = bigdecimal.new("1");  b = bigdecimal.new("0.8903");  x = / b * b;  puts x 

output: 0.9999999998379

however rearranging original formula x = c / b gives correct answer.

a = bigdecimal.new("1");  b = bigdecimal.new("0.8903");  c = bigdecimal.new("0.8903");  x = c / b;  puts x 

output: 1.0

any ideas what's causing or whether bigdecimal bug?

thanks comment telling me what's wrong floating point maths, that's why i'm using bigdecimal not floats, unless bigdecimal has same issue?

this has limit on maximum digits new bigdecimal object.

if use limit of 4 math work out.

a = bigdecimal.new("1");  b = bigdecimal.new("0.8903");  c = bigdecimal.new("0.8903");  x = a.div(b, 4) * c;  puts x 

output: 0.8903

you can find more limit here


Comments

Popular posts from this blog

aws api gateway - SerializationException in posting new Records via Dynamodb Proxy Service in API -

asp.net - Problems sending emails from forum -