Fixed Point Number - Precision

About

Precision is the maximum number of significant digits (on either side of the radix point) in the fixed point notation

If a value exceeds the precision, then an application such as a database would return an error.

Example

Single number

  • The number <wrap box>00123456.78</note> has a precision of 8
  • 123.45 has a precision of 5
  • 1.2345 has also a precision of 5

Decimal and precision

Precision Rounded to
significant figures
Rounded to
decimal places
6 12.3450 12.345000
5 12.345 12.34500
4 12.35 12.3450
3 12.3 12.345
2 12 12.35
1 10 12.3
0 N/A 12

Powered by ComboStrap