Fixed Point Number - Precision

1 - About

Precision is the maximum number of significant digits (on either side of the radix point) in the fixed point notation

If a value exceeds the precision, then an application such as a database would return an error.

3 - Example

3.1 - Single number

  • The number <wrap box>00123456.78</note> has a precision of 8
  • 123.45 has a precision of 5
  • 1.2345 has also a precision of 5

3.2 - Decimal and precision

Precision Rounded to
significant figures
Rounded to
decimal places
6 12.3450 12.345000
5 12.345 12.34500
4 12.35 12.3450
3 12.3 12.345
2 12 12.35
1 10 12.3
0 N/A 12

Data Science
Data Analysis
Statistics
Data Science
Linear Algebra Mathematics
Trigonometry

Powered by ComboStrap