mathLib.precision

The system function mathLib.precision returns the maximum precision (in decimal digits) for a number. For floating-point numbers (8-digit HEX for standard-precision floating-point number or 16-digit HEX for double-precision floating-point number), the precision is the maximum number of decimal digits that can be represented in the number for the system on which the program is running.


mathLib.precision syntax diagram

result

An item that receives the precision of numericItem. The result item is defined as type INT or the following equivalent: type BIN with length 9 and no decimal places.

numericItem

Any numeric or HEX item, as described in Mathematical (system words).

Example

  result = mathLib.precision(myItem);

Related reference
Mathematical (system words)