Subject: Re: [netmod] canonical form of floats

From: David Partain <[email-masked]>
Date: Mon, 9 Feb 2009 15:00:17 +0100

On Monday 05 January 2009 11.01.18 Martin Bjorklund wrote:
> Andy Bierman <[email-masked]> wrote:
> > XPath parsers will not accept the "+/- exponent" part,
> > and will (incorrectly) accept an AdditiveExpr [rule 25] instead.
> >
> > Since that won't work, even though it exactly meets
> > the real needs of the NETCONF community, setting
> > the precision manually for the %f option seems
> > the obvious next choice. The values to set manually
> > are not so obvious.
>
> So, to summarize... we'd like to be compatible with existing
> technology.  At a first glance, XSD seems to be the obvious candidate,
> but XSD defines the canonical form to be in a form that XPath does not
> understand.  So if we use the XSD definition, XPath won't work.
>
> So we need to come up with our own definition.
>
> So, what value should we use for the precision in %f?  Is there a
> theoretical maximum we could use?  Otherwise we'll have to pick a
> "large" number.  Unfortunately, %f pads with trailing zeros, but OTOH,
> since it's straightforward to remove these trailing zeros after the
> call to snprintf, we could state that trailing zeros are removed in
> the canonical form.
>
> Also, if we use "%f" we might want to align our "[-]INF" and "NaN" to
> "[-]inf" and "nan".

As this issue remains in -03 (see section 9.3.2)...  and the thread went cold 
about a month ago.

I'd like to see comments on Martin's suggestion (that we come up with our own 
definition and that we use %f).  To me, this seems like a reasonable way of 
dealing with this.

Cheers,

David

---

From: Andy Bierman <[email-masked]>
Date: Mon, 09 Feb 2009 06:30:12 -0800

> definition and that we use %f).  To me, this seems like a reasonable way of 

I did some experiments:
   - %.14f seems to be safest, although sometimes 15 decimal
       places of precision is OK.
   - the only glibc option that matches YANG and XPath is %f,
     and you have to manually remove trailing zeros and maybe
     the decimal point as well.
   - strtof is broken in glibc, so float32 is quite problematic.
     strtod works fine, so 'double' is not affected, just 'float'.
     (try strtof('1.3')).
   - I prefer to use the glibc names for inf, -inf, and nan,
     instead of inventing new terms.

Attached is a terse summary of the precision issue (x.c and its output)

> David

Andy

#include <stdio.h>

int main (void)
{

    double d;

    d = 1.3;
    printf("\n%.14f", d);
    printf("\n%.15f", d);
    printf("\n%.16f", d);
    printf("\n%.17f", d);
    printf("\n%.18f", d);
    printf("\n");

    d = 1.99999999999999;
    printf("\n%.14f", d);
    printf("\n%.15f", d);
    printf("\n%.16f", d);
    printf("\n%.17f", d);
    printf("\n%.18f", d);
    printf("\n");

    d = 1.999999999999999;
    printf("\n%.14f", d);
    printf("\n%.15f", d);
    printf("\n%.16f", d);
    printf("\n%.17f", d);
    printf("\n%.18f", d);
    printf("\n");

    d = 1.9999999999999999;
    printf("\n%.14f", d);
    printf("\n%.15f", d);
    printf("\n%.16f", d);
    printf("\n%.17f", d);
    printf("\n%.18f", d);
    printf("\n");

    return 0;
} 

1.30000000000000
1.300000000000000
1.3000000000000000
1.30000000000000004
1.300000000000000044

1.99999999999999
1.999999999999990
1.9999999999999900
1.99999999999999001
1.999999999999990008

2.00000000000000
1.999999999999999
1.9999999999999989
1.99999999999999889
1.999999999999998890

2.00000000000000
2.000000000000000
2.0000000000000000
2.00000000000000000
2.000000000000000000

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 9 Feb 2009 15:32:56 +0100

On Mon, Feb 09, 2009 at 03:00:17PM +0100, David Partain wrote:
> As this issue remains in -03 (see section 9.3.2)...  and the thread
> went cold about a month ago.
> 
> I'd like to see comments on Martin's suggestion (that we come up
> with our own definition and that we use %f).  To me, this seems like
> a reasonable way of dealing with this.

I understanding is that %f alone limits the precision to 6 digits
which is likely not what we want. So what is the suggested precision?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Randy Presuhn" <[email-masked]>
Date: Mon, 9 Feb 2009 12:11:09 -0800

Hi -

> From: "Andy Bierman" <[email-masked]>
> To: "David Partain" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Monday, February 09, 2009 6:30 AM
> Subject: Re: [netmod] canonical form of floats
...
> Attached is a terse summary of the precision issue (x.c and its output)
...

Thanks for making such a clear demonstration of why converting
things to floats may have results surprising to many humans, and
why the choice of comparison as string / comparison as float /
comparison of internal representation of whatever the data type is
matters.

In addition to the precision issue, there's also a question of
accuracy (e.g, 1.3 does not have an exact representation),
as well as the question of how you'd like the various flavours
of NaN, negative zero, and so on to appear.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 17 Feb 2009 08:28:26 +0100 (CET)

> which is likely not what we want. So what is the suggested precision?

The problem with %f is that if we want to be able to print all float64
values, the precision has to be Very Large (300+?)  For example, the
value 1E-300 will be printed as 

0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001

And large values (1E300) will not be easy to ready:

1000000000000000052504760255204420248704468581108159154915854115511802457988908195786371375080447864043704443832883878176942523235360430575644792184786706982848387200926575803737830233794788090059368953234970799945081119038967640880074652742780142494579258788820056842838115669472196386865459400540160.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

That's why I suggested %.17E as an alternative.  I believe that this
ensures that all float64 (double) values can be printed as decimal and
converted back to the same value, which seems to be a useful property.

One drawback with this is that XPath 1.0 does not understand this
form.  If it is important, we could solve this for YANG's "must" and
"when" expressions by introducing a new XPath function
(e.g. strtod()), but we would still have an issue with NETCONF XPath
filtering.

I would rather use "%.17E" (with trailing zeros removed) and loose the
ability to use XPath filters, then having to use "%.300f".

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 Feb 2009 03:35:37 -0800

> value 1E-300 will be printed as 

This is more than a drawback.
This is a full-stop, XPath is the wrong hammer, kind of decision.
We looked at this months ago and realized that the %e or %E
format will be misinterpreted in XPath as an AdditiveExpr.

    container foo {
      must "bar+1.45234+E7<foo";

      leaf E7 {...}
      leaf foo {...}
      leaf bar {...}
    }

Let's see. We need to use XPath to remain compatible
with off-the-shelf XML tools.  But the NULL namespace
is really the module namespace, the root of the 'document'
keeps changing depending on the usage, the 'document order'
is implementation-specific, and now the IEEE 754 numbers
are 'special' as well? (!)

That means that people will need to write 1.2345+E4 instead
of 12345?  That is not easier to read. It only helps for
corner-case numbers that are very unlikely to occur.
Do routers even have floating point math?  What do they
need these large FP numbers for?

> ability to use XPath filters, then having to use "%.300f".

You mean drop must/when completely?
You mean drop :xpath "select" from 4741?

(I prefer to keep must/when and :xpath).

If we were to choose the correct hammer from the toolbox,
what would it look like?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 17 Feb 2009 12:50:07 +0100 (CET)

> This is a full-stop, XPath is the wrong hammer, kind of decision.

We *can* use %.350f and accept the more-than-ugly output.  Then XPath
will work.

Note that this is an issue with plain NETCONF and :xpath, regardless
of data modelling language.  If floats are sent using the exponent
format, XPath filtering will not work the way you might think.

>       must "bar+1.45234+E7<foo";

Right, so we *could* solve this with:

       must "bar+strtod('1.45234+E7')<foo";

> You mean drop must/when completely?

No.  I mean either introduce 'strtod' as above or accept that you will
not be able to compare floats in must/when expressions.

> You mean drop :xpath "select" from 4741?

No.  I mean that you will not be able to compare floats in the select
expression.  (unless strtod() is introduced in :xpath as well).

> If we were to choose the correct hammer from the toolbox,

An "adapted subset" of XPath 1.0 - which does not sound too
attractive.

What is your suggestion for how to deal with this?  %.350f?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 Feb 2009 04:16:11 -0800

> What is your suggestion for how to deal with this?  %.350f?

no.  FP numbers are more complicated than that.
Asking for too much precision produces garbage results.
You end up with spurious numbers in the fraction, not all zeros.
(So 3 might turn into 3.000000000000000000000000000000000065).

IMO, FP numbers are an extremely low priority
in our NM problem space.  We deal with Counter32 and Counter64.
On very rare occasions when FP has been needed, suitable hacks
using ordinal numbers or more than 1 MIB object have been
sufficient instead.  Statistical analysis is usually done
on the NMS, so FP division is not even needed for that on the agent.

What are the priorities for the solution?
The strtod() function looks promising for input.

How does the agent generate FP numbers in rpc-reply
and notification responses for float32 and float64 data types?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 Feb 2009 04:45:11 -0800

> An "adapted subset" of XPath 1.0 - which does not sound too

No, but in reality it is extremely close to 'real' XPath.
Out of around 450 implementation details, about 5 of them
are different for YANG.  In every instance of unexpected
behavior (if you are an XPath expert), there are better
ways to use XPath also available to the data modeler
that do not have problems (e.g., foo[name='fred'], not foo[3]).

IMO, a function library extension for some really useful NM-stuff
is a feature, not a bug.  Advertise a :yanglib capability and
it is no different than anything else we put in the data model.

> What is your suggestion for how to deal with this?  %.350f?

Take a look at %g output in glibc.
That looks really nice -- the format is fixed for small numbers
and all the trailing zero 'cleanup' is done by glibc.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 17 Feb 2009 05:20:49 -0800

...
>        must "bar+strtod('1.45234+E7')<foo";

Actually, the syntax is 1.45234E+07.
This is slightly better, since it will cause a fatal XPath
error, rather than being silently accepted as the wrong
kind of expression.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 18 Feb 2009 10:39:44 +0100

Ericsson has nodes with a proprietary programming language that does not even have a float 
datatype. I fully agree this is very low priority.

Did you ever need more then 3 digits of precision in any counters, config parameters?
Balazs

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 18 Feb 2009 11:13:34 +0100

On Wed, Feb 18, 2009 at 10:39:44AM +0100, Balazs Lengyel wrote:
> Ericsson has nodes with a proprietary programming language that does not 
> even have a float datatype. I fully agree this is very low priority.
>

Precision definitely yes - especially if you mention counters in the
way SNMP people understand counters.

And I believe if we do floats, we would be badly advised to limit
their precision to 3 digits.

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 18 Feb 2009 02:16:56 -0800

> Did you ever need more then 3 digits of precision in any counters, 

Actually, SNMP counter size is picked based on whether
or not the counter can roll over in less than an hour.
(I worked on IOS for 10 years, which didn't have FP at all.
The interrupts were 'borrowed' for other purposes on some platforms.)

As I've said before: Adding FP support for network management
is a non-starter on so many levels.  But a YANG implementation
can hardwire the must/when (as the draft points out), and
not really do any FP math at all (yeah, right ;-)

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 18 Feb 2009 11:40:35 +0100

I meant float values with a precision of more then 3 digits. (I know counter64 has more then 3 
digits.)

I am curios whether you have ever used float values with more then 3 digits of precision?
Sorry for the misunderstanding.
Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
TSP System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320     email: [email-masked]