Subject: [netmod] Fwd: New Version Notification for draft-ietf-netmod-yang-json-01.txt

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 13 Oct 2014 13:08:54 +0200

Hi,

this revision should address all issues and comments raised since -00. Major changes are:

   o  Metadata encoding was moved to a separate I-D, draft-lhotka-netmod-yang-metadata.

   o  JSON encoding is now defined directly rather than via XML-JSON
      mapping.

   o  The rules for namespace encoding has changed.  This affect both
      node instance names and instance-identifiers.

   o  I-JSON-related changes.  The most significant is the string
      encoding of 64-bit numbers.

   o  When validating union type, the partial type info present in JSON
      encoding is taken into account.

   o  Added section about I-JSON compliance.

   o  Updated the example in appendix.

   o  Wrote Security Considerations.

   o  Removed IANA Considerations as there are none.

Lada

Begin forwarded message:

> From: [email-masked]
> Subject: New Version Notification for draft-ietf-netmod-yang-json-01.txt
> Date: 13 Oct 2014 13:05:16 GMT+2
> To: Ladislav Lhotka <[email-masked]>, "Ladislav Lhotka" <[email-masked]>
> 
> 
> A new version of I-D, draft-ietf-netmod-yang-json-01.txt
> has been successfully submitted by Ladislav Lhotka and posted to the
> IETF repository.
> 
> Name:		draft-ietf-netmod-yang-json
> Revision:	01
> Title:		JSON Encoding of Data Modeled with YANG
> Document date:	2014-10-13
> Group:		netmod
> Pages:		17
> URL:            http://www.ietf.org/internet-drafts/draft-ietf-netmod-yang-json-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-ietf-netmod-yang-json/
> Htmlized:       http://tools.ietf.org/html/draft-ietf-netmod-yang-json-01
> Diff:           http://www.ietf.org/rfcdiff?url2=draft-ietf-netmod-yang-json-01
> 
> Abstract:
>   This document defines encoding rules for representing configuration,
>   state data, RPC input and output parameters, and notifications
>   defined using YANG as JavaScript Object Notation (JSON) text.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 13 Oct 2014 16:20:00 +0200

Lada,

thanks for the update. I think we still have open issues:

- Namespace name. The YANG namespace statement declares a URI to be
  used as a namespace while you propose to use module names in JSON.
  This makes round-trip conversions more difficult that needed. I do
  not think we resolved this issue. Lets not reiterate it but lets
  also not declare it resolved. ;-)

- The draft says YANG lists are always mapped to unordered arrays in
  JSON. It is unclear how this works with lists ordered-by user, e.g.
  a packet filter list where the order matters. I understand that
  assuming no order is also an I-JSON requirement. Unfortunately, we
  do seem to have ordered data to deal with.

- I am not sure I understand this logic:

    This document defines no mapping between the contents of JSON- and
    XML-encoded anyxml instances.  It is not necessary because anyxml
    contents are not subject to YANG-based validation (see sec. 7.10 in
    [RFC6020]).

  Are you saying things outside of validation can be simply ignored?
  It is not totally clear why this is the case. I have something like
  Kent's configlets in mind for example.

  You maybe wanted to say that anyxml data should be rendered as much
  as possible using the various JSON types but how this is done is
  implementation specific. Since anyxml itself is essentially open
  ended XML, this is no worse that using anyxml in the first place.

- It is not clear whether it is legal to encode everything into a
  string. This was discussed some time ago since essentially YANG's
  type system is based in literal string representations.

- Related to the above: Do we really treat "bar": 13.5 as an encoding
  error? What is gained by doing this in comparison to simply convert
  to a string? I heard that some implementations do kind of ignore the
  JSON type in order to improve chances of interoperability.

Does it make sense to track issues somewhere?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 14 Oct 2014 12:12:24 +0200

>   also not declare it resolved. ;-)

I believe you are the only one who wants to clutter JSON text with
namespace URIs. Consider, for example, how awkward instance-identifiers
would be. Namespace URI definition is in YANG only for the sake of
XML. Round trip conversions are not difficult (see pyang), the only
caveat being that you need access to the data model, but this is
inevitable anyway.

>   do seem to have ordered data to deal with.

No, this is not what the draft says. JSON arrays are ordered, so
user-ordered lists are supported. The draft just says that the order
of members in a list *entry* is arbitrary. This means list keys needn't
precede other siblings in the entry as it is required in XML.

>   Kent's configlets in mind for example.

The central theme of the draft is no more XML-JSON translation, it just
defines JSON encoding is a similar was as the "XML Mapping Rules"
sections in RFC 6020 do for XML. So section 5.5 specifies the valid
encoding of an anyxml instance.

If we have "anyxml foo;",  then we know

"foo": [null, [1, 2, true]]

is a valid instance. Why should the draft try to specify the translation to XML?

In the case of configlets, Kent might provide a specification of their
encoding in both XML and JSON.

>   ended XML, this is no worse that using anyxml in the first place.

I am not sure I understand, the draft just says that an anyxml instance
is encoded as a name/value pair and that the value part has to be a valid
JSON value. The sentence you cited is related to sec. 3 which relies on
JSON->XML mapping for instance validation - and for this purpose it is
really irrelevant how anyxml content is translated as long as we start
with valid JSON and end up with valid XML. 

>   type system is based in literal string representations.

What do you mean? For example, sec. 6.3 clearly says that boolean values
are encoded as JSON literal names 'true' and 'false', i.e. no
string. The same is true for numbers except for 64-bit ones.

I don't agree that YANG type system requires string encoding of the
values.

>   error? What is gained by doing this in comparison to simply convert

Yes. The client says it is a number so why should the server coerce it
to a string on its own?

>   JSON type in order to improve chances of interoperability.

Do you have an example how this could lead to interoperability problems?

> Does it make sense to track issues somewhere?

Any suggestion? The draft is now being developed as a project on github
(https://github.com/yang-json), so we can track issues there but I think
we agreed not do that.

One general note: I think we need to clarify the purpose of this
document. My aim has been to find an encoding of YANG concepts in a way
that's appropriate and natural for JSON. Your preference seems to be to
bend JSON encoding so that it is as close to XML as possible.

The correspondence between JSON and XML encoding of the same data is
important but it is not the primary goal. I assume some systems or
deployments can use exclusively JSON without performing any
translations.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 14 Oct 2014 15:43:00 +0200

> caveat being that you need access to the data model, but this is

Yes, I prefer to be able to do conversions without having to have the
data model at hand. You like to design things such that it is required
to have the data model available for any conversion. It seems
everything else follows from this.

> precede other siblings in the entry as it is required in XML.

OK.
 
> with valid JSON and end up with valid XML. 

I can agree on "the draft just says that an anyxml instance is encoded
as a name/value pair and that the value part has to be a valid JSON
value." - where I am having trouble is the reasoning given. Perhaps
this does the job equally well:

OLD

   An anyxml instance is translated to a name/value pair.  The value can
   be of any valid JSON type, i.e. an object, array, number, string or
   any of the literals 'true', 'false' and 'null'.

NEW

   An anyxml instance is translated to a name/value pair.  The value can
   be of any valid JSON type, i.e. an object, array, number, string or
   any of the literals 'true', 'false' and 'null'. How the anyxml content
   is mapped to JSON types is implementation specific.

> I don't agree that YANG type system requires string encoding of the

The question is whether you mandate that the encoder has type
information available when rendering the JSON structure and whether
you mandate that a receiver fails if the JSON type does not match its
expectations (given the YANG data model it knows about). It again is a
question whether we mandate type information to be available or
whether a sender is allowed to send "1234" as "1234" in case no type
information is available.

> Yes. The client says it is a number so why should the server coerce it

Because the API used internally is not using the JSON type system but
a type system that stems from YANG. JSON is an encoding, not more. And
the union type shows that to follow what YANG does, you have to
stringify values or you have to drop values because of type conflicts.
I think I recall that at least one RESTCONF implementation converts to
strings instead of raising encoding errors.

> Do you have an example how this could lead to interoperability problems?

There are implementations today where the lower layers doing the
encoding/decoding work do not have access to YANG type information. If
we require knowledge of data models by the encoder, we surely raise
the bar for such implementations. The fact that pyang can do things
easily (which is no surprise since it is a YANG compiler) does not
mean all implementations can do this easily.

> (https://github.com/yang-json), so we can track issues there but I think

I do not care much _how_ issues are tracked. I am fine with letting
those pick a format who will finally maintain the issue tracker.

> deployments can use exclusively JSON without performing any

The basic question is whether we require YANG type information to be
present at all JSON encoders or not (and whether JSON type information
has to be taken authoritative by the decoder). I think this is where
we differ and the rest follows from that.

/js

PS: I just found that the I-D does not talk about uint32, likely simply
    an omission.

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 14 Oct 2014 17:32:09 +0200

> everything else follows from this.

It is not about conversions, I can imagine an implementation that doesn’t touch XML at all. The right question for me is: If we have YANG and forget about XML, what is the most natural representation of instance data in JSON? The answer probably won’t be that numbers be encoded as strings. Actually, I think it is a testimony to YANG design that the JSON encoding can be done quite nicely.

>   is mapped to JSON types is implementation specific.

The problem with this is that there needn’t be any mapping involved, e.g., a client could just put a JSON value there and the server will receive it without doing any conversion. So in fact anyxml has the meaning of “anyjson” or “anydata” here.

With respect to sec. 3, the text could perhaps say that the mapping of anydata instances to XML *for validation purposes* is an empty element. Nothing else is needed. 

> information is available.

My understanding is that even XML encoder has to use the data model in order to be able to distinguish integers from decimal numbers and encode them properly. And similarly for decoders - they have to know whether a received (string) value is to be put into an integer or string variable.  

> strings instead of raising encoding errors.

We can discuss this but I think at the root of the issues with unions is the inability to indicate the actual type of an instance. I don’t agree though with introducing a hard rule that all union values be encoded as strings. 

> mean all implementations can do this easily.

So in order to allow existing implementations that were designed for XML to use JSON, you propose to force the JSON encoding into a mode that is XML compatible. This however makes life harder for implementors of *new* applications written specifically for JSON because they have to do the work that’s normally done by a JSON parser. For instance, if a number encoded as a string is received, the JSON parser parses it as a string and then another layer has to parse it as a number.

> those pick a format who will finally maintain the issue tracker.

OK, so I propose to file the issues at

https://github.com/yang-json/yang-json/issues

and I will ocassionally report about the status of the issue tracker in the mailing list.

> we differ and the rest follows from that.

I guess the dilemma really is this:

Do we want to define an ultra restricted JSON profile where all scalar values are strings, or not?

I’d suggest that folks who want to use the JSON encoding speak up.

> PS: I just found that the I-D does not talk about uint32, likely simply

Oh yes, copy-and-paste error, fixed, thanks.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 15 Oct 2014 10:57:50 +0200

> It is not about conversions, I can imagine an implementation that doesn’t touch XML at all. The right question for me is: If we have YANG and forget about XML, what is the most natural representati…

I can imagine many things as well. This discussion does not help.

> With respect to sec. 3, the text could perhaps say that the mapping of anydata instances to XML *for validation purposes* is an empty element. Nothing else is needed. 

If you dislike 'mapped to' replace it with 'represented by'.

> My understanding is that even XML encoder has to use the data model in order to be able to distinguish integers from decimal numbers and encode them properly. And similarly for decoders - they have…

I think I explained it before. I am familiar with one implementation
that maintains a tree structure where all values are in string
representation and the tree structure does not carry any type
information. Serializing this tree structure to XML is straight
forward. Serializing this tree structure to JSON is difficult since it
is now required to know type information and namespace to module name
mappings. Perhaps the concensus will be that this kind of
implementation is wrong and needs to change for JSON (and perhaps
other encodings) but please do not argue that this code does not
exist.

> We can discuss this but I think at the root of the issues with unions is the inability to indicate the actual type of an instance. I don’t agree though with introducing a hard rule that all union v…

Whether we/you like it or not, this is how things work today. Section
9.12. says:

   When a string representing a union data type is validated, the string
   is validated against each member type, in the order they are
   specified in the "type" statement, until a match is found.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 15 Oct 2014 11:43:32 +0200

> I can imagine many things as well. This discussion does not help.

Well, so what does help? I think it comes down to the role we assign to JSON in the NETCONF/YANG ecosystem, i.e. whether the encoding provides a first-class representation of YANG-modelled data, or just a dummy intended for appeasing those malcontents that push JSON.

Also, JSON is more than just another encoding in the sense of UTF-8 versus UTF-16, it is a different API. 

> If you dislike 'mapped to' replace it with 'represented by’.

Then either the last sentence says the same as the preceding sentence (any valid JSON value), or I don’t understand what “anyxml content” means.

> other encodings) but please do not argue that this code does not

But this is all implementation detail, as Andy likes to say. I understand that applications that were written only for XML may be structured in a certain way. An application with support for JSON should perhaps be structured in a different way but this doesn’t mean it is more difficult to write.

The jsonxsl plugin in pyang represents the JSON encoding logic as an XSLT stylesheet, so I guess any programming language should be able to do the same.  

>   specified in the "type" statement, until a match is found.

In our case we are validating a number representing union data type, not string.

Lada

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 15 Oct 2014 11:51:19 +0200

On Wed, Oct 15, 2014 at 11:43:32AM +0200, Ladislav Lhotka wrote:
> Well, so what does help? 

It helps if we two shut up and others who followed the discussion tell
us their opinion.

To be clear, I will not involve myself in determining consensus on
this set of issues. I will ask my co-chair to do this.

/js