Subject: [netmod] namespaces in YIN

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 12:01:01 +0100

Hi,

Section 11.2 describing YANG-2-YIN algoritm should specify the namespace
for the argument attribute/element - it is important for YIN encoding of
extensions. I propose the following change to the 5th paragraph:

    the argument's "yin-element".  If "yin-element" is false, the
    argument is encoded as an XML attribute to the keyword's element.  If
    "yin-element" is true, the argument is encoded as a subelement to the
-   keyword's element.  The name of the attribute or element is the name
-   of the argument.
+   keyword's element.  The local name of the attribute or element is the
+   name of the argument. The attribute has no namespace whereas the
+   element is in the same namespace as the keyword's element. 

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 06:46:36 -0800

> +   element is in the same namespace as the keyword's element. 

The attribute needs a namespace.  No namespace
is not the element namespace.  It is NULL.

There aren't going to be any 'special' rules for YIN, right?
It is not some 'special' XML that works almost the same
as regular XML, except for 3 or 4 things?  We already have an XPath
like that.

What happens if there are multiple XML attributes within
an element in a YIN document?  Is this allowed?
If so, then the mapping is not symmetrical because
XML attributes are unordered within an element.
XML parsers do no preserve the relative order of
attributes.  There are some YANG statements
where the relative order of the statements is important.
How does YIN preserve statement order in this case?

IMO YIN is under-specified.

Is it possible to convert from YANG to YIN and back to YANG
perfectly?  Even comments?  If not, why not?
Explain in all cases what exact information can be lost.
Explain how every single element and attribute in a YIN
document maps to exactly 1 construct in the YANG ABNF,
since all the normative text is in terms of the YANG clauses.
It is not clear how any particular node in a YIN document
maps to a precise subset of the YANG ABNF.  Doesn't the
draft need a detailed YIN-2-YANG algorithm as well
as YANG-2-YIN?  There are no examples of YIN in the
entire draft.  Is this going to change?

> Lada

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Feb 2009 16:07:48 +0100 (CET)

Hi,

> There aren't going to be any 'special' rules for YIN, right?

No.

> an element in a YIN document?  Is this allowed?

No; this needs to be specified.

> How does YIN preserve statement order in this case?

I think this is not a problem since there is just one attribute,
right? 

> IMO YIN is under-specified.

So now is a good time to fix that!

> perfectly?

Yes, with the exception of whitespace and comments.

>  Even comments?  If not, why not?

We currently say:

  Comments in YANG MAY be transformed into XML comments.

and:

  XML comments in YIN MAY be transformed into YANG comments.

Comments should not be used to carry semantic information.

Most comments *can* easily be translated; except for constructs like:

   description "foo" + /* one */ "bar" + /* two *' "baz";

> Explain in all cases what exact information can be lost.

I will add text that explains that whitespace and comments can be
lost.

> since all the normative text is in terms of the YANG clauses.

I think this is covered by the text in 11.3 and the table in 11.2.

> as YANG-2-YIN?

See 11.3

> There are no examples of YIN in the

There is one!  See 11.2.1.  Do we need more examples?

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 16:09:34 +0100

> is not the element namespace.  It is NULL.

Right, but this is the normal handling of attributes in XML under
namespaces. Unlike elements, attributes do not get the default namespace
within the XML instance and are thus mostly in the null namespace,
unless an explicit prefix is used. In YIN, these argument attributes
could be also in the keyword's elemnt namespace but I don't see any
reason for this - it's just more typing.

> as regular XML, except for 3 or 4 things?  We already have an XPath

It is not special at all:

     <rpc message-id="101"
          xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"
          xmlns:ex="http://example.net/content/1.0"
          ex:user-id="fred">
       <get/>
     </rpc>

Here the rpc element is in the netconf namespace but message-id
attribute has no namespace.

> an element in a YIN document?  Is this allowed?

How could this happen? Every YANG statement has at most one argument
hence the corresponding YIN element may have zero or one attribute.

...

> perfectly?  Even comments?  If not, why not?

I think the conversion is perfect in both directions. Perhaps the
specification may be improved in some places but I would call it
underspecified.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 07:25:02 -0800

> reason for this - it's just more typing.

We agree on how XML works.
I think the attribute namespace always has
to be specified.  It MUST be the YIN namespace
or an extension namespace.

Machines are generating YIN and processing it.
That is the reason it exists, according to
the draft.  'YIN readability' is not a design goal.

>           xmlns:ex="http://example.net/content/1.0"

Interesting example.
I actually send <nc:rpc nc:message-id="101" ... in my code
because the XSD says the attribute is defined
in a target namespace.  But the agent accepts 'nc' or none
in every case because the RFC says to do it wrong.
I'm sure 4741-bis will fix it though.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 07:50:32 -0800

> I think this is not a problem since there is just one attribute,

I am still unclear on the text about the YIN XML namespace prefix
exactly matching the YANG module prefix.  YANG module prefixes
are not unique.  That's why the prefix can be specified in
the import-stmt (to give the data modeler the joy of
resolving all the prefix collisions manually ;-)

So how does YIN deal with this problem?

IMO this prefix mapping is all broken.
It needs to work the same way our XPath code has to work.

   XML prefix --> namespace URI --> module namespace-stmt value

Leave module prefixes out of it.

> So now is a good time to fix that!

Great! :-(

What about the XPath prefix issues?
How are all the prefixes within must/when path expressions
represented?  What happens when they are missing?
Is this behavior different for elements vs. attributes?

Why does YIN need to use attributes at all?
Why can't YANG and YIN be simplified by always using
elements in YIN?  (This should be in the draft.)

The draft also says that YIN can be schema-validated.
Theoretically, yes, but who is going to write the XSD or DSDL
for YIN, or better yet, the yin.yang module?

> There is one!  See 11.2.1.  Do we need more examples?

you always need more examples!
But only correct ones.  Bugs in examples
are worse than no examples.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 16:59:53 +0100

> or an extension namespace.

Why? There cannot be any confusion since the attributes are enclosed in
and actually belong to an element that has a non-null namespace. I'd
guess some 90 % XML attributes are handled this way. Namespace is really
needed only for global attributes that are attached randomly to elements
from foreign namespaces.

BTW, the YIN translator in pyang does it exactly as it is said in the
text I proposed.

> the draft.  'YIN readability' is not a design goal.

Well, I for one write YIN by hand - the fantastic nxml mode in emacs
knows all the YIN keywords and autocompletes them so it's actually less
typing than writing YANG.

> I'm sure 4741-bis will fix it though.

Then I think the XML Schema is broken and should be fixed.

> > specification may be improved in some places but I would call it

Oh, I wanted to say "... I would NOT call it underspecified."

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 08:03:37 -0800

Martin Bjorklund wrote:
....
> I will add text that explains that whitespace and comments can be

What about YANG strings?
There are very specific semantics for double-quote strings
vs. single-quote strings.  How are these semantics preserved
in YIN <--> YANG round-trip translation?  Especially all
the formatting and whitespace rules for double-quotes.

How is string concatenation preserved?

   must "/foo" + "/bar" + '[name = "fred"]' + "< 4";

What is the YIN translation of this must-stmt
(in element and attribute form if different)?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 08:15:16 -0800

> BTW, the YIN translator in pyang does it exactly as it is said in the

OK.  Just make it clear in the draft.
I do not mind having the draft match the free tool for
this feature.  Nobody else is going to write one.

> typing than writing YANG.

IMO, debating implementation and tool choices can
never really be productive.   In this case, we might
even end up in the "emacs vs. vi" tar-pit. :-)

> Oh, I wanted to say "... I would NOT call it underspecified."

I think after a little more work it will be fine.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 17:34:47 +0100

> the formatting and whitespace rules for double-quotes.

IMO, the value-space value of the YANG string must be taken and put as
the argument attribute value or element text, substituting entities like
&lt; where necessary.

> (in element and attribute form if different)?

The draft says the argument of must goes to the "condition" attribute,
so it cannot be an element. The result will be

<must condition='/foo/bar[name = "fred"]&lt; 4'>.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 17:56:23 +0100

> > draft need a detailed YIN-2-YANG algorithm as well

Actually, I wrote the RELAX NG schema for YIN pretty much by directly
translating the ABNF grammar:

http://www.yang-central.org/twiki/pub/Main/YangTools/yin.html

Perhaps this schema should also appear in an appendix of the draft - I'd
have to update it to the current syntax though.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 08:56:52 -0800

> <must condition='/foo/bar[name = "fred"]&lt; 4'>.

So the draft needs to say that all YANG string features
are lost in translation and any YANG module which uses
them may not translate correctly (e.g., \n left in
the string vs. converted to an actual newline).
String concatenation is not translated at all.

There are some corner-cases that I have found (e.g., patterns)
where the YANG double-quotes break the pattern and
single-quoted strings must be used instead.
In your example, some dbl-quoted strings were
grouped together into a single-quoted string.
I think this can change the semantics of some YANG strings.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 25 Feb 2009 18:14:22 +0100

> I think this can change the semantics of some YANG strings.

It shouldn't. These single/double quotes and concatenation are just
features of the lexical representation of strings in YANG, but the value
space value of the argument string is always well-defined and unique and
must not change during the conversion. Of course, going from YANG to YIN
and back to YANG may not yield exactly the same module text that we
start with, but they should be equivalent.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Feb 2009 18:30:55 +0100 (CET)

> I'm sure 4741-bis will fix it though.

No this is not correct.  The XSD has 

               attributeFormDefault="unqualified"

which means that locally declared atttributes MUST NOT be qualified
with the namespace.  So:

       <nc:rpc nc:message-id="101" ..,>

is not valid according to the schema.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Feb 2009 18:51:13 +0100 (CET)

> So how does YIN deal with this problem?

I don't understand the problem.  All prefixes in a YANG module are
unique.  Each prefix maps to a module, which defines a namespace.
Maybe you can provide an example which has this problem?

> Is this behavior different for elements vs. attributes?

Note that YIN is just a different syntax.  The mapping does not modify
any string *values* (just on the syntax level, e.g. '>' must be
quoted in XML).  So if you believe we have any prefix issues in the
interpretation of XPath in YIN, then we have them in YANG as well.

> elements in YIN?

There are of course many ways an XML syntax of YANG could be
specified.  We chose one which we believe is correct (i.e. 1-1
mappable), readable, writable, and simple to implement.  The reason
some elements have attributes is for readability and writability.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 10:03:23 -0800

> is not valid according to the schema.

But we always show a prefix on the 'operation' attribute
in examples. (pg 15, 38, 39)

If a prefix is not allowed on 'message-id' than it is
not allowed on 'operation' either. Neither attribute declaration
has the 'form' option, so the attributeFormDefault of 'unqualified' is
used.  IMO, the attributeFormDefault=unqualified is a bug.

The WG intended that the operation attribute is in
the netconf namespace and all the attributes in the <rpc>
element are mirrored back exactly, regardless of form.
In fact, the mandatory 'message-id' is considered a bug.
It should not exist at all, and the error for a missing
message-id should be removed.  It is not used anywhere
for anything.  Just a Big CLR.

We had limited XSD expertise at the time (as demonstrated by some
other bugs found so far).

BTW, the YANG attributes MUST have a prefix and be
in the YANG namespace.

IMO this is all a bit confusing.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Feb 2009 19:29:40 +0100 (CET)

Andy Bierman <[email-masked]> wrote:
> > No this is not correct.  The XSD has attributeFormDefault="unqualified"

The 'operation' attribute is globally declared.  Globally declared
attributes always need the namespace.

> element are mirrored back exactly, regardless of form.

Sure.

> for anything.  Just a Big CLR.

I agree.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 25 Feb 2009 11:08:13 -0800

> attributes always need the namespace.

OK.  Got it.
The <attribute> definition for 'operation' is a global definition
so it must always be qualified.

The <attribute> definitions for 'message-id', 'type', and 'select'
are inside <complexType> elements and so the attributeFormDefault
is applied to them.  Therefore they must never have a prefix.

>> for anything.  Just a Big CLR.

Is anybody enforcing this error or not?
Phil already explained to everybody why
the message-id is already covered by
the 'mirror-all-attributes' rule.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Feb 2009 08:59:21 -0800

> Maybe you can provide an example which has this problem?

My concern is 11.2, para 3, sentence 2, and para 4, sentence 2:

    Elements which represent keywords that are imported extensions from
    other modules MUST be properly namespace qualified, where the
    namespace is the namespace of the imported module.  The XML prefix
    for such extensions MUST be the same as the prefix defined in the
    module's "import" statement.

    Elements which represent keywords that are included extensions from
    other submodules MUST be properly namespace qualified, where the
    namespace is the namespace of the module that the submodule belongs
    to.  The XML prefix for such extensions MUST be the same as the local
    prefix, i.e. for a module it is as defined in the "prefix" statement,
    and for a submodule, as defined in the submodule's "belongs-to"
    statement.

There is no reason for this CLR about the XML prefix matching
the module prefix.  Sentence 1 in each paragraph is enough.
Sentence 2 is just a CLR.

Here is the only example of YIN in the draft (plus an extension):

      <leaf name="mtu">
        <acme:foo xmlns:acme="acme-module-namespace">
           <acme:bar>extension value</acme:bar>
        </acme:foo>
        <type name="uint32"/>
        <description>
          <text>The MTU of the interface.</text>
        </description>
      </leaf>

module X {
   namespace "acme-module-namespace";
   prefix X;

   extension foo {
     description "";
     argument bar {
        yin-element true;
     }
   }
}

According to the draft, the example above is an error
because the arbitrary value I used for my XML prefix
is 'acme', not 'X'.  IMO, it is wrong to force XML tools
to use YANG hack values instead of the allowing the
freedom they expect from the XML standard.

But this hack is needed, because the import-stmt
specifies the module prefix (e.g., X).  The tool
does not know where to find the module that
has namespace-stmt = 'acme-module-namespace'.

I seriously doubt any off-the-shelf XML tools are going
to be clever enough to add semantics to the arbitrary
values used in XML prefixes, match them up with the
particular XML subtrees for the import-stmts,
then go find the module X to make sure it has
an extension 'foo', and check its argument-stmt
to see if 'bar' is OK.  All this, above and beyond
the plain schema validation normally done?

> some elements have attributes is for readability and writability.

Fine.

Since YANG does not support XML attributes at all
because they are inferior to elements, it seems odd
that YIN makes such heavy use of them.  But I'm sure people
who write YIN will eventually memorize the 2 page table
in sec. 11.2, so it doesn't matter.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 26 Feb 2009 18:29:45 +0100

> Sentence 2 is just a CLR.

+1

> freedom they expect from the XML standard.

Schema languages typically use other mechanisms than xmlns:xxx
attributes for declaring the namespace for their target vocabulary, so
this is not really special. The only difference is that the extensions
in YANG add new elements to the schema language itself that are in the
same namespace as the vocabulary being defined by the module - i.e., the
vocabulary is mixed with meta-vocabulary. I think this is a wrong
language design - the extensions should actually be specified to extend
the ABNF for YANG or XML schema for YIN and should have a different
namespace than any YANG module defining data models.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Feb 2009 10:24:54 -0800

>> Sentence 2 is just a CLR.

This is one of those emails that, by the time I got to the end,
I realized what Martin was talking about, and I understand why
the hack is needed.

The YIN parser needs to find all the appropriate schema files
in some manner -- normal XML stuff.

Here is where YIN translation based on pure XML breaks.
The namespace-stmt value in module X is not required to change
every time a new revision is published.  (IMO, it MUST NOT change).
So ignoring the import-stmt doesn't always work.

This is different than parsing XPath in PDUs.
The agent has all the modules pre-loaded and the PDU must
match what it knows.

The YANG or YIN compiler starts with [sub]module X
and does not know which versions of which imports
are needed in advance.  The import revision date
cannot be ignored if it is present.  The module
name is obviously needed as well, in order to use the
revision date (instead of the XML namespace that
matched, but the module name cannot be derived from that URI).

> namespace than any YANG module defining data models.

The extensions are required to use a prefix,
and must be some other namespace than the one for YIN.
They are not mixed in YANG at all.  They must always
use a prefix there as well.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Feb 2009 20:32:50 +0100 (CET)

> > Sentence 2 is just a CLR.

Yes, I agree.  In fact, the yin parser in pyang already accepts any
XML prefix for this case.  I will remove this CLR.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Feb 2009 11:42:44 -0800

> XML prefix for this case.  I will remove this CLR.

Do you make sure the module you pick matches an import-stmt?

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Feb 2009 21:23:50 +0100 (CET)

> Do you make sure the module you pick matches an import-stmt?

Yes.  The XML prefix maps to a URI, and from the imports I create a
table which maps URI to module.  So given the XML prefix its easy to
locate the module.  If the module is not imported, its URI will not be
in the table, and an error is generated.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Feb 2009 13:38:21 -0800

> in the table, and an error is generated.

OK, remove the CLR then.

It works the same way as for parsing YANG,
like you were saying all along.  In YANG,
it helps to defer validation of extensions
in case they appear before the import-stmt
that defines them.  I guess the same
applies in YIN as well.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sat, 28 Feb 2009 15:40:25 +0100 (CET)

Hi,

Ladislav suggested that we change the way YIN is described to a
declarative style, rather than the imperative style we have today.  He
proposed the following text (thank you Lada!), which I think is better
than the current one.  This text would replace sections 11.1, 11.2, and
11.3.

11.1.  Formal YIN Definition

   There is a one-to-one correspondence between YANG keywords and YIN
   elements.  The local name of a YIN element is identical to the
   corresponding YANG keyword.  This means in particular that the
   document element (root) of a YIN document is always <module> or
   <submodule>.

   YIN elements corresponding to the core YANG keywords belong to the
   namespace whose associated URI is
   "urn:ietf:params:xml:ns:yang:yin:1".  [XXX IANA].

   YIN elements corresponding to extension keywords belong to the
   namespace of the YANG module where the extension keyword is declared
   via the "extension" statement.

   The names of all YIN elements MUST be properly qualified with their
   namespaces specified above using the standard mechanisms of [XML-
   NAMES], i.e. xmlns and xmlns:xxx attributes.

   The argument of a YANG statement is encoded in YIN either as an XML
   attribute or a subelement of the keyword element.  Table 1 defines
   the encoding for the set of core YANG keywords.  For extensions, the
   argument encoding is specified within the "extension" statement (see
   Section 7.17).  The following rules hold for arguments of both core
   and extension statements:

   o  If the argument is encoded as an attribute, this attribute has no
      namespace.

   o  If the argument is encoded as an element, it is placed to the same
      namespace as its parent keyword element.

   o  If the argument is encoded as an element, it MUST be the first
      child of the keyword element.

   Substatements of a YANG statement are encoded as (additional)
   children of the keyword element and their relative order MUST be the
   same as the order of substatements in YANG.

   Comments in YANG MAY be mapped to XML comments.

Comments?

/martin