Subject: [netmod] What does unknown-statement2 represent in YANG grammar?

From: Jernej Tuljak <[email-masked]>
Date: Wed, 15 May 2013 11:37:47 +0200

Hi,

I'm confused about the unknown-statement2 production in section 12. What 
is the purpose the non-prefixed version of unknown statements in this 
grammar?

This production is never mentioned in the rest of the document directly. 
I think it's supposed to enable usage of standard YANG statements in 
unknown statement (usages of statements defined using extension 
statement) subtrees. But.

7.17. The extension statement

    The substatements of an extension
    are defined by the extension, using some mechanism outside the scope
    of this specification.  Syntactically, the substatements MUST be YANG
    statements, or also defined using "extension" statements.  YANG
    statements in extensions MUST follow the syntactical rules in
    Section 12  <https://tools.ietf.org/html/rfc6020#section-12>.

I don't understand.  If I use a prefix statement inside an unknown 
statement subtree and it doesn't conform to the prefix-stmt rule of the 
grammar, this text makes it valid YANG, since my prefix conforms to 
unknown-statement2 which is also a "syntactical rule in Section 12". It 
is actually the only applicable one.

If the intention was to enable usage of standard YANG statements inside 
subtrees of unknown statements, then this production should contain an 
(alternation)* of all possible standard YANG statements plus 
unknown-statement (or at least a comment about it). I find 
unknown-statement2 redundant. It also unnecessarily complicates 
implementation of lexers/parsers.

BTW, why doesn't the document (RFC6020) use two different words for 
extensions and extension usages? Everything is just an extension which 
at least I find confusing. Consistent usage of ["extension" statement] 
and [extension] would also suffice:

    The "extension" statement allows the definition of new statements
    within the YANG language.  This new statement definition can be
    imported and used by other modules.

    The statement's argument is an identifier that is the new keyword for
    the extension and must be followed by a block of substatements that
    holds detailed extension information.  The purpose of the "extension"
    statement is to define a keyword, so that it can be imported and used
    by other modules.

    The extension can be used like a normal YANG statement, with the
    statement name followed by an argument if one is defined by the
    "extension" statement, and an optional block of substatements.  The statement's
    name is created by combining the prefix of the module in which the
    extension was defined, a colon (":"), and the extension's keyword,
    with no interleaving whitespace.  The substatements of an extension
    are defined by the "extension" statement, using some mechanism outside the scope
    of this specification.  Syntactically, the substatements MUST be YANG
    statements, or also extensions.  YANG statements in extensions MUST follow the
    syntactical rules inSection 12  <https://tools.ietf.org/html/rfc6020#section-12>.

Jernej

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 15 May 2013 12:22:28 +0200 (CEST)

Hi,

Jernej Tuljak <[email-masked]> wrote:
> Hi,
> 
> I'm confused about the unknown-statement2 production in section
> 12. What is the purpose the non-prefixed version of unknown statements
> in this grammar?
> 
> This production is never mentioned in the rest of the document
> directly. I think it's supposed to enable usage of standard YANG
> statements in unknown statement (usages of statements defined using
> extension statement) subtrees.

Yes, this is the intention.

> But.
> statement subtree and it doesn't conform to the prefix-stmt rule of
> the grammar, this text makes it valid YANG, since my prefix conforms
> to unknown-statement2 which is also a "syntactical rule in Section
> 12". It is actually the only applicable one.
> 
> If the intention was to enable usage of standard YANG statements
> inside subtrees of unknown statements, then this production should
> contain an (alternation)* of all possible standard YANG statements
> plus unknown-statement (or at least a comment about it).

Yes, I think an alternation should work (and would be useful).

> I find

Sure, it should be consistent.  I don't know where you think it is not
consistent.

/martin

>    syntactical rules inSection 12
>    <https://tools.ietf.org/html/rfc6020#section-12>.
> 
> Jernej

---

From: Jernej Tuljak <[email-masked]>
Date: Wed, 15 May 2013 12:31:19 +0200

>>     <https://tools.ietf.org/html/rfc6020#section-12>.

This text contains fixes where I think inconsistencies exist. All of 
them are in the third paragraph (first, third and fourth sentence).

Jernej

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 May 2013 07:12:00 -0700

>> Yes, this is the intention.

This is a CLR and has unintended consequences.
By definition, an external statement is outside the
YANG language.  Compilers are only required to
properly skip over them.

E.g., if a leaf-stmt is embedded in an external statement,
then it is not a YANG leaf definition.  It is not required
to have a 'type' sub-statement present.  It is not required
to parse and validate to a unique and valid leaf definition.
Only leaf-stmts within the YANG language have these requirements.

Andy

>>>     Section 12  <https://tools.ietf.org/html/**rfc6020#section-12<https://tools.ietf.org/html/rfc6020#section-12>
>>> >.
>>>
>>>     are defined by the "extension" statement, using some mechanism
>>> outside
>>>     the scope
>>>     <https://tools.ietf.org/html/**rfc6020#section-12<https://tools.ietf.org/html/rfc6020#section-12>
>>> >.
>>>
>>
> This text contains fixes where I think inconsistencies exist. All of them
> are in the third paragraph (first, third and fourth sentence).
>
> Jernej
>
>
>>> Jernej
>>>
>>
> ______________________________**_________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/netmod<https://www.ietf.org/mailman/listinfo/netmod>
>

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 15 May 2013 16:22:02 +0200 (CEST)

> YANG language.  Compilers are only required to

All extension must be properly defined.

> then it is not a YANG leaf definition.

Yes it is.

> It is not required
> to have a 'type' sub-statement present.

Yes it is required to have 'type'.

The idea is that you are free to use core YANG statements if and only
if they have the exact same syntax as normal.  If you want to have
different syntax in your 'leaf' statement you would have to define an
extension for it.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 May 2013 07:55:52 -0700

On Wed, May 15, 2013 at 7:22 AM, Martin Bjorklund <[email-masked]> wrote:

> > On Wed, May 15, 2013 at 3:31 AM, Jernej Tuljak <[email-masked]
> >wrote:
> >
RFC 6020, sec 6.3.1, para 3:

   If a YANG compiler does not support a particular extension, which
   appears in a YANG module as an unknown-statement (see Section 12),
   the entire unknown-statement MAY be ignored by the compiler.

IMO, the ABNF production in section 12 is correct because of this paragraph.
The ABNF and meaning of the word "ignored" are clear.  A compiler MAY
skip over extensions it does not support.  This simply requires finding
the end of a well-structured YANG statement, as defined in sec 6.3:

 statement = keyword [argument] (";" / "{" *statement "}")

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 15 May 2013 17:21:39 +0200

> All extension must be properly defined.

Hmm… An extension extends the YANG language, so if it was to be properly defined, such a definition should be placed in the ABNF. The "extension" statement only serves documentation purposes, and also YIN translation - but the latter could be easily done without it, using simply the same attribute name for all extension arguments, e.g. "arg".

Lada

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 May 2013 08:39:18 -0700

On Wed, May 15, 2013 at 8:21 AM, Ladislav Lhotka <[email-masked]> wrote:

> Hmm… An extension extends the YANG language, so if it was to be properly
> defined, such a definition should be placed in the ABNF. The "extension"
> statement only serves documentation purposes, and also YIN translation -
> but the latter could be easily done without it, using simply the same
> attribute name for all extension arguments, e.g. "arg".
>
>
The extension (unknown) statement is in the ABNF.
It is correctly defined.  The text "MAY be ignored..."
was put in the RFC for a reason.  A compliant implementation
is allowed to ignore the entire external statement.   By "ignore"
I mean treat it as a comment -- it has no semantic meaning to
the compiler at all and therefore no knowledge of the sub-statements
that are even allowed within the external statement.

The 'arg' sub-statement is the "argment" part of the ABNF,
not the nested sub-statements.

Lada
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 15 May 2013 18:05:19 +0200

> The extension (unknown) statement is in the ABNF.

Yes, but this will also match statements that are not declared with the "extension" statement. For a compiler or any software, the "extension" statement provides little added value.

> that are even allowed within the external statement.

Yes, by way of analogy, RELAX NG validators simply ignore any elements of a foreign namespace.

>  statement = keyword [argument] (";" / "{" *statement "}")

What I meant was that for any extension like

foo:bar "123";

we could always use

<foo:bar arg="123"/>

in YIN syntax.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 May 2013 09:36:49 -0700

...

> Yes, but this will also match statements that are not declared with the
> "extension" statement. For a compiler or any software, the "extension"
> statement provides little added value.
>
>
So you are proposing that the ABNF be changed so unknown-statement2 is a
choice between its current value and all possible YANG statements?
So if any arbitrary YANG statement is used out of context, it MUST
be used correctly?

There are no rules wrt/ the proper context in which an unknown-statement is
used
or the sub-statements that are allowed and have meaning within the
unknown-statement.

> Yes, by way of analogy, RELAX NG validators simply ignore any elements of

And they probably ignore all nested elements within foreign elements.
That is what the YANG ABNF says now, and it should stay that way.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 May 2013 09:50:20 -0700

Hi,

Here is an example of why its better not to define clever little rules
for YANG statements.  Let's say I want an extension that says
what font color to print keywords in a YANG HTML-ify program.
Since the extension argument can only be 1 unstructured string,
nested statements are used to identify the keywords:

extension font-color {
   argument color;
   description
      "Specifies the font color to be used for the keywords
       identified as sub-statement identifiers within this extension.

           acme:font-color red {
               key;
               type;
               revision;
           }
      ";
  }

IMO we should not make rules about YANG sub-statements when they
are used in a foreign context.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 15 May 2013 19:03:37 +0200

> are used in a foreign context.

I agree. IMO, the unknown-statement production could allow any content instead of "*unknown-statement2".
This would make the parsing much simpler and extensions more flexible, without causing any harm to generic YANG software.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 15 May 2013 19:21:30 +0200 (CEST)

> So if any arbitrary YANG statement is used out of context, it MUST

Yes.  This is what the current text in 7.17 tried to say:

   Syntactically, the substatements MUST be YANG
   statements, or also defined using "extension" statements.

The first idea was to require that all substatements to an extension
were also extensions.  But that would mean if you define some
extension that need all container / leaf / list ... they would have to
copy almost all core YANG stmts.  (for example, suppose we add a new
"alarm" keyword in an extension module, and an alarm has data nodes as
substmts.)

So we said that it is ok to use core YANG statements, as long as they
are used the way they are defined.  For a reader, it is much easier if
a statement you know always follow the same rules, instead of having
context sensitve syntax.

/martin

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 16 May 2013 08:46:04 +0200

> a statement you know always follow the same rules, instead of having

IMHO, this might be useful an is definitely easier to implement than 
what Andy suggests. Also, this way all possible statements have an 
implicit or explicit definition of their basic structure (keyword and 
argument at the very least) without having to rely on context.

Jernej

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 16 May 2013 09:27:47 +0200

> This would make the parsing much simpler and extensions more flexible, without causing any harm to generic YANG software.

I disagree. What's wrong with defining another acme extension, 
acme:yang-statements with a "keywords" argument instead of using YANG 
standard statements directly?

extension yang-statements {
argument keywords;
description "Specifies a space separated list of YANG statement keywords 
and may be used in various contexts";
}

acme:yang-statements "key type revision";

Much more readable (subjective) and well defined (fact). In Andy's 
example "key", "type" and "revision" are extensions and can only be 
extensions. Who provides their definitions? How does the compiler know 
that they are not missing arguments? Does acme:font-color provide this 
information for them? Who provides the same information for their 
substatements?

I think it is essential that each possible YANG statement including ALL 
extensions have their base structure defined with YANG syntax or in some 
other well known way (like RFC text). Possible substatements of 
extensions and their cardinalities might be out of scope of RFC6020, but 
not their signatures (keyword and argument). They shouldn't be.

Jernej

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 16 May 2013 09:39:17 +0200 (CEST)

Jernej Tuljak <[email-masked]> wrote:
> > This would make the parsing much simpler and extensions more flexible,
> > without causing any harm to generic YANG software.
> 
> acme:yang-statements with a "keywords" argument instead of using YANG standard
> statements directly?
> 
> description "Specifies a space separated list of YANG statement keywords and
> may be used in various contexts";
> }
> 
> Much more readable (subjective) and well defined (fact). In Andy's example
> "key", "type" and "revision" are extensions and can only be extensions. Who
> provides their definitions? How does the compiler know that they are not
> missing arguments? Does acme:font-color provide this information for them? Who
> provides the same information for their substatements?
> 
> extensions have their base structure defined with YANG syntax or in some other
> well known way (like RFC text). Possible substatements of extensions and their
> cardinalities might be out of scope of RFC6020, but not their signatures
> (keyword and argument). They shouldn't be.

+1

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 09:47:17 +0200

On May 16, 2013, at 9:27 AM, Jernej Tuljak <[email-masked]> wrote:

> I disagree. What's wrong with defining another acme extension, acme:yang-statements with a "keywords" argument instead of using YANG standard statements directly?
> 
> description "Specifies a space separated list of YANG statement keywords and may be used in various contexts";
> }

How is this declaration different from Andy's example with font color?

Lada

> Much more readable (subjective) and well defined (fact). In Andy's example "key", "type" and "revision" are extensions and can only be extensions. Who provides their definitions? How does the compiler know that they are not missing arguments? Does acme:font-color provide this information for them? Who provides the same information for their substatements?
> 
> I think it is essential that each possible YANG statement including ALL extensions have their base structure defined with YANG syntax or in some other well known way (like RFC text). Possible substatements of extensions and their cardinalities might be out of scope of RFC6020, but not their signatures (keyword and argument). They shouldn't be.
> 
> Jernej
> 
>> 
>> Lada
>> 
>>> 
>>> Andy
>>> 
>>> 
>> --

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 16 May 2013 10:00:47 +0200

> How is this declaration different from Andy's example with font color?

It implies nothing about the signature of key, type and revision 
extensions. Andy's example does.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 10:26:02 +0200

On May 16, 2013, at 10:00 AM, Jernej Tuljak <[email-masked]> wrote:

> It implies nothing about the signature of key, type and revision extensions. Andy's example does.

Sorry, I don't understand. If a compiler, or any generic YANG software, sees

acme:font-color red { … }

it will just ignore it (treat it as a comment), no matter what's inside the braces (*). Software that uses this extension can parse the contents inside braces using its own rules, which need not necessarily be the same as YANG's. I think it is pretty much the same as "anyxml" in instance documents.

(*) The only problem for a YANG parser would be if there are unbalanced braces inside { … }.

Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 16 May 2013 10:34:49 +0200

> (*) The only problem for a YANG parser would be if there are unbalanced braces inside { … }.

What if the compiler supports acme:font-color and it doesn't support key 
as an extension? How do you even declare support of such extensions if 
you don't have it's declaration to refer to? What if you have two 
"extension" statements and extensions of both may contain key, type and 
revision as substatements (out of scope), yet you only support them in 
one of the two extensions?

Jernej

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 16 May 2013 10:40:39 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> it will just ignore it (treat it as a comment), no matter what's inside the
> braces (*). Software that uses this extension can parse the contents inside
> braces using its own rules, which need not necessarily be the same
> as YANG's.

This is not how it works.  6020 says:

   The substatements of an extension are defined by the extension,
   using some mechanism outside the scope of this specification.
   Syntactically, the substatements MUST be YANG statements, or also
   defined using "extension" statements.  YANG statements in
   extensions MUST follow the syntactical rules in Section 12

A compiler that doesn't understand the extensions can still validate
that the extension is defined, and that the argument is there if the
extension needs one.

/martin

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 16 May 2013 10:44:17 +0200

> that the extension is defined, and that the argument is there if the

Exactly. It actually should do so. If an extension requires an argument, 
but it is missing, it is a syntax error, even if the extension is not 
supported by the compiler.

Jernej

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 11:31:47 +0200

>   extensions MUST follow the syntactical rules in Section 12

I know, but we are discussing whether this CLR was necessary.

BTW, I wonder if it is only syntactical rules: Is the following allowed or not?

  grouping bar {
    container top {
      acme:foo {
        grouping bar {
          leaf bottom {
            type empty;
          }
        }
      }
    }
  }

(the grouping inside the extension shadows the outer one).

> that the extension is defined, and that the argument is there if the

But "some mechanism outside the scope of this specification" still has to be used to check the argument value and substatements, so it can also catch the missing argument error. The "extension" statement only guards against typos in the extension name.

In RELAX NG, foreign-namespace annotations work fine without any declaration, though it is true that they rely on the notion of well-formed XML.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 11:35:22 +0200

On May 16, 2013, at 10:34 AM, Jernej Tuljak <[email-masked]> wrote:

> What if the compiler supports acme:font-color and it doesn't support key as an extension? How do you even declare support of such extensions if you don't have it's declaration to refer to? What if you have two "extension" statements and extensions of both may contain key, type and revision as substatements (out of scope), yet you only support them in one of the two extensions?

If I write software that supports acme:font-color, I'd use a special grammar defining the exact syntax of its contents. Then, if acme:font-color is encountered, the software will switch to a special parsing mode, so it is irrelevant whether the contents breaks any YANG rules.

But it is also true that the same can be achieved by embedding the foreign contents in the argument. Andy's extension could be used as follows:

acme:font-color "name: red; key; type; revision";

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 May 2013 03:55:55 -0700

.....

>    extensions MUST follow the syntactical rules in Section 12
6020 also says:

  If a YANG compiler does not support a particular extension, which
  appears in a YANG module as an unknown-statement (see Section 12),
  *the entire unknown-statement *MAY be ignored by the compiler.

> that the extension is defined, and that the argument is there if the

Notice the words "the entire unknown-statement".
This clearly includes sub-statements within the entire unknown-statement.

Since any YANG sub-statements present do not actually define any
YANG semantics I fail to see the value in checking the syntax completely.

    extension: An extension attaches non-YANG semantics to statements.
      The extension statement defines new statements to express these
      semantics.

Since the entire extension is clearly non-YANG semantics what is the
point of checking the completeness of the syntax for semantics that do not
apply in the foreign context?  Why is "type-stmt" mandatory for "leaf-stmt"?
Because when defining a real leaf, it is needed.  Used in an arbitrary
foreign context, maybe the 'type' is not needed and therefore not mandatory.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 16 May 2013 13:02:04 +0200 (CEST)

> This clearly includes sub-statements within the entire unknown-statement.

Yes.  But even if the compiler ignores them they have to follow the
syntactic rules of 'unknown-statement', and Lada suggested that they
didn't have to.

> foreign context, maybe the 'type' is not needed and therefore not mandatory.

Then you shouldn't piggy-back on the YANG statement 'leaf', but define
your own extension 'andy:leaf' instead.

You might think that this is too restrictive, but I think it is a
feature.  Also, my experience with the restrictive syntax in 6020 is
that it works pretty well.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 13:49:33 +0200

> that it works pretty well.

OK, let's have some fun after lunch:

$ cat acme.yang
module acme {
  namespace "http://example.com/acme";
  prefix "acme";
  extension foo;
  acme:foo {
    identity bar;
    identity baz {
      base bar;
    }
  }
}

$ pyang acme.yang
acme.yang:8: error: identity "bar" not found in module acme

So pyang checks the definition of identity "baz" but ignores the definition of "bar". 

IMO, this "restrictive" approach would need a whole bunch of other CLRs in order to be sound and well-defined.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 16 May 2013 13:55:29 +0200 (CEST)

> So pyang checks the definition of identity "baz" but ignores the definition of

Bug in pyang then.   It shouldn't check the "baz" identity.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 May 2013 14:17:12 +0200

> Bug in pyang then.   It shouldn't check the "baz" identity.

OK, next one:

  acme:foo {
    extension bar;
    acme:bar;
  }

It is exactly as Andy said: inside an extension, we have to give up the standard semantics of YANG statements, so it makes very little sense to insist on their syntax.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 16 May 2013 14:34:20 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> >> So pyang checks the definition of identity "baz" but ignores the definition
> >> of
> >> "bar".
> > 

I don't see the problem.  If 'bar' is not defined as an extension in
the acme module, this is an error.  If 'bar' is defined, it is fine.

> It is exactly as Andy said: inside an extension, we have to give up the
> standard semantics of YANG statements, so it makes very little sense to insist
> on their syntax.

Let's agree to disagree maybe.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 May 2013 09:08:06 -0700

...

> > So pyang checks the definition of identity "baz" but ignores the
> definition of
> > "bar".
>
Technically, identity-stmt is only allowed as a body-stmt and not allowed
within any other context, so there are no identities actually defined.

module acme {

yangdump-pro generates an error:

Warning: no revision statements for module 'acme'
Error: Local module extension 'bar' not found
acme.yang:13.5: error(250): definition not found

The compiler does not ignore the contents.
It follows the ABNF and checks that each nested statement
is either YANG or a known extension.  Missing right brace
would be an error as well.

IMO, parsing the generic 'statement' structure is all
that is required of a compiler.  The extension-stmt is
capable of defining a generic leaf (type empty if no argument
and type string if argument present).  All the disputed mechanics
are actually defined by the statement separator, not the
extension-stmt.

Andy