Subject: Re: [netmod] Fw: New Version Notification for draft-bjorklund-netmod-yang-xpath-extensions-00.txt

From: Jernej Tuljak <[email-masked]>
Date: Wed, 06 Nov 2013 11:09:32 +0100

Dne 25.10.2013 21:06, piše Ladislav Lhotka:
> On 25 Oct 2013, at 19:24, Martin Bjorklund <[email-masked]> wrote:
>
>> Ladislav Lhotka <[email-masked]> wrote:
>>> On 25 Oct 2013, at 17:01, Martin Bjorklund <[email-masked]> wrote:
>>>
>>>> Ladislav Lhotka <[email-masked]> wrote:
>>>>> If the server doesn't advertise the "ietf-yang-xpath-extensions" module, can
>>>>> other advertized modules still use new XPath functions in "must" and "when",
>>>>> if
>>>>> they import that module?
>>>> Yes.  Note that all XPath expressions are conceptual.  If a server
>>>> claims that it implements a module that uses such an xpath function,
>>>> it must implement the contraint somehow.  It doesn't have to use an
>>>> XPath evaluator; it can be implemented in code - just like ane
>>>> constraints defined in description statements.
>>> Yes, but one has to be careful with revisions - sec. 10 in RFC 6020 allows new
>>> extensions in an updated module, but I think in this case it could be dangerous
>>> because the server cannot easily ignore unknown functions appearing in XPath
>>> expressions.
>> You typcially do not just load a new module into a server; it needs
>> some instrumentation to be useful.  So a new revison that uses some
>> new XPath functions or not needs an implemention effort in the
>> server.
>>
>> But maybe I misunderstood your concern?
> Sec. 6.3.1 says:
>
> If a YANG compiler does not support a particular extension, which appears in a YANG module as an unknown-statement (see Section 12), the entire unknown-statement MAY be ignored by the compiler.
>
> So my point here is that while ignoring extensions is easy because they are self-contained, it is not the case for extensions in XPath expressions. But maybe it is not an issue.
>
> Lada
>

It is an issue. I share Ladislav's concerns and add my own. I do not 
think YANG extensions should be used for extending the value space of an 
existing YANG statement's argument (not this way). Extensions should 
only be allowed to extend YANG statement structure, thus providing new 
YANG statements with special meaning, which may or may not be utilized 
in YANG client code. The purpose of the "extension" statement is to 
define a keyword (7.17.). A must/when statement argument is a string, 
not a keyword. 7.17. also never mentions extensions being used/utilized 
in any way other than as _YANG statements_. In my view, importing a 
module that contains an extension's definition is not enough to 
"use/utilize" an extension. A YANG unknown statement is also required.

While I agree that additional XPath functions would greatly benefit YANG 
language, I do not like the way this draft introduces them for usage. I 
think that the proper way of introducing them would be to simply create 
yangxp:must and yangxp:when extensions, alternatives to RFC6020 
statements, which would allow extended XPath syntax in their arguments.

IMO, all extensions should be self-contained, meaning no side-effects. 
Extending (augmenting actually) arbitrary statement arguments is a 
side-effect - you have to change valid YANG syntax into something else 
in order to use it. And use of an unsupported extension should under no 
circumstances result in compiler error, but it clearly does with this 
draft (unknown XPath function).

You seem to be saying that as a modeler, I can now supposedly declare 
support for an extension by simply importing a module and using a 
specific character sequence inside a standard YANG statement argument 
(string), yet I'm not even at liberty to make a decision about that. 
That is an implementor's job, later in the development cycle.

An actual problematic modeling use case:
1. you are using a YANG development IDE and have just defined 
ietf-yang-xpath-extensions module
2. you create a new module that imports ietf-yang-xpath-extensions
3. you create a must statement which uses an extended XPath function 
from ietf-yang-xpath-extensions
4. you expect the IDE's compiler to be quiet about the unknown function?

I'd say no. It should raise hell because the offending must statement is 
clearly not YANG, nor is it an extension. Note that this case is 
applicable to any extension that would presume it is okay to distort an 
existing statement's argument value space. I could, for example, augment 
a revision statement's argument with some extra chars, and then expect 
compilers to keep quiet. I could also make namespace statement's 
argument not comply to URI syntax and expect it to be okay. 
Semantically, there's no difference between doing either of those and 
what this draft is attempting to do. That is, legalizing _deviations of 
standard YANG syntax_ in a way that is undefined from a RFC6020 
compiler's perspective. Forget server instrumentation code. This is a 
language problem and has nothing to do with implementors or server-side. 
Are you allowing YANG description statements, children of an extension 
definition statement, to govern argument syntax of existing standard 
YANG statements, by them merely being present somewhere in the module 
chain? That is the real question here and I'd like to see RFC6020 text 
that allows this to be considered as something that is actually plausible.

XPath expressions in YANG may be conceptual but are currently well 
defined. I urge you to keep them as such. What the current draft is 
doing requires YANG 2.0, IMHO. The must statement in section 2.1 of the 
draft is not YANG, since it's argument does not comply to RFC6020 
requirements. It doesn't matter if there's another document, which 
claims it is.

Jernej

>>
>> /martin
> --
> Ladislav Lhotka, CZ.NIC Labs
> PGP Key ID: E74E8C0C
>
>
>
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: Andy Bierman <[email-masked]>
Date: Wed, 6 Nov 2013 06:33:49 -0800

On Wed, Nov 6, 2013 at 2:09 AM, Jernej Tuljak <[email-masked]>wrote:

>> If a YANG compiler does not support a particular extension, which appears
>> in a YANG module as an unknown-statement (see Section 12), the entire
>> unknown-statement MAY be ignored by the compiler.
>>
>> So my point here is that while ignoring extensions is easy because they
>> are self-contained, it is not the case for extensions in XPath expressions.
>> But maybe it is not an issue.
>>
>> Lada
>>
>>
> It is an issue. I share Ladislav's concerns and add my own. I do not think
> YANG extensions should be used for extending the value space of an existing
> YANG statement's argument (not this way). Extensions should only be allowed
> to extend YANG statement structure, thus providing new YANG statements with
> special meaning, which may or may not be utilized in YANG client code. The
> purpose of the "extension" statement is to define a keyword (7.17.). A
> must/when statement argument is a string, not a keyword. 7.17. also never
> mentions extensions being used/utilized in any way other than as _YANG
> statements_. In my view, importing a module that contains an extension's
> definition is not enough to "use/utilize" an extension. A YANG unknown
> statement is also required.
>
> yangxp:must and yangxp:when extensions, alternatives to RFC6020 statements,
> which would allow extended XPath syntax in their arguments.
>
> side-effect - you have to change valid YANG syntax into something else in
> order to use it. And use of an unsupported extension should under no
> circumstances result in compiler error, but it clearly does with this draft
> (unknown XPath function).
>
> support for an extension by simply importing a module and using a specific
> character sequence inside a standard YANG statement argument (string), yet
> I'm not even at liberty to make a decision about that. That is an
> implementor's job, later in the development cycle.
>
> 3. you create a must statement which uses an extended XPath function from
> ietf-yang-xpath-extensions
> clearly not YANG, nor is it an extension. Note that this case is applicable
> to any extension that would presume it is okay to distort an existing
> statement's argument value space. I could, for example, augment a revision
> statement's argument with some extra chars, and then expect compilers to
> keep quiet. I could also make namespace statement's argument not comply to
> URI syntax and expect it to be okay. Semantically, there's no difference
> between doing either of those and what this draft is attempting to do. That
> is, legalizing _deviations of standard YANG syntax_ in a way that is
> undefined from a RFC6020 compiler's perspective. Forget server
> instrumentation code. This is a language problem and has nothing to do with
> implementors or server-side. Are you allowing YANG description statements,
> children of an extension definition statement, to govern argument syntax of
> existing standard YANG statements, by them merely being present somewhere
> in the module chain? That is the real question here and I'd like to see
> RFC6020 text that allows this to be considered as something that is
> actually plausible.
>
>

Can you verify that this is a real problem in specific tool chains?
AFAIK, yangdump is the only offline validation tool that completely checks
must/when
statements. SInce XPath in groupings has a few corner-cases that would fool
the compiler,
full validation of XPath can only occur on "cooked" YANG (all
uses/refine/deviations resolved).
Even if yangdump found an unknown function it would be a warning not an
error.

>From my interpretation of XPath 1.0, the function library and variable set
could be
injected at run-time, so the yuma code allows for that.
Servers can be smart enough to deal with this new extension, even with
automation tools.
It requires a significant development effort, but it's just another
internal callback API
loaded with the server instrumentation library for the YANG module.

I agree this use of a YANG extension pushes the limits and could appear to
violate
the rule about MAY skip over unknown statements, but I think Martin's
solution
is the best way to add this feature to YANG 1.0.  A few of us have laundry
lists
of improvements we would like to add to YANG.  We could easily spend 3
years working
on YANG 2.0. IMO we need to get some content out there.  We need
other WGs developing YANG modules (like ACL, topology).

Jernej
>>> /martin

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 06 Nov 2013 08:39:57 -0800 (PST)

Hi,

Jernej Tuljak <[email-masked]> wrote:
> > If a YANG compiler does not support a particular extension, which
> > appears in a YANG module as an unknown-statement (see Section 12), the
> > entire unknown-statement MAY be ignored by the compiler.
> >
> > So my point here is that while ignoring extensions is easy because
> > they are self-contained, it is not the case for extensions in XPath
> > expressions. But maybe it is not an issue.
> >
> > Lada
> >
> 
> think YANG extensions should be used for extending the value space of
> an existing YANG statement's argument (not this way).

Please note that this draft updates 6020, see section "XPath
Evaluation Context".  Without this section you are right - extensions
cannot arbitrarily make changes like this.  This is why a new RFC is
needed.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 06 Nov 2013 11:31:12 -0800

> AFAIK, yangdump is the only offline validation tool that completely checks

DSDL mapping, as implemented in pyang, does it, too.

> injected at run-time, so the yuma code allows for that.

No, this is not possible, and RFC 6020 is very clear in this sense. Some tools do allow for extensions, and support some like EXSLT. While it is possible to write an extension to xsltproc, it is not the version that people have installed on their computers. So this discussion also shows the division line between YANG tools that are home-baked, and those that use standard XML tools.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 06 Nov 2013 11:36:28 -0800 (PST)

Hi,

Ladislav Lhotka <[email-masked]> wrote:
> So this
> discussion also shows the division line between YANG tools that are
> home-baked, and those that use standard XML tools.

XPath clearly allows custom functions.  We should not limit ourselves
just b/c there exist some tools that don't support this.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 6 Nov 2013 13:34:32 -0800

> XPath clearly allows custom functions.  We should not limit ourselves

Strictly speaking, XPath 1.0 only knows the core function library and offers no provisions for defining extensions. You can define other functions, as XSLT does, but it is no more XPath.

> just b/c there exist some tools that don't support this.

Well, libxslt and xsltproc do support extensions, but if you patch them, you lose the benefit of relying on ubiquitous tools that everybody already has pre-installed. Of course, it is not an issue if you only distribute statically linked binary blobs.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 6 Nov 2013 13:41:57 -0800

> cannot arbitrarily make changes like this.  This is why a new RFC is

… and a new YANG version number, too.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 06 Nov 2013 13:53:01 -0800 (PST)

Ladislav Lhotka <[email-masked]> wrote:
> Strictly speaking, XPath 1.0 only knows the core function library and
> offers no provisions for defining extensions. You can define other
> functions, as XSLT does, but it is no more XPath.

No this is not correct.  The XPath 1.0 spec says that an expression is
evaluated in a context, and in the context there is a function
library.  Then the spec defines the core library, which every
implementation must support.  The spec does not say that there is a
fixed set of functions.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 6 Nov 2013 14:03:23 -0800

> implementation must support.  The spec does not say that there is a

OK, you are right.

Lada

---

From: Jernej Tuljak <[email-masked]>
Date: Thu, 07 Nov 2013 09:36:05 +0100

> just b/c there exist some tools that don't support this.

But RFC6020 doesn't support it. A decision about that has already been 
made. You need YANG 2.0 to change that. What XPath 1.0 spec says about 
this matter is irrelevant, IMO. RFC6020 implies a subset of it is being 
used for YANG. You can't just ignore this. You can't argue that a 
RFC6020 tool that reports an error, when a statement argument contains 
unknown XPath functions, is non-standard.

Take a look at how XSLT specs handle this. There's a section that deals 
with extension functions specifically. It even defines when the XSLT 
processor should treat unknown functions as an error and when not. Why 
doesn't RFC6020 have such a section? That's a major problem. And I'm not 
even mentioning how extension statements need to be abused in order for 
this whole thing to work. XSLT specs also contain a section for 
extension elements, which is how I understand YANG extension statements.
http://www.w3.org/TR/xslt#extension

It is unfortunate, but even another RFC can't change this. Or can it? 
I'm not familiar with how this works inside IETF. Can you just publish a 
new RFC which updates a section in an existing RFC? Isn't that more 
suited for errata or something else that inherently obsoletes the 
existing section?

Jernej

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 7 Nov 2013 13:54:40 +0100

On Thu, Nov 07, 2013 at 09:36:05AM +0100, Jernej Tuljak wrote:
> 
> It is unfortunate, but even another RFC can't change this. Or can
> it? I'm not familiar with how this works inside IETF. Can you just
> publish a new RFC which updates a section in an existing RFC? Isn't
> that more suited for errata or something else that inherently
> obsoletes the existing section?
> 

A new RFC can update sections in existing RFCs. An implementation of
the new RFC will claim to be compliant to the OLD RFC and the NEW RFC
while an implementation that implements the OLD RFC will only claim to
be compliant to the OLD RFC. So from the procedural side, this is all
fine.

The WG in charge must of course consider whether any interoperability
issues may arise from such an update. This is I think the discussion
we are having here and there is no generic answer.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 7 Nov 2013 05:43:56 -0800

Hi,

I think that extensions must really obey the rule stated in sec. 6.3.1 of RFC 6020, namely that “the entire [extension] statement MAY be ignored". This it not the case for extensions that have side effects on core YANG statements, and “xpath-function” in particular.

This leads me back to what I proposed in Berlin: 6020 needs to be opened and a limited set of updates applied, which would result in a new spec and a new YANG version.

The set of changes of course requires some discussion. A useful criterion might be that the changes must not make any existing module invalid.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 7 Nov 2013 07:41:14 -0800

Hi,

I do not have any objection to dropping this work because it
cannot be added as an extension.   I am strongly opposed
to starting any work on a new version of YANG.

There is no such thing as a minor update to a DML.
Introducing a new language version while many projects still
depend on the current version causes market confusion, increases
support costs and reduces adoption and deployment.  It should
only be done if the new features and problems fixed are worth the cost.
These XPath extensions are not that important.

IMO, NETMOD should finish up the current YANG modules and
then shut down for several years.  Let other WGs create data models.
The IETF needs more experience with YANG before knowing what
really needs to be in YANG 2.0.

Andy

On Thu, Nov 7, 2013 at 5:43 AM, Ladislav Lhotka <[email-masked]> wrote:

> Hi,
>
> I think that extensions must really obey the rule stated in sec. 6.3.1 of
> RFC 6020, namely that “the entire [extension] statement MAY be ignored".
> This it not the case for extensions that have side effects on core YANG
> statements, and “xpath-function” in particular.
>
> This leads me back to what I proposed in Berlin: 6020 needs to be opened
> and a limited set of updates applied, which would result in a new spec and
> a new YANG version.
>
> The set of changes of course requires some discussion. A useful criterion
> might be that the changes must not make any existing module invalid.
>
> Lada
>
> On 07 Nov 2013, at 04:54, Juergen Schoenwaelder <
> [email-masked]> wrote:
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 7 Nov 2013 10:29:59 -0800

> There is no such thing as a minor update to a DML.

Why not?

> These XPath extensions are not that important.

IMO (a) it is rather important (e.g. derived-from), and (b) there are other things that need fixing.

> really needs to be in YANG 2.0.

With (hopefully) many YANG modules around, it will become much harder to change anything in YANG than it is now.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Thu, 7 Nov 2013 10:58:54 -0800

> > There is no such thing as a minor update to a DML.

Because the cost of deployment is so high.

> other things that need fixing.

I don't think this is a big problem.
The purpose of YANG is officially the DML for the NETCONF protocol.
Within NETCONF the must/when statements are conceptual -- an XPath parser
is not required to implement a YANG module in the server. Support for
off-line message validation tools is not the focus of NETCONF.

The concern with identityrefs is that they are really QNames but they are
encoded
in XPath as strings.  This is only a problem if the server actually
supports multiple
identities with the same base with the same local-name.  We could easily
publish
a convention for encoding this QName as a string -- e.g., same as your JSON
--
use the module name as the prefix.

> change anything in YANG than it is now.
IMO changing the DML version should only be done as a last resort.
YANG 1.0 is good enough for WGs and vendors to use now.

Andy

> > I think that extensions must really obey the rule stated in sec. 6.3.1
> of RFC 6020, namely that “the entire [extension] statement MAY be ignored".
> > The set of changes of course requires some discussion. A useful
> criterion might be that the changes must not make any existing module
> invalid.
> >
> > Lada
> >

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 7 Nov 2013 11:28:06 -0800

> Because the cost of deployment is so high.

You agreed with introducing the XPath functions in a sneaky way via extensions. Would the cost of deployment be any lower in this case? Changing the version number only means that we are honest with the users.

> off-line message validation tools is not the focus of NETCONF.

No, the expressions must be valid XPath and must be evaluated exactly as an XPath processor would do.  

> 
> The concern with identityrefs is that they are really QNames but they are encoded
> in XPath as strings.  This is only a problem if the server actually supports multiple
> identities with the same base with the same local-name.  We could easily publish
> a convention for encoding this QName as a string -- e.g., same as your JSON --

Sure, we could do that, but only in a YANG 2 (or 1.1) spec. RFC 6020 defines an encoding that is based on XML context and prefixes, and this is where an equality test may fail. This is where things can break in subtle and unexpected ways, especially if it happens in a “when” statement.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 07 Nov 2013 12:27:16 -0800 (PST)

Ladislav Lhotka <[email-masked]> wrote:
> Sure, we could do that, but only in a YANG 2 (or 1.1) spec. RFC 6020
> defines an encoding that is based on XML context and prefixes, and
> this is where an equality test may fail.

I think we should keep the prefixed name.  The problem is an omission
in 6020 - it specifies that the prefixed name is ued in the "default"
statement, but it should say "default", "when" and "must".

Anyway, this only solves the equality test, not the 'derived-from'.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 7 Nov 2013 12:44:44 -0800

On Thu, Nov 7, 2013 at 11:28 AM, Ladislav Lhotka <[email-masked]> wrote:

> You agreed with introducing the XPath functions in a sneaky way via
> extensions. Would the cost of deployment be any lower in this case?
> Changing the version number only means that we are honest with the users.
>
>

I agree with Phil's comment about the must/when statements being conceptual
within the server.  If the server says it supports a YANG module that uses
a new
XPath function, then that is just 1 more implementation detail for the
server.

The machine-readable XPath could be replaced by description-stmt text.
Both are just implementation requirements for a server develop to implement.

> an XPath processor would do.

It is valid XPath.  There are no requirements at all for the server to
evaluate any XPath.
The server code must implement the correct behavior no matter what.  The
client cannot
tell if the server used an XPath parser or not.

> defines an encoding that is based on XML context and prefixes, and this is
> where an equality test may fail. This is where things can break in subtle
> and unexpected ways, especially if it happens in a “when” statement.

must/when statements use YANG prefixes, not XML prefixes.

I have no confidence that a "quick" YANG 1.1 can be done.
I think that IETF charter milestones are a cruel joke and taking 3 years to
complete an 8 month project is unfortunately the norm, not the exception.
The use of extensions is much better because it only affects the YANG
modules
that use the new XPath functions.  A server vendor can choose to implement
the
new module or not. A client that needs to support the new module may need to
change.  If the YANG version changes the client must change to even parse
the YANG module.

Andy

> > > This leads me back to what I proposed in Berlin: 6020 needs to be
> opened and a limited set of updates applied, which would result in a new
> spec and a new YANG version.
> > >

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 7 Nov 2013 13:05:07 -0800

On 07 Nov 2013, at 12:27, Martin Bjorklund <[email-masked]> wrote:

> statement, but it should say "default", "when" and "must”.

It says “… identity name in the default value MAY have a prefix.” (sec 9.10.3), and the preceding paragraph defines the default namespace if no prefix is used. So if a prefix is on one side of ‘=‘ and no prefix on the other side, the quality test will fail.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 7 Nov 2013 13:26:56 -0800

On 07 Nov 2013, at 12:44, Andy Bierman <[email-masked]> wrote:

> within the server.  If the server says it supports a YANG module that uses a new
> XPath function, then that is just 1 more implementation detail for the server.

But it will be the same even if the version number is bumped.

> Both are just implementation requirements for a server develop to implement.

So why do we bother with the XPath stuff?

container universe {
    description
        “Dear implementor, remove me with all my content if ../foo = 42.”;
…
}

> It is valid XPath.  There are no requirements at all for the server to evaluate any XPath.
> The server code must implement the correct behavior no matter what.  The client cannot

Exactly, so identities must be compared as strings, because this is what an XPath processor does.

> must/when statements use YANG prefixes, not XML prefixes.

"An identityref is encoded as the referred identity's qualified name as defined in [XML-NAMES].”

> The use of extensions is much better because it only affects the YANG modules
> that use the new XPath functions.  A server vendor can choose to implement the
> change.  If the YANG version changes the client must change to even parse the YANG module.

I don’t see how an extension differs from a function definition that is a part of YANG spec. What really matters is whether new functions appear in XPath expressions or not.

Lada