Subject: [netmod] YANG 1.1: clarify nested submodule behavior

From: Andy Bierman <[email-masked]>
Date: Tue, 6 May 2014 07:46:04 -0700

Hi,

The question about submodules reminded me of our previous debate
on how nested submodules are treated.  I just did a test with
pyang and yangdump-pro.  They do not produce the same results.

   tinc2     <----  tinc1
                         |
                         + tinc1a
                         |
                         + tinc1b   <-- tinc1c

Problem: Need to clarify behavior when a submodule (tinc1c) is not
included by the main module.

pyang says this is an error and refuses to even compile tinc2.
I cannot find any text where it says this is an error.

andy@andy-swdev:~/modules$ pyang tinc1.yang
tinc1b.yang:4: error: submodule tinc1c is included by tinc1b, but not by
the module tinc1

andy@andy-swdev:~/modules$ pyang tinc2.yang
tinc1b.yang:4: error: submodule tinc1c is included by tinc1b, but not by
the module tinc1

yangdump-pro does report an error for using the tinc3a typedef in an
external module.  However it does not flag an error when that same type
is pulled through within a grouping.  The unexported submodule tinc1c is not
treated as an error.

andy@andy-swdev:~/modules$ yangdump-pro tinc1
*** /home/andy/modules/tinc1.yang
*** 0 Errors, 0 Warnings

andy@andy-swdev:~/modules$ yangdump-pro tinc2
Error: typedef definition for 'tinc1:type3' not found in module tinc1
tinc2.yang:19.12: error(250): definition not found

*** /home/andy/modules/tinc2.yang
*** 1 Errors, 0 Warnings

Solution:  Need to reach WG consensus on proper behavior and update
the appropriate text in YANG 1.1.

   * Is it an error to have an unexported submodule (IMO, no)

   * Is it an error to use an unexported definition (yes; seems to be
consensus
     on this point)

    * Is it an error to use an unexported definition if it is used in a
grouping
      that is exported?  (IMO, no)

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 May 2014 08:25:56 +0200 (CEST)

Hi,

Added as Y49.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 May 2014 08:29:53 +0200 (CEST)

Hi,

> pyang and yangdump-pro.  They do not produce the same results.

[...]

> the appropriate text in YANG 1.1.

I agree that this needs to be clarified.

This might not come as a surprise to you, but I prefer solution
Y49-02; make "unexported" submodules illegal.

IMO allowing "unexported" submodules would be really confusing.  Would
they solve any problem?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 02:03:01 -0700

> IMO allowing "unexported" submodules would be really confusing.  Would
The module may be under development and not all definitions are final yet.
There is not a single line of text in RFC 6020 that says it is an error to
have a nested submodule.  There is not 1 place in the RFC where it
an error to have an unused definition (e.g. nested unused groupings).

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 May 2014 12:10:29 +0200 (CEST)

> The module may be under development and not all definitions are final yet.

I think this needs to be handled by the toolchain (if at all).  The
rules we define for YANG modules applies to valid modules.  How you
got there doesn't matter.

> have a nested submodule.

I already agreed that we need to clarify this.  (But nested submodules
in general are allowed by 6020).

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 07 May 2014 16:51:48 +0200

Hi,

IMO, it would suffice to include all submodules in the main module and have the belongs-to statements in submodules. Definitions from every submodule would be available in the main module as well as in any other submodule.

I think that including a submodule from another submodule is pretty much useless because the main module and all submodules share the same namespace in any case.

Lada
  
Andy Bierman <[email-masked]> writes:

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 08:25:19 -0700

On Wed, May 7, 2014 at 7:51 AM, Ladislav Lhotka <[email-masked]> wrote:

> Hi,
>
> IMO, it would suffice to include all submodules in the main module and
> have the belongs-to statements in submodules. Definitions from every
> submodule would be available in the main module as well as in any other
> submodule.
>
> I think that including a submodule from another submodule is pretty much
> useless because the main module and all submodules share the same namespace
> in any case.
>
>
I think I like this idea (!)

It has always been my opinion that there is only 1 namespace within a
module.
Submodules provide organization, not namespace partitioning.  Include
statements
within submodules enforce partitioned namespace and lead to strange export
behavior.

But it is not backward-compatible to say include-stmt within a submodule is
not allowed.

We were here before.  You and Martin concluded that "include tinc1c" MUST
be present
in the main module and I said the module namespace is flat so 'type3' was
exported to tinc2
even if it was not included in the main module.

The fundamental issue is whether a module namespace is a hierarchy of
submodule namespaces
of whether the module namespace is flat and all submodules contribute to it.

Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 7 May 2014 17:30:58 +0200

On 07 May 2014, at 17:25, Andy Bierman <[email-masked]> wrote:

> It has always been my opinion that there is only 1 namespace within a module.
> Submodules provide organization, not namespace partitioning.  Include statements
> within submodules enforce partitioned namespace and lead to strange export behavior.
> 
> But it is not backward-compatible to say include-stmt within a submodule is not allowed.

It may remain allowed but not required for making definitions from other submodules (and the main module) available. I think this is a compatible change.

> 
> We were here before.  You and Martin concluded that "include tinc1c" MUST be present
> in the main module and I said the module namespace is flat so 'type3' was exported to tinc2

In my view, this is an additional complexity that buys you nothing.

Lada

> 
> The fundamental issue is whether a module namespace is a hierarchy of submodule namespaces

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 08:53:24 -0700

On Wed, May 7, 2014 at 8:30 AM, Ladislav Lhotka <[email-masked]> wrote:

> It may remain allowed but not required for making definitions from other
> submodules (and the main module) available. I think this is a compatible
> change.
>
> >
Does this mean we agree?  IMO the "belongs-to foo" statement is enough.
That says "my definitions are for namespace 'foo', and I can access all
definitions in namespace 'foo'.

That way we cannot have 2 definitions of the same thing in different
submodules,
or 1 in the main module and 1 in a submodule.

Why can't a submodule access definitions in the main module?
What problem does that solve?

IMO it is a lot simpler not to have any concept of submodule namespaces.

Lada

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 7 May 2014 18:00:49 +0200

On 07 May 2014, at 17:53, Andy Bierman <[email-masked]> wrote:

> Does this mean we agree?

I guess so. :-)

> IMO the "belongs-to foo" statement is enough.
> definitions in namespace 'foo’.

All submodules have to be included in the main module so that an importing module is aware of them.

> 
> That way we cannot have 2 definitions of the same thing in different submodules,

This is not possible anyway (IMO).

> What problem does that solve?

None, I think.
 
> IMO it is a lot simpler not to have any concept of submodule namespaces.

+1

Whenever namespace separation is needed, modules should be used.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 09:09:35 -0700

> submodules (and the main module) available. I think this is a compatible

OK.

I still do not agree that all include-stmts need to be in the main module.
IMO it is much more intuitive that they work like C #include (nested
#includes are pulled into the main module).

The value of nested submodules is the ability to sub-divide a sub-component.
The implementation of a submodule should not impact importers (same as in
C).
I think this is also allowed by the change rules in sec 10:

  release-1:

       foo <--- foo-inc

  release-2:

       foo <-- foo-inc   <--- foo-inc-a, foo-inc-b

In C, the foo source file does not need to be touched when
foo-inc is modified.  IMO YANG should work the same way.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 May 2014 18:11:21 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> > > IMO, it would suffice to include all submodules in the main module and have
> > > the belongs-to statements in submodules. Definitions from every submodule
> > > would be available in the main module as well as in any other submodule.
> > >

I am confused.   There are no additional namespaces today - just one,
introduced by the module.  What did you agree on?

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 7 May 2014 18:16:39 +0200

> introduced by the module.  What did you agree on?

That ‘include’ statements in a submodule are useless. Because of the shared namespace, different submodules cannot define nodes, identities, groupings and typedefs with colliding names, no matter whether they use include or not.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 7 May 2014 18:20:07 +0200

> #includes are pulled into the main module).

With no includes in the main module, how do you know which submodules are part of the data model? Searching the universe for all submodules that belong-to the main module doesn’t seem very effective.

Lada

> The implementation of a submodule should not impact importers (same as in C).

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 09:26:12 -0700

On Wed, May 7, 2014 at 9:11 AM, Martin Bjorklund <[email-masked]> wrote:

> > All submodules have to be included in the main module so that an
> importing
> > module is aware of them.
> >
> > >
There are 2 issues -- we agree on 1

 1) how to indicate that submodule contents are exported from the main
module
     to external modules?

    a) include must be in main module
    b) include must be in main module or submodule

 2) what definitions are visible within a submodule?

    a) only self and includes
    b) main module, self, and includes

IMO, belongs-to-stmt is enough to declare submodule contents as part of the
main module.
The parser needs to be able to find all submodule, and it can by following
the include-stmts.
An include-stmt does not need to be in the main module for a parser to find
it.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 09:31:02 -0700

On Wed, May 7, 2014 at 9:20 AM, Ladislav Lhotka <[email-masked]> wrote:

> > > I think that including a submodule from another submodule is pretty
> much useless because the main module and all submodules share the same
> namespace in any case.
> > >
> > >
> With no includes in the main module, how do you know which submodules are
> part of the data model? Searching the universe for all submodules that
> belong-to the main module doesn't seem very effective.
>
>

I did not say that.  Of course the module only contains submodules that are
explicitly included.  But a parser is required to follow the include chain.
The parser must process an include-stmt in a submodule (and detect loops).
Of course only the 'found' submodules are parsed.  (It is up to the designer
to make sure the include-chain is complete and no intentional orphan
submodules
exist in the universe ;-)

Lada
Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 7 May 2014 18:44:13 +0200

On 07 May 2014, at 18:31, Andy Bierman <[email-masked]> wrote:

> to make sure the include-chain is complete and no intentional orphan submodules

I am not sure what you actually propose. Could you summarise it? Which statements would be used and where?

FWIW, my proposal is to make the submodule logic extremely simple: only a single level of submodules and references in both directions via ‘include’ in the main module and ‘belongs-to’ in the submodule. Essentially, an ‘include foo;’ in the main module means the same as pasting the submodule contents in that place.

Subdividing a submodule is still possible, only the main module will have two includes instead of one.

Lada

---

From: Andy Bierman <[email-masked]>
Date: Wed, 7 May 2014 09:46:46 -0700

On Wed, May 7, 2014 at 9:16 AM, Ladislav Lhotka <[email-masked]> wrote:

> much
> That 'include' statements in a submodule are useless. Because of the
> shared namespace, different submodules cannot define nodes, identities,
> groupings and typedefs with colliding names, no matter whether they use
> include or not.
>
>
They are not useless, they are misused.
They are needed to find all the files that are part of the module namespace.
They can be nested and still serve this purpose.

I agree that include-stmt just gets in the way and makes YANG much less
useful
with current submodule restrictions.

   belongs-to foo { prefix foo; }

It is absolutely clear anywhere foo:some-def is used what is intended,
so there is no need to list all the accessed submodules.

Use of include-stmt to find files is mandatory, but use within a submodule
to access the module namespace should not be required.

A flat namespace would mean that including different versions of the same
submodule within the module namespace would be an error.  It is currently
possible with nested include-stmts (include by revision is just as awful
as import by revision).

Lada
Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 07 May 2014 22:21:20 +0200 (CEST)

>     b) main module, self, and includes

Thank you, this is a clear summary of the alternatives.

I think we were all over the map when we discussed this during the
initial design of YANG.  We looked at all these alternatives, and
ended up with what we have today.  I'll go back and look in the
archives to see how much of the discussions are captured.

One design rule we had was that we wanted it to be possible to compile
/ check each submodule on its own (only follow explicit
import/include), w/o having to bring in the complete module.  With
option 2b, this is no longer possible.

Re. issue 1, it seems to me no real problem is solved by going from 1a
to 1b.

Re. issue 2, solution 2b actually solves a real problem - today you
have to place common definitions in a separate submodule; they cannot
go into the main module.  Since circular includes are not possible,
references within a module (across submodules) have to be done
carefully, and might impose a non-intuitive submodule design.

So I think we need to think more about this, but solution 2b seems
attractive to me.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 8 May 2014 07:54:30 +0200

> archives to see how much of the discussions are captured.

I wasn’t there. :-)

> option 2b, this is no longer possible.

I don’t think this is important - the top-level definitions from the main module and all submodules are easy to look up. On the other hand, submodules cannot use conflicting names even if they don’t include each other, so it doesn’t help submodule writers in this respect. 

> Re. issue 1, it seems to me no real problem is solved by going from 1a

I agree. It would be different if a submodule tree could be grafted to a non-root node, but it is not the case.

> carefully, and might impose a non-intuitive submodule design.

Right, this is a serious contender for the most peculiar rule in YANG. 

> So I think we need to think more about this, but solution 2b seems

With 1a, the following is a logical option:

2c) main module and all submodules included from it.

Lada

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 08 May 2014 09:54:26 +0200 (CEST)

Ladislav Lhotka <[email-masked]> wrote:
> I don’t think this is important - the top-level definitions from the
> main module and all submodules are easy to look up. On the other hand,
> submodules cannot use conflicting names even if they don’t include
> each other, so it doesn’t help submodule writers in this respect.
> 
> > 

Yes, that's how I interpreted 2b!  Essentially each submodule has
access to all definitions in all other submodules.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 8 May 2014 19:50:02 -0700

On Thu, May 8, 2014 at 12:54 AM, Martin Bjorklund <[email-masked]> wrote:

> > I wasn't there. :-)
> > I don't think this is important - the top-level definitions from the
> > submodules cannot use conflicting names even if they don't include
> > each other, so it doesn't help submodule writers in this respect.
> >
> > >
This is what I wanted from the start.
That is why I do not agree that the include-stmts need to be in the main
module.

module A {
   prefix a;
   include B;
   container foo;
 }
 submodule B {
    belongs-to A { prefix a; }
    include C;
 }
 submodule C {
    belongs-to A { prefix a; }
    leaf top { type string; }
    augment /a:foo {
       leaf should-be-ok { type string; }
     }
 }

IMO, the include-stmts just say "go find this file and add all the
definitions to the module namespace". Therefore, leaf 'top'
is visible in the main module and also to external modules.
This is the detail we do not agree on.  Forcing the include-stmt
to also appear in the main module seems like a CLR.

It is not really OK to declare that submodules can use definitions
without include-stmts, unless we don't care about the self-compiling
sub-modules. (IMO a real low priority). If so, then it is also
OK to allow submodules to access the main module definitions.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 09 May 2014 11:54:33 +0200

> to also appear in the main module seems like a CLR.

The include statement would appear *only* in the main module. I think it is better because the main module is advertized but submodules aren't, so it's safer to get all submodules in one step and from an authoritative source. With includes in submodules it is IMO much more likely that the server and client will unintentionally work with different data models.  

> OK to allow submodules to access the main module definitions.

Since submodules aren't really isolated, I think the self-compilation is largely a red herring. A submodule defining a name that's in conflict with the main module will be sucessfully compiled even though it is broken.  

Lada

---

From: Andy Bierman <[email-masked]>
Date: Fri, 9 May 2014 06:11:57 -0700

On Fri, May 9, 2014 at 2:54 AM, Ladislav Lhotka <[email-masked]> wrote:

> The include statement would appear *only* in the main module. I think it
> is better because the main module is advertized but submodules aren't, so
> it's safer to get all submodules in one step and from an authoritative
> source. With includes in submodules it is IMO much more likely that the
> server and client will unintentionally work with different data models.
>
>
This is a CLR needed because you implemented it that way in pyang.
It does not follow the principle of self-compiling modules.
It doesn't matter where the include-stmt is wrt/ finding YANG files.

> Since submodules aren't really isolated, I think the self-compilation is
> largely a red herring. A submodule defining a name that's in conflict with
> the main module will be sucessfully compiled even though it is broken.
>
>
I agree compiling submodules is not a very useful feature since they can
never be used
that way in a server.

Andy