Subject: Notes on draft-crocker-mast-analysis-01.txt

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 27 Oct 2003 21:59:15 -0600

My notes from a quick look-through of the MAST-ANALYSIS draft are
attached.

Although I suggest plenty of changes, I'm very pleased with the
structure and contents of the document. I'm rearranging in many cases.

Dave, thanks for all your hard work. Sorry it's taking me so long to
read drafts these days!

Spencer

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 28 Oct 2003 23:16:41 -0600

Dear Dave,

A couple of (what I hope are) clarifications, but the e-mail is
getting an order of magnitude shorter on each cycle...

Spencer

----- Original Message ----- 
From: "Dave Crocker" <[email-masked]>
To: "Spencer Dawkins" <[email-masked]>
Cc: "Multi6 Mailing List" <[email-masked]>
Sent: Tuesday, October 28, 2003 3:45 PM
Subject: Re: Notes on draft-crocker-mast-analysis-01.txt

> SD>      This paper reviews the basic requirements for support of
> SD>      multiaddressing (mobility and multihoming), and the efforts
to
> SD>      support them. Barriers to adoption, administrative
overhead, and
> SD>      operational efficiency are of particular concern. In
addition,
> SD>
> SD> S: You know you want to say here that these are the low-tech
obstacles
> SD> that will cripple proposals no matter how technically brilliant.
Go ahead
> SD> and say it!
>
> I think the issue is more serious and substantive than that.
>
> Adoption and Administration hassles are fundamental concerns for
everything we
> do in the IETF, and I think they frequently get too little
attention. I see
> them as co-equal factors in IETF success. In fact, neither of them
is simple,
> minor, low-tech, or any other condescension, though it is easy to
think as if
> they were... Until they later pop up and bite us.

Could not agree more.

> SD>           Host Identity Protocol [HIP] is used to establish a
rapid
> SD>           authentication between two hosts and to provide
continuity
> SD>           of communications between those hosts independent of
the
> SD>           networking layer. The [LIN6] protocol defines a layer
that
> SD>           supports multiple locators, between IPv6 and
transport.
> SD>           Multiple Address Service for Transport [MAST] supports
> SD>           association of multiple IP Addresses during the life
of any
> SD>           transport instantiation, by defining a layer between
IP and
> SD>           transport. It operates only in the endpoints and works
with
> SD>           IPv4 and IPv6.
> SD>
> SD> "...These proposals differ in services provided but all operate
only in
> SD> endpoint hosts, with no new infrastructure required." Is this
actually
> SD> true?
>
> hmmm.  I certainly thought so.  I know it is true for MAST.  LIN6
and HIP do
> specify a third-party assistant service, so maybe they are closer to
mIP4?

Mumble. It seemed like there were various flavors of
"infrastructure" -

- A home agent (in the MIP6 sense),
- A foreign agent (in the MIP4 sense, or
- Changes to the routing infrastructure,

each being a higher hurdle than the previous flavor.

>
>
> SD> S: Dave, the point you're making about mobility morphing into
multihoming and vice versa
> SD> is too important to hide in the definitions section, spread over
the next three definitions.
>
> Ummmm.  ok.  Any suggestions for text or where/how to do that?

Maybe in its own section, immediately following the terminology
section ("An Aside on Multihoming and Mobility")? You already have the
text, strewn through the last half of each definition! It just needs
to be a lot more obviously visible.

>
>
> SD> S: This term stinks, if you're marketing to TSV guys.
> SD> They already know what Path MTU Discovery is, and it's nothing
like this.
> SD> My suggestion would be something like Interface Discovery,
except that
> SD> you also allowed the "I have one interface, but I can see both
of my
> SD> network homing points from where I am" case.
> SD>
> SD>      Path discovery      provides a sender with the means for
learning
> SD>                          about the locators from which they can
send.
>
> 1. I did not choose the terminology framework.  Some of the other
proposals
> used the term 'path'.  In fact I originally disliked it, because it
trod on
> the IP-level routing world.  On further thought, I came to like the
reference
> quite a lot, because it helps us to try to re-use routing-world
ideas.
>
> 2. I think that 'discovery' and 'selection' should be based on the
same core
> term.
>
> I notice that you suggest 'interface' -- which I dislike -- and then
point out
> the problem of using it. That is, I notice you do not suggest an
alternative.
> Unfortunately, I'm not having any success coming up with a new one.

"Locator Discovery"? Duh, sorry, I was apparently an idiot to not
suggest this in the first place.

>
>
> SD> S: Not sure what to say, except that Rendezvous needs help!
> SD>
> SD>      Rendezvous          refers to one endpoint making contact
with an
> SD>                          explicitly identified   other endpoint.
>
> Feel free to elaborate...

I kept typing and deleting... But I wasn't sure what seemed to need
help. At the very least, "rendezvous" conjures up the notion of two
endpoints making contact with each other, for me, anyway.

>
> SD>      The difference between mobility prior to initial contact
and
> SD>      mobility during an existing association is significant.  In
the
> SD>      latter case, the mobile host can use the association state
when
> SD>      needing to inform the other endpoint about the change.
Prior to
> SD>      an association -- or when both endpoints are mutually
mobile --
> SD>      an independent referral service is required.
> SD>
> SD> S: You know, the document hasn't really distinguished between a
mobile endpoint with one locator
> SD> and a mobile endpoint
> SD> with two locators having different characteristics (GPRS and
WLAN, for instance). Does it make any
> SD> difference if you
> SD> need to communicate movement when your single interface
stabilizes, vs. communicate movement over
> SD> your "other, stable"
> SD> interface?
>
> This strikes me as a very important and rather dangerous question.
>
> I'm a fan of having stuff above IP know nothing about the attributes
of the
> lower layer.  I see this as one of the very, very big wins in the
design and
> use of OP.
>
> It's often important to know about actual performance, but I think
we get
> distracted by classing these in terms of media. So, yes, knowing
that one path
> has ping times of days (IP over avian carrier) and that another
seems to have
> infinite bandwidth (giga-foo) is useful. Knowing GPRS vs. WLAN vs.
gigafoo)
> strikes me as the wrong way to label and use these differences.

OK, you understood what I meant to say - it's not that it's GPRS and
WLAN and gigafoo, it's that it's two or three locators with
dramantically different attributes. I spent some time on X.25-ish
Multi Link Protocol (MLP) in the late 80s - would love to see TCP
running over three gig-E interfaces, but not if it's sensitive to
out-of-order delivery. The GPRS/WLAN deal was basically to say that if
you have two interfaces, and one is an order of magnitude higher
bandwidth than the other, it's tempting to just forget the slower one,
but life doesn't have to be that way.

At my last two jobs, I had 100BASE-T at my desk and 802.11 wandering
around the building - again, an order of magnitude difference, and
easier to simply ignore 802.11 when I was at my desk, but it didn't
have to be that way.

>
>
> SD>      3.2.2.    Site
> SD>
> SD>      For networks with multiple attachments to a backbone,
external
> SD>      routing technology already permits propagation of alternate
> SD>      routing information. However it does not make these
alternatives
> SD>      visible to endpoints.
> SD>
> SD>      Further a domain name may have multiple locator records
that
> SD>      point to the same network. However there is no indication
whether
> SD>      the same records are, instead, pointing to different,
redundant
> SD>      systems; on the other hand the importance of this ambiguity
is
> SD>      not clear.
> SD>
> SD> S: It SHOULDN'T matter, because (since if you ask for
www.yahoo.com
> SD> and get nine IP addresses, there's not much telling you how to
pick one)
> SD> theoretically the "different" systems are equivalent. Anycast is
the
> SD> extreme case. But if you're using a stateful application, this
> SD> equivalence starts to unravel.
>
> Some offline discussions are leading to my seeing this as needing
two
> different identifiers.
>
> One is for initial rendezvous.  The initiator is selecting from an
> undifferentiated set of locators to use.  Whether they go do the
same endpoint
> or different ones is none of the initiator's business.
>
> After there is context, the requirements are different.  The
initiator _must_
> get back to the same endpoint.  In my current thinking, the endpoint
therefore
> needs a different identifier than the one that may have referred to
a _set_ of
> endpoints.
>
> So, this is a good example of the significant difference between
initial
> rendezvous and rehoming rendezvous.
>
>
> SD>      3.2.3.    Endpoint
> SD>
> SD>      What is notably missing is a means for an existing
association to
> SD>      directly use multiple paths, in particular when the paths
> SD>
> SD> S: This is slightly confused. It seems to me that you're saying
something like "An existing
> SD> association between two
> SD> Internet endpoints can easily use multiple paths, as long as
neither endpoint terminates the last
> SD> hop of
> SD> more than one path." I think.
>
> Sorry, but I am not understanding the text you are offering.

(I'm shocked, NOT.) What I was trying to say was that an existing
association can easily use multiple paths, via routing, as long as
each endpoint has only one (active?) locator, so neither endpoint sees
path changes on its first hop link(s). If one of the endpoints has
more than one locator, we have the problem you're trying to analyse in
this draft.

>
>
> SD>      Having a public EID means that a new, global registration
service
> SD>      must be developed and operated.  Some believe network
operators
> SD>      will not mind this additional work; others disagree.
> SD>
> SD> S: The extent to which such additional work is attractive
probably has a lot to do with the extent
> SD> to which the result
> SD> looks like/does not look like per-host routing at the EID level.
This is actually one of the things
> SD> I wonder about in
> SD> some multiaddressing proposals...
> SD>
> SD>      Having an EID in every datagram means that the string must
be as
> SD>      short as possible.  Even then it will add significant
overhead to
> SD>      datagram header size.  However given the need to process
> SD>      multiaddressing, having the EID in every datagram probably
will
> SD>      not alter datagram processing overhead, in the endpoints,
from
> SD>      any other approach to using EIDs.
>
> In my experience, operators do not care about abstract
architectures. They
> care about the effort to administer and operate it. Extra
administration
> sucks. Extra components suck (unless there is a strong,
counterbalancing
> reduction.)

Per-host routing tables, without hierarchy/aggregation, suck. Sorry
for being unclear.
>
>
> SD>      4.1.1.    Dynamic DNS
> SD>
> SD>      For maintaining target availability during an association,
DNS
> SD>      dynamic update [DNSDYN] is a plausible mechanism to obtain
new
> SD>      locator information.  Although this would probably not be
helpful
> SD>      for transitions during an association, it could suffice for
> ...
> SD> S: Isn't there also an ugly dependence on a non-deployed PKI, if
you
> SD> want DNS updates from individual endpoints?
>
> Not sure what change you are suggesting to the text, or whether it
is
> essential for this discussion.  Please clarify.

If you think Joe Random Host on the Internet will be updating the DNS
dynamically, you're relying on widely-deployed PKI, aren't you? If so,
I'm not sure DNSDYN is all that plausible. Sorry for being vaguee.

> SD>      4.5.2.    Host Identity Protocol (HIP)
> SD>      HIP works with IPv4 and IPv6.  Also, it:
> ...
> SD>           *    Creates a new DNS RR entry
> SD>
> SD> S: Do you understand how many RR entries are required for
general deployment? I'm still trying to
> SD> understand the answer.
>
> I have assumed one per target.  No?

That Would Be Unfortunate. I understand that no single DNS needs RRs
for every conceivable endpoint, but this still sounds like a lot of
RRs being added to the DNS.

>
>
> SD>      4.5.4.    MAST
> SD>
> SD>      MAST is a control protocol for the exchange of IP Addresses
> SD>      notification and authorization, to use additional IP
Addresses in
> SD>      a given host-pair context.
> SD>
> SD> S: The following bullet list seems oddly formatted, not sure
what it's saying...
>
> It's two lists.  The first has one bullet.  Yes, that's odd, but the
other
> bullets got dropped during revisions.  Besides, the idea is to match
the
> other, bulleted lists in this section.

It just confused me - it didn't seem possible that you would have a
one-item bullet list...

>
> SD>      [TCP-MH]  Matsumoto, A. Kozuka, M., Fujikawa, K., Okabe,
> SD>                Y., "TCP Multi-Home Options", draft-arifumi-tcp-
> SD>                mh-00.txt, 10 Sep 2003
> SD>
> SD> S: The actual date for tcp-mh is 7 Oct - there was a bogus
.tx.txt draft submitted previously.
>
> Good grief.  That level of careful, detailed reading is really
scary,
> Spencer...

Heh, heh, heh...

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 30 Oct 2003 17:04:10 +0100

On 29 okt 2003, at 6:16, Spencer Dawkins wrote:

> I spent some time on X.25-ish
> out-of-order delivery.

What is sensitive to out of order delivery? The GE interfaces or TCP?

As soon as you start using multiple links in parallel, you're going to 
see out of order packets, that's pretty much guaranteed. (Consider the 
case where an application sends 1800 bytes worth of data, so a 1500 
byte packet is transmitted over link A and a 300 byte packet over link 
B. The second packet will get to the other side first because it's 
shorter.)

If TCP has problems with out of order packets, then it hasn't been 
implemented terribly well.

I have no doubt that as soon as we have mechanisms for multiaddress 
multihoming, we'll see people trying to load share across the different 
addresses. This could be very useful.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 30 Oct 2003 16:16:37 -0800

Iljitsch,

IvB> What is sensitive to out of order delivery? The GE interfaces or TCP?

TCP and IP.  Having packets arrive out of order hurts fast-path performance.
the inherent requirement is to take a packet and put it into a holding
position until it becomes 'in order'.  The required checking and holding and
retrieving hurt performance.

IvB> If TCP has problems with out of order packets, then it hasn't been
IvB> implemented terribly well.

If you know of a TCP implementation that does not take a performance hit with
out-of-order packets, please cite the data

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: "Spencer Dawkins" <[email-masked]>
Date: Thu, 30 Oct 2003 21:38:14 -0600

Hi, Iljitsch,

What Dave said, at least re: TCP.

If I send a segment that arrives out of order at the recipient, the
recipient decides that the "hole" may be a lost packet and begins
sending duplicate ACKs every time it receives a segment, until the
hole is filled in. When the sender receives the third duplicate ACK,
it says "oops, must be a lost packet", performs Fast Retransmit, and
then cuts its congestion window in half (Fast Recovery).

This does not make TCP run faster, since increases are additive (one
additional segment per RTT, building back up to a full congestion
window) while decreases are multiplicative (if you cut the congestion
window in half, have a couple of good RTTs, and then have another
out-of-order Fast Retransmit/Fast Recovery, the sender cuts the
congestion window in half AGAIN).

If your out-of-order-delivery problem is only one or two packets
before the hole is filled in, that's OK, but in the general case,
deployed TCPs don't handle much out-of-order-delivery before you start
seeing performance hits.

There are experimental proposals to make the Fast Retransmit/Fast
Recovery threshold adaptable, but they are not standardized or
deployed.

Spencer

p.s. Mark Allman - am I still telling the truth?

----- Original Message ----- 
From: "Dave Crocker" <[email-masked]>
To: "Iljitsch van Beijnum" <[email-masked]>
Cc: "Spencer Dawkins" <[email-masked]>; "Multi6 Mailing List"
<[email-masked]>
Sent: Thursday, October 30, 2003 6:16 PM
Subject: Re: Notes on draft-crocker-mast-analysis-01.txt

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 31 Oct 2003 10:55:17 +0100

> then cuts its congestion window in half (Fast Recovery).

Yes, this is all in RFC 2001. Note that in the case of two identical 
pipes being used side by side, you're not going to see that that many 
successive duplicate acks because generally only one packet will pass 
another. Obviously with gigabit ethernet alongside with GPRS things are 
going to be somewhat different.

My point is that there are no fundamental properties that make it 
impossible for TCP to perform well in the presence of reordering. For 
instance, if additional memcopies would be required, that would be a 
big problem. But the design of the TCP/IP family of protocols and the 
way our kernels work is such that there must always be a copy cycle 
anyway, so no problems there.

In my experience, the measures taken to avoid reordering have been much 
more harmful than reordering itself ever was to begin with.

Implementations just need to change a bit to avoid unnecessary fast 
retransmits if they want to support using parallel links. I don't think 
this is a huge deal.

One thing that bothers me about TCP and that also doesn't do us any 
favors here is that it sends packet trains back-to-back all the time. 
This can at times be very harmful as it unnecessarily increases 
burstiness. Especially cheap switches that must convert between 
different ethernet speeds don't have the buffering to handle this 
properly.