Subject: about draft-nordmark-multi6-noid-00

From: "marcelo bagnulo" <[email-masked]>
Date: Sat, 25 Oct 2003 19:39:49 +0200

Hi Erik,

After an initial reading of the noid draft, i have a couple of questions:

- Are you assuming that the multi-homed site runs bgp with each of its
direct providers and that it obtains a full bgp feed from each of them?

- I understand that ingress filtering compatibility is guaranteed by source
address rewriting, is that ok? if so, how do handle ingress filtering issues
when sending initial packets whose source address cannot be rewritten?
(rewrite ok bit not set)

Thanks, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Sat, 25 Oct 2003 23:19:32 +0200 (CEST)

> direct providers and that it obtains a full bgp feed from each of them?

Not required.
You need routing to work for all of the sites locators.
And for border router rewriting of the source locator to be a useful way
to detect the working path the site's border routers need to have enough
information to tell which outbound path to use. Depending on the failures
you are concerned about this could be as simple as the border routers
being able to tell whether the link to the ISP is up or down, or it
could depend on the site receiving each ISPs routing table to see which
prefixes appear to be reachable over which ISP/link.

I don't think even in the latter case it is necessary to run BGP; if the
ISP could redistribute a its view of reachable prefixes using a separate
instance of an IGP running across the border that should suffice.
(Not that I know if configuring that is much simpler than configuring BGP).

But there is no need for the site to advertise anything to its ISPs; each
ISP only need to know which prefix it has delegated to the site.

> when sending initial packets whose source address cannot be rewritten?

I don't think ingress filtering can be made strictly better any time soon
even if we adopt some multihoming proposal; if an attacker would want to
exploit holes in IPv6 ingress filtering the attacker could just use
non-multihoming packets (e.g., TCP over IPv6 as currently defined).

Having said that, if the multihoming protocol performs explicit
initiator-driven state creation (draft-nordmark-multi6-sim-00.txt is an
example) instead of having a data packet cause responder state creation
actions as in noid, then one can make the state creation work with "rewrite
ok" enabled.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Sun, 26 Oct 2003 12:12:17 +0100

Let's consider the following scenario.
Let's suppose that all ISPs are performing ingress filtering
Also that hosts inside mh sites have assigned one prefix per provider

            +----+
        ----|ISPA|_             +----+
       /    +----+ \_+------+  _|ISPC|_
 +----+              |      |_/ +----+ \__+----+
 | mh1|             _|      |            _| mh2|
 +----+     +----+_/ |      |_          / +----+
       \____|ISPB|   +------+ \_+----+_/
            +----+              |ISPD|
                                +----+

There are two different situations to be considered when addressing locator
reachability issues: initial contact and preservation of the established
communications.

i.) Initial contact

i.i) Initiator's perspective

i.i.i) No failure has occurred

Suppose that mh1 initiates a communication with mh2, so mh1 selects PA:mh1
as initial source address i.e. initial locator and AID1, and also selects
PC:mh2 as destination address and AID2.
The initial packet must be sent with the "rewrite ok" bit not set.
Suppose now that the internal routing of the site mh1 is such that packets
sent to PC:: are forwarded through ISPB. This means that the packet
generated by mh1 described above will be discarded by ingress filtering in
ISPB because its source address is incompatible and cannot not be rewritten.
The question is then, how do you deal with this case? (a tunnel between exit
routers, so that the packet is forwarded through the appropriate isp, an
icmp back to the initiator?)

i.i.ii) A failure has occurred in the link between ISPB and its upstream
provider.

Suppose now that the packet generated by mh1 has PB:mh1 as source address
and PC:mh2 as destination address.
Now, since the internal routing system has information about prefix
reachability through the different ISPs (as you explained, mh1 obtains such
information through some routing protocol), the internal routing system
knows that it has to forward the packet through ISPA, since there is no
route available through ISPB. However, the actual packet has PB:mh1 as
source address, making the packet incompatible with ingress filtering. The
packet then will be dropped. Note that retrying from the application running
in mh1 with alternative address is not good since the problem is not related
to the destination address but it is related to the source address. So, how
do you address this issue? (a new ICMP message back to the initiator host?)

i.i.iii) A failure occurred in the link between ISPC and mh2

mh1 sends a packet to destination address PC:mh2 and any of the source
addresses
PC:mh2 is not reachable and this is not visible by the routing system.
Since it is the initial packet, no feedback from the source address
rewriting mechanism can be obtained.
I guess the only option here is to wait a timeout and let the application at
the initiator to retry. My point is that the routing system will not be able
to deal with all the failure scenarios.

i.ii) Target perspective

Question: the first reply that the target sends back to the initiator, is
the "rewrite ok" bit set or not?

ii) Preserving established communications

Suppose that there is communication established between mh1 and mh2.
Suppose that all the locators are working and that all of them were verified
so they acceptable to be used as valid locators in the communication.
Currently the communication is using PA:mh1 and PC:mh2 as locators.
Suddenly a failure in the link between ISPA and its upstream provider
occurs.
The idea then is that ISPA detects the situation and then it announces the
event to mh1. The internal routing system reconverges and packets are no
longer routed through ISPA but they are routed through ISPB. Since the
"rewrite ok" would be set at this moment of the communication, then packets
that are originated by mh1 with PA:mh1 as source address are rewritten by
the exit router so that PB:mh1 is included as source address. Now mh2 learns
that it has to use PB:mh1 as destination address because this is the address
contained in the packets that it is receiving. Is that ok?
Now my question is how long does this mechanism takes to react? i.e. the
response time. In order to preserve established communications, the
interruption shouldn't be too long. The problem here is that intradomain
routing is pretty slow detecting outages. Suppose that ISPA is running BGP
with its upstream provider, then if i am correct, the value recommended in
the BGP spec to detect an outage between peers is 90 seconds. This means
that the interruption in the communication will be longer than 90 seconds.
IMHO this is not good enough to preserve established communications. Am i
missing something?

My point is that the usage of the routing system to determine which locator
to use and to preserve established communication is not a good option
because:
- It is pretty expensive, since you need to inject the reachability of all
prefixes from the ISPs to the multi-homed site. This may be ok for medium to
big sites, but i am not so sure in the case of a very small site with CATV
and ADSL
- Some failures have to be handled by the host itself, since not all
failures are visible from the routing system because of aggregation issues.
see point i.i.iii
- The main drawback is that the time required by the routing system to
detect outages is simply incompatible with preserving established
communications. This IMHO is a fundamental incompatibility since it is
caused by design choices. I mean, the intra-domain routing system prefers a
bit outdated but stable view of the topology rather than an very accurate
but unstable view of the topology. this implies that the time required by
the routing system to accept a change in the topology will be long, at least
longer that what it is acceptable to preserve an established communication.

It is then my conclusion that path outage detection and locator selection
has to be performed by the end-host themselves and not by the routing
system.

Sorry if this was too long, but thanks for reading so far.

Regards, marcelo

> -----Mensaje original-----
> De: [email-masked] [mailto:[email-masked]]En
> nombre de Erik Nordmark
> Enviado el: sabado, 25 de octubre de 2003 23:20
> Para: [email-masked]
> CC: [email-masked]; [email-masked]
> Asunto: Re: about draft-nordmark-multi6-noid-00
>
>

---

From: masataka ohta <[email-masked]>
Date: Sun, 26 Oct 2003 22:16:55 +0900

Erik;

Red herring.

Encapsulation and notification formats are a merely minor issue
of multihoming.

An essential problem of multiaddress multihoming is when to
try alternative addresses, which involves transport/application
layers, as the IP layer has no notion of time (save TTL limitation
of 255 of IPv4).

However, your draft says that

>  The solution does not
>  introduce a "stack name" type of identifier, instead it ensures that
>  all upper layer protocols can operate unmodified in a multihomed
>  setting while still seeing a stable IPv6 address.

it does not address the problem and is, like Dave's MAST,
totally meaningless.

It should also be noted that changes on host identification
(from IP address to something else such as FQDN) means a
protocol change at upper (at least at the transport) layers
that "all upper layer protocols can operate unmodified" is
a false statement.

							Masataka Ohta

PS

It is really a wast of bandwidth to read poor proposals not
understanding requirements described in my drafts long ago.

Do read the drafts.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 26 Oct 2003 23:05:16 +0100

> ISPB because its source address is incompatible and cannot not be 

There are solutions possible that allow rewriting all packets. However, 
IPv6 as we know it today does not. So we need to solve this problem 
anyway. I think source address dependent routing makes the most sense 
here.

> the internal routing system
> packet then will be dropped.

When an ISP fails, it is of course extremely important to have traffic 
rerouted over the other one. Today, we do this using BGP. We can still 
do that, but it has the disadvantage that packets for every destination 
only flow over one ISP. If that ISP is incapable of delivering the 
packets, there is trouble (although this should be rare if we do BGP 
rather than just a default). Less important, but still annoying: we 
have to depend on BGP's limited best path selection capabilities.

If we do source address dependent routing on the other hand, a host can 
utilize both ISP links for the same destination at the same time. Since 
the destination address in the packet is then no longer relevant, we 
don't need to run BGP either.

Of course now the multihoming system in the host must detect 
unreachability and initiate a rehoming event when there is a problem in 
the path over one ISP.

> The internal routing system reconverges and packets are no
> the exit router so that PB:mh1 is included as source address. Now mh2 
> learns that it has to use PB:mh1 as destination address because this 
> is the address contained in the packets that it is receiving. Is that 
> ok?

The source address used by the other side is a good hint when there is 
no real preference as to which destination address for the other side 
is preferred. However, each side must be able to jump destination 
addresses as it is possible that failures occur somewhere along the way 
where neither side knows that something is wrong so there is no change 
of source address.

Also, there is the case of asymmetric reachability, where there are two 
links that only work in one direction so packets from A to B must use 
different addresses than packets from B to A. This can happen with 
radio links, or a weaker case where there is still symmetric 
connectivity but the asymmetric connectivity is of a higher bandwidth, 
is common with one way satellite links.

> interruption shouldn't be too long.

We probably want to monitor transport layer events to see whether 
rehoming is needed. This could be pretty fast in most cases.

> The problem here is that intradomain routing is pretty slow detecting 
> outages.

Yes, the worst case is pretty pathetic. Fortunately, the worst case 
isn't the most common one.

> Suppose that ISPA is running BGP
> the BGP spec to detect an outage between peers is 90 seconds.

Correct, but Cisco in their infinite wisdom doubled this to 180 
seconds. But you can set this much lower. I used to recommend 15 
seconds but some BGP implementations are too braindead to work well 
with that. Not many problems with 30 seconds, though.

Note that in many cases, the sessions will go down much faster because 
BGP monitors the interface the session runs over.

> This means that the interruption in the communication will be longer 
> than 90 seconds. IMHO this is not good enough to preserve established 
> communications. Am i missing something?

Isn't the TCP timeout 240 seconds? Sessions should survive, not sure if 
the user is this patient, though.

> to use and to preserve established communication is not a good option

I agree with your points here.

> has to be performed by the end-host themselves and not by the routing

Indeed.

Iljitsch

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 00:58:43 +0100 (CET)

> icmp back to the initiator?)

One could deal with it any number of ways; same as what has already been
discussed by Christian and others.

I think the easiest way to *avoid* it is to, instead of the setup being
triggered by a data packet as in NOID, have an explicit setup from
the sender (as in SIM) (with ULP packet piggybacked on the setup if need be).
That way the context creation packets can carry the AIDs in the payload
and allow router rewriting of the locators.
In this case of NOID one would have to think carefully about the verification
issues in such a case (since the AID could be different than the source
locator).

> the "rewrite ok" bit set or not?

For data packets "rewrite ok" can not be set until the context confirm has
been sent.
The context request (which might be the first packet the target sends back)
I think rewrite ok can be set.

> contained in the packets that it is receiving. Is that ok?

ok

> response time. 

There are two factors: how quickly the site border routers detect 
that a path through an ISP isn't working, and how quickly after that
the peer is aware of the change.
The first factor is independent of any multihoming solution AFAIK; has
to do with stuff like how routes are propagated, routing protocol hello
timers,  etc. 
In NOID the second factor depends on the ULP; if the ULP is sending frequently
enough the peer will notice quite quickly. If the ULP is patiently sitting
waiting for the peer to transmit and not sending anything then the peer
never notices anything using this mechanism and instead has to rely on
ULP hints about retransmissions.
I think this is also rather independent of the multihoming approach;
locator rewriting provides an efficient mechanism when it works.
The only alternative I know is to have all hosts continuosly "ping"
eachother every 10 or so seconds, which is rather inefficient.

> The problem here is that intradomain

The routing system seems to be good enough for this purpose when sites are
not multihomed. Why does multihoming make the routing system behave worse
in this respect?

> prefixes from the ISPs to the multi-homed site. 

Things are a bit less black and white than that. Depending on what failures
you  are concerned about you may not need anything but a default route from
each ISP. For example,
 - the link to the ISP failing - just a default handles that
 - the ISPs peerings with other ISPs fail and the ISP has no alternate path;
   maybe you assume that the ISP knows how to handle this. If not you
   might need the ISP to give you the whole routing table.
 - internal failures in the ISP e.g. cutting off your PoP; perhaps just a
   default route can handle this if the routers in the PoP base advertising
   the default on hearing IGP routes from other parts of the ISP's network
 - the ISP having a broken setup where the BGP routes they offer to you
   don't get withdrawn even if they've lost all upstream connectivity 3 days
ago.
   Oops - even receiving the full BGP table from then doesn't handle that
   type of failure.

> has to be performed by the end-host themselves and not by the routing

This means that every host-pair that communicate (have an existing ULP
connection/association - btw I don't know how to tell this for associations
implemented on top of UDP) have to exchange end-to-end hello packets every 10
seconds or so. If you want to do VoIP you need to do them about every 100 
milliseconds. Imagine 100 Billion hosts or so communicating across the Internet
doing such e2e hello's. Couldn't that be another form of scaling problem?

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 10:44:36 +0100 (CET)

> Encapsulation and notification formats are a merely minor issue

Thanks for sharing, but being able to do this without making security
worse than in the current Internet is a challenge. See 
draft-nordmark-multi6-threats for an initial cut at the threats that we
need to be concerned about.

You might believe that security is a minor issue that can be added
as an afterthought but experience has shown that this
is not the most efficient and expedient way to design secure enough protocols.

> that "all upper layer protocols can operate unmodified" is

I honestly disagree.

> understanding requirements described in my drafts long ago.

The tone of your note in general and this sentence in particular makes
me wonder what you want to accomplish in this working group.
Can we please have a civil discussion!!!

> Do read the drafts.

I did. And I understood them I think. But they didn't seem to address key
important issues like redirection attacks - saying "cookie-based weak security 
for a host authorize changes of locators of its peer." is missing all the
details of the complexity of providing this without introducing new (DoS)
attacks, and is probably too weak since it would make redirection attacks much
easier to perform and harder to detect than in today's Internet.
 
  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Mon, 27 Oct 2003 13:25:47 +0100

Erik,

I will address the issues concerning the preservation of the established
session here and i will consider the issues related to ingress filtering is
a different thread.

> not multihomed.

Well, i am not sure about this.
Current studies of bgp convergence time are talking about up to 15 minutes
of BGP convergence when a route is withdrawn. Established communications
clearly don't survive this.
As i mentioned, bgp outage detection timer is recommended to be set to 90
secs, and as Iljistch mentioned, the default value in some commercial
routers is the double of that. Additionally, you have to add to this the
time required to reconverge, which can be even higher than that.
So, currently available intradomain routing does not always preserve
established seesions when an outage occurs. This is ok, since i guess that
this was not a requirement imposed to the desing of routing system (i mean
the preservation of the established sessions).

Now, this is a requirement to the multi-homing solution. Moreover, it is a
very difficult requirement. I mean, the goal of all this loc/id separation
is the preservation of established communications, since most of the
remaining requirements don't need this and could be addressed with
alternative (more conventional) mechanisms.
So, we are evaluating modifying all the hosts in the Internet to support
this feature, and it is ok.
The problem is that outage detection will be performed by bgp and it will
take 2 minutes. So we have build this loc/id split in all hosts but the
resulting solution is not useful to preserve established communications of
most apps (since i guess most apps need a better response time than this). I
don't know if it is worth it, what do you think?

>
> > - It is pretty expensive, since you need to inject the
> > reachability of all prefixes from the ISPs to the multi-homed site.
>
> you  are concerned about you may not need anything but a default
> route from each ISP. For example,
>

Yes, also RFC3178 supports that and it is much simpler, since it doesn't
require the modification of all hosts to support it.

>    maybe you assume that the ISP knows how to handle this.

How would the isp know how to handle this situation? i mean, the ISP has no
alternate path, so the only path is through the alternative isp of the
multi-homed site.
This is the relevenat case, imho, since is the case not covered by rfc3178.
So i would say that anyone implementing a solution other than rfc3178 would
want to be tolerant to this type of failure.

> If not you might need the ISP to give you the whole routing table.

Yes, this is it.

>    Oops - even receiving the full BGP table from then doesn't handle that

Right, but end-to end failure detection protect you from it

> implemented on top of UDP)

Well, an option would be to have a cache of host that the node is
communicating with, so that it can ping them periodically until the cache
expires. The cache lifetime is extended everytime a packet from/to that
destiantion flows.

>  have to exchange end-to-end hello packets every 10

:-)
Well, som additional cosndierations about this.
First, many existing ULP already exchange this type of messages, TCP ack for
instance, so hellos can be piggybacked into this messages. This would
require some tuning, though. Another way of considering this point is that
hosts already exchange millions of ack.
Second, the response time obtained is directly related to the cost. So, i
agree that preserving a VoIP requires exchanges every 100ms (even every 20
ms), but this is the cost of preserving this type of communication
uninterrupted. You could exchange hellos every 2 seconds, and the
communication would be preserved, but an interruption of a couple of seconds
will be perceived when an outage occurs. Nothe that the response time of
this solution is still a couple of orders of magnitude better than the bgp
based solution. Moreover, the exchange frequency can be negotiated by the
end-hosts, so that they can establish which performance and which cost they
are willing to accept.
Finally, the demands of such a solution is in terms of bandwidth, where i
guess that improvement is expected and scalability doesn't seems to be so
diffcult (as routing table growth, for instance)

Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 14:26:45 +0100 (CET)

> clearly don't survive this.

OK, but fixing the BGP convergence time isn't in scope for this WG.

My logic is to try to find where multihoming makes things different.
The fact that BGP convergence time in the currrent Internet is worse than
desired is something that should be addressed independently of multihoming,
right?

If not, shouldn't multihoming fix other pressing problems like world hunger?

> alternative (more conventional) mechanisms.

I would state the goal differently: taking advantage of the multiple 
attachments to the Internet to make the availability of the communication
better than with a single Internet attachment.

Whether the result would be good enough to
 - make the ftp connection stay alive
 - make the user wait at the web browser or abandon and try another site 
  (order 10 seconds according to some studies I think)
 - make VoIP survive without a click in the phone
is a different aspect of the goal which we haven't talked much about.

> want to be tolerant to this type of failure.

Actually, in my mind but not in my writing, there are two subcases.
1) the ISP looses one peering and there is no alternate path to reach certain
   destinations (due to the policy for the other peerings the ISP has)
   If there are alternatives then the ISP can still reach all destinations
   hence there is no observable failure.
2) the ISP looses peering(s) so that it can no longer reach any destination

The second case can be handled without passing all routes to the site.
How common is the first case?
It depends on how the ISPs handle their own redundancy.

I don't do operations thus I'd be interested in folks with operational
experience commenting on the common and likely failures that a site should
worry about.
What I've heard of are failures due to links being cut between sites and their
ISP, backbone links back-hoed (but don't understand what actual impact
they had on each ISPs network), and ISPs going bankrupt.

> expires. The cache lifetime is extended everytime a packet from/to that

Yes, but what is the lifetime of the cache entry after the last packet?
How can you pick a value without knowing the actual behavior of the (UDP)
application?
Finally, any simulation data on how many peridical pings would be
added to the Internet when everybody implements such a scheme?

> instance, so hellos can be piggybacked into this messages.

The TCP ssh connection I've had open for the last 4 days have not exchanged
any packets as far as I know. (Well, perhaps there are the keepalive packets
every 2 hours or so, but they wouldn't be sufficient to ensure that
once I type something there is indeed a rapid response.)

And we are not considering a TCP-only solution, are we?

My take is that we should make the multihoming solution improve the
availability of sites with multiple Internet attachments without requiring or
assuming e2e periodic pings to quickly detect failures.
Some applications/upper layer protocols might want to use such mechanisms
in addition to the rehoming support in the multihoming solution for quicker
failures (For instance, SCTP already has such a mechanism; heartbeats.)
Thus I'm advocating not assuming that every ULP connection/session/assocatin
has hearbeats when solving rehoming since we know of neither the performance
implications of this on a large scale, nor the benefits that applications
in general will derive from it.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Mon, 27 Oct 2003 15:45:05 +0100

About the goals of the multi-homing solution:

> > > The routing system seems to be good enough for this purpose
> > > when sites are not multihomed.
> >

The goal as stated in RFC 3582 is that:

3.1.6.  Transport-Layer Survivability

   Multihoming solutions should provide re-homing transparency for
   transport-layer sessions; i.e., exchange of data between devices on
   the multihomed site and devices elsewhere on the Internet may proceed
   with no greater interruption than that associated with the transient
   packet loss during the re-homing event.

   New transport-layer sessions should be able to be created following a
   re-homing event.

   Transport-layer sessions include those involving transport-layer
   protocols such as TCP, UDP and SCTP over IP.  Applications which
   communicate over raw IP and other network-layer protocols may also
   enjoy re-homing transparency.

So the question is: do current BGP responses times good enough to achieve
this goal?

IMHO, the problem is not whether we have to fix bgp or not, since i guess we
agree that don't have to fix it (or at least that it is not within the wg
scope)

I guess that the question is whehter we can base a solution that have to
provide transport layer suvivability on the current bgp capabilities.

Perhaps the answer is yes and there is no problem, maybe the answer is no
and we have to workaround and build a solution that don't use bgp.

The other point that you mention is that current non-multi-homed
communication already rely on bgp to be preserved through outages and so
that letting multi-homed communications rely on bgp for surviving outages
should be ok.
Well i don't fully agree with this pov.
We want to provide a solution that provides transport layer survivability,
that is the point of all this. Currently single homed communication probably
don't survive outages, even if there is an alternative path, becuase bgp
detection and reconvergence time is too long. We are providing a new
capability here, that is not available in non multi-homed communications.
So the tranport layer suvivability is one of the things that is different in
multi-homing, and in order to support this we may need not to base our
solution in bgp.

> is a different aspect of the goal which we haven't talked much about.
Yes. we should discuss this. I would say that allowing the user to define
what he wants would be the best case... I mean, a solution that allows the
adaptation of the failure detection to need of the actual established
communication, what do you think?

I guess that the other relevant question would be what are the expected
capabilities in terms of response time for bgp in the IPv6 internet?

[...]
> 2) the ISP looses peering(s) so that it can no longer reach any

I actually think that there is a middle ground situation where things are a
bit more complex
the following example for instance:

                     +---------+
                     |Internet |
                     +---------+
                       /     \
             outage-> =       \
                     /         \
                +---------+  +---------+
                |   ISP1  |  |  ISP2   |
                +---------+  +---------+
                  | |    |      |
     _____________| |    |      |
    /         ______|    |      |
   /         /           |      |
 +---+     +---+        +---------+
 |S1 | ... |Sn |        |  mhsite |
 +---+     +---+        +---------+

In this case, mhsite can reach all the customers of ISP1 through ISP1 and
all the rest through ISP2.
I have always considered the general case, where after an outage some
destiantions can only be reachable through one of the ISPs and a disjoint
set of destiantions can only be reached through the other isp.
Do you think we should start considering a simplified scenario?

> experience commenting on the common and likely failures that a site should

Yes, i would like that too.

 What I've heard of are failures due to links being cut between
> sites and their ISP, backbone links back-hoed (but don't understand what
actual impact
> How can you pick a value without knowing the actual behavior of the (UDP)

YEs this is complex, perhaps we could do something like an exponential
backoff? i guess that i would be more acceptable a delay when trying to
transmit after an interuption of a long period than interrupting a
communication that it is exchanging packets continiously, but that's just a
guess.

> added to the Internet when everybody implements such a scheme?

No :-)
But i do have some simulation about how would the bgp detection mechanism
would affect an ongoing voip communication ;-)
Considering a sample every 20 msec, that would be 3000 samples lost.

> in general will derive from it.

I don't know.
My concern is that:

All the proposed solutions to preserve established communications require a
major update in the installed base, since they require modifications in both
nodes involved in a communication. (in brief big cost) Agree?

However,if we use bgp to detect outages, the benefit obtained will only be
partial, since the failure detection will take a long time in the general
case. This basically means that in multiple scenarios, established
communications will not be preserved. Agree?

Now if this is the cost and this is the benefit i don't know if this
solution is interesting. really high cost and reduced benefits.

Other arguments are that:

A loc/id solution provides a basic capability required to preserve
established communication, but the solution doesn't provide all what it is
needed. Additional stuff needed, such a failure detection mechanism will be
provided by the upper layer.
My answer to that is then why don't we just let the upper layer handle all
the problem then? I mean, you mention SCTP, well sctp provides heartbeat,
allright but it also manages multiple addresses in a connection, so the
loc/id solution doesn't provides anything useful to sctp. The same thing
could be done with TCP and also with application protocols

Another argument is that the loc/is split solution provides an high level
solution that can be also used to satisfy other needs, and not only for
multi-homing, so it should be adopted not becuase it is great for preserving
established communications in multi-homed environments, but because it is
architecturally sound. Well, i may agree with this, but in this case a wider
view is required to uderstand if such a solution is good for all other
requirements imposed, so we shouldn't be doing this in this wg (or only in
this wg at least)

BTW, if the argument to accept a solution is the last one, this would
discard solution without a strong loc/id separation that doesn't provides
good response times (noid for instance), right?

Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 27 Oct 2003 18:23:15 +0100 (CET)

> and we have to workaround and build a solution that don't use bgp.

Perhaps the transport surviability statement in the goals document is a bit
off the mark.
Building a solution which assumes that routing basically works
i.e. only handling the fact that a multihomed site will have multiple
locators and need to have able to "rehome" communication between
those locators seems to make sense to me.
Assuming that routing is broken and making multi6 a general purpose overlay
which works over broken routing doesn't seem like a useful approach -
if routing doesn't work how can you reach the root DNS servers?
Oh - just reimplement DNS as part of the overlay. I think with those
assumptions you end up doing a complete overlay the same way that P2P systems
build a complete overlay (with their own directory, application layer routing,
etc)

But perhaps we started off the wrong foot here.

The concern you expressed was about BGP convergece time.
But if something happens in ISP A that will only affect traffic using 
the locator containing the A prefix. Thus the site doesn't need BGP to converge
about routing for A's prefix (which is the convergence time issue); the site
would just like to receive some indication from A (or detect that the link to
A is down) that something is not working well for A.  That could be sufficient
to have the site border routers start using the path through ISP B.

> the following example for instance:

The above is another case to consider.

> communication that it is exchanging packets continiously, but that's just a

That is an interesting idea. But it doesn't actually provide transport
survivability.
For example, a TCP connection with TCP keepalive enabled and no data 
traffic exchanged for the last few hours expects a response in
a few minutes (the regular time). Same for using VoIP to listen
to a presentation - after having been quiet for 40 minutes I want to ask
a question and want the transmission of that question to be reliable.
Thus I think in general it isn't possible to infer how quickly failover needs
to  happen for a ULP based when the ULP last sent/received packets.

> nodes involved in a communication. (in brief big cost) Agree?

I think you can do multihoming proxies so that a site can transition
to using multi6 even though the endnodes do not yet support it.
Basically the multi6 shim layer gets implemented in the proxy.

But yes, solving multihoming without changing code somewhere in the system
isn't feasible. No free lunch etc.

I think you must have incremental deployment i.e. allow a host or
site to introduce multihoming support and at least take advantage of that 
when communicating with other upgraded hosts/sites.

> communications will not be preserved. Agree?

I've seen papers on bgp convergence time. Are there studies on
the time it takes to infer that "something is problematic"
in bgp? ("something is problematic" might mean a lot of churn in the routing
table) If both ISPs pass their "bgp churn rate" to the site would that
be enough information to start using the other locator?

> provided by the upper layer.

That isn't what I've been saying. ULPs that have very specific time
constraints and are willing to pay the packet overhead, might want to consider
their own e2e heartbeats. The precense of these heartbeats will aid the multi6
layer to detect locator failures faster than the multi6 layer can do 
independently. 
But not all ULPs need such heartbeats.

> could be done with TCP and also with application protocols

Yes, but the number of such ULPs is larger than 1; with a common multi6 shim
you only need to build this in one place instead of N. With per-ULP
solutions your deployment concern doesn't go away.
So I don't see an argument for implementing this N times is better than
implementing it one time,

> good response times (noid for instance), right?

Presumably this wider view would require that it there is a system which
can lookup identifiers e.g. to find the locators.
While one can speculate about such systems (see the future work section in
the SIM draft for an example of such speculation) with desirable properties,
it would take some time to understand what properties are needed,
design, and then deploy such a system.
And it isn't clear to me that there is sufficient impetus for such a system,
even if we had it all specified and implemented today, would actually get
deployed.

But in general I think whether this group decides to take the narrower 
multihoming view or the broader "id/loc separation in its own right"
view, it needs to be understood by the broader IETF community to
avoid any late surprises.

  Erik

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 27 Oct 2003 18:19:39 +0100

> they had on each ISPs network), and ISPs going bankrupt.

Don't forget power outages.

The result is always the same: one or more links go down. If these are 
attached to (real) routers on both ends, the failover happens pretty 
quickly as rerouting is triggered by the interface going down. If there 
there is layer 2 gook in the middle, it takes longer as BGP sessions 
must time out. But even in these cases failover is usually fast enough 
so that when the user experiences a problem, the problem is already 
gone when they investigate or retry. (10 seconds - 1 minute.)

The real fun starts when a large network has routing problems.

> assuming e2e periodic pings to quickly detect failures.

Agree, but note that there are ways to optimize these so they're not as 
evil as they could be if done using some kind of proxy.

> in general will derive from it.

Layer 4 already needs to receive acknowledgements from the other end in 
order to be sure it can continue to send. It's not much of a stretch to 
have a layer 4 protocol send a message back to the mh layer saying "you 
may want to trigger rehoming" when there are no ACKs for a while.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 27 Oct 2003 19:24:15 +0100

> clearly don't survive this.

Ok, but answer me this: how does established communication survive the 
fact that the route to one of the endpoints is withdrawn in the first 
place? No route means no connectivity.

(Un)fortunately these researchers often don't really understand BGP so 
they model it in the wrong way. If you build a huge network, infuse it 
with instability and then revoke a route, you could perhaps end up with 
a 15 minute wait before the route has really disappeared everywhere. 
However in practice it isn't this bad. I've done some tests a while ago 
and what I saw pretty consistently was two minutes.

But apart from that revoking a route at the source isn't a very typical 
condition or one that we need to survive. What we need is to detect a 
broken link and reroute. The first part can take up to three minutes 
because of Cisco's unbelievable default hold time, but it doesn't have 
to: if one end sets a lower hold time, that one is used, and routers 
are by default configured to immediately tear down BGP sessions when 
the associated interface goes down.

Rerouting is a matter of seconds in the absense of prior instability, 
or 30 seconds otherwise.

So many outages don't even lead to a noticable interruption, a good 
number make TCP sessions hang for a relatively long time (30 - 180 
seconds) but don't break them, and only under very unfortunate 
circumstances rerouting takes longer than a TCP timeout.

Note that there are efforts underway to detect a dead neighbor very 
quickly. This should take the edge off of these problems by eliminating 
the 180 second timeout. See 
http://www.ietf.org/internet-drafts/draft-katz-ward-bfd-01.txt

>> milliseconds. Imagine 100 Billion hosts or so communicating across 
>> the Internet doing such e2e hello's. Couldn't that be another form of 
>> scaling problem?

> instance, so hellos can be piggybacked into this messages.

If you get an ACK, you know you have reachability so no need to 
piggyback anything. I think it would be a very good idea to dump the 
job of detecting _potential_ failures in the lap of layer 4 (which for 
many real time protocols means the application). TCP has a very good 
idea of how reachable the other end is. For real time protocols, it's 
probably either impossible or too expensive to make rehoming seamless. 
But why would we have to? A short hiccup in the audio/video would be 
bad, but not as bad as 1. sending ACKs ever 100 ms and 2. rehoming 
every time a single packet is lost. It's not like there are outages so 
rehoming is necessary every few minutes.

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Mon, 27 Oct 2003 19:42:42 +0100

On måndag, okt 27, 2003, at 14:26 Europe/Stockholm, Erik Nordmark wrote:

> ply-To: Erik Nordmark <[email-masked]>
>

Agreed with Erik. There is also no need to use BGP as we know it 
forever.

> If not, shouldn't multihoming fix other pressing problems like world 

Now that you mention it....;-)

- - kurtis -

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 27 Oct 2003 23:51:39 +0100

> Considering a sample every 20 msec, that would be 3000 samples lost.

If we assume for a few moments that we can't make failover seamless, 
what would be an acceptable delay for VoIP?

Applications such as streaming media have the interesting property that 
the arrival of new data is extremely predictable. This makes it 
possible to use negative acknowledgements to trigger failover rather 
than the lack of regular acknowledgements, which is what TCP does.

If you expect a sample every 20 ms and you didn't get one for 100 ms, 
you can be pretty sure that something bad is happening, so it might be 
a good idea to signal the other side to try another address. This 
should give you failover within a fraction of a second without having 
to send any packets during regular communication. (Note that you still 
need ACKs in order to detect a correspondent going away altogether so 
you're not firing off packets into a black hole.)

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 28 Oct 2003 10:05:18 +0100 (CET)

> may want to trigger rehoming" when there are no ACKs for a while.

Yes.

In general one would also need to be concerned about the end sending
the acknowledgements when the reverse path is broken since the layer
4 sender of the ACKs don't observe any problems (other than seeing
retransmissions from the other end).

However, one of the interesting aspects of being able to use the source locator
in the receive packets as the best destination locator for responses coupled
with locator rewriting by routers is that the ACKing end  will not need to do
anything special; it is sufficient that the retransmitting end give the above
type of advice to the multihoming layer.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Tue, 28 Oct 2003 12:39:44 +0100

Erik, Iljistch,

About the transport survivability:

> Perhaps the transport survivability statement in the goals
> document is a bitoff the mark.
>

Well, i don't know. It is the closest to a consensus that we could get in
those issues.
We could reconsider it, i guess. But imho we should agree on what we are
looking for.

But what i really think is that we should be aware of the capabilities and
limitations of the solutions. We should not believe that we are making a big
effort to provide established communications survivability when what we
actually are providing is a limited solution that will only preserve some
communications but not the general case.

IMO, we should try to satisfy this requirement, or at least provide some
tools to allow to achieve this functionality, even if additional mechanisms
are required to do it.

About the broken routing system:

> Assuming that routing is broken

Actually i don't think that the routing is broken. IMHO is a bit more
complex than that.
I think that the constraints imposed to a large routing system are
incompatible with the requirements of preserving established communications.
I mean, a global routing system must be stable. this means that changes on
the topology of short duration cannot be propagated to all the network. The
system has to filter the changes with high frequency in order to obtain
stability. OTOH, applications are susceptible to this short cuts,
essentially this means that what is not significant for the routing system
to propagate it is significant for the application.
This doesn't mean that the routing system is broken, it just means that the
transistorizes of the routing system to reach a stable view are filtered and
that during that transistorizes the view of the topology by the routing
system will not be accurate during that period.
this means that the routing system is useful to provide the view of the
network in stationary state and that high frequency changes will not be
reflected
IMHO this is ok
The problem is that you cannot ask the routing system to react fast enough
when changes occur (actually you don't want that)

So, in a few words, the routing system has some inherent constraints that
limit its response time.
The resulting response time may be ok for some apps but it may not be ok for
other apps.
So, one approach is to accept that the routing system response is limited,
and let the apps that require additional functionality to build their own
stuff.

> and making multi6 a general purpose overlay

you can see it that way, you can also consider it as the host selecting the
path since he is the only one who can do it in a proper way.

> if routing doesn't work how can you reach the root DNS servers?

Well, DNS already have their own fault tolerance support, don't they?
I mean if a root server is not available because bgp is reconverging for 15
minutes, i don't wait 15 minutes, i just pick the next one on the list.

> But perhaps we started off the wrong foot here.

FWIW i don't like keepalives more than you do.
It is just that i don't think that the routing system can provide the
capabilities imposed by the preserving established communications
requirement, and i can't think any better solution

But let's explore alternatives, we can always come back to this if we don't
find anything better.

Moving forward:

The routing system has a limited response time that may not be suitable for
some apps. agree?

We have then two approaches to improve this (that may be complementary)

- Obtain hints from the routing system that something is wrong

> to have the site border routers start using the path through ISP B.

I like that idea, especially the churn rate idea, i think we should study it
more in depth.

The benefits resulting from this approach is that the response time obtained
using information from the routing system is improved.

I guess that some apps will need better response time anyway, so the
complementary approach is using some ULP hints (as Iljistch been proposing
for several years by now)

This means that if the ULP protocol detects something wrong it would notify
to the shim layer and the shim layer would act accordingly changing
locators.

Some potentials problems that i can think of:

My first concern is about the interaction of this two mechanisms.
I mean, in the routing system based mechanism the knowledge of which locator
is better resides in the routing system, So, since the routing system knows
which locator is better, routers are enabled to rewrite locators, since they
can choose the best one.
In the other mechanism, the one who knows which is the bests locator is the
host itself, so it forces the selection of the ISP to be used by selecting
the appropriate locators (both source and destination)

Now my concern is how these two mechanism interact, i mean the host selects
one locator because it has an upper layer hint that something is going wrong
and then the routing system that still haven't detected anything yet, just
rewrites the locator.

Let's suppose that we have two hosts A and B and both of them reside in
multi-homed hosts, so A has two addresses P1:A and P2:A and B also has two
addresses P3:B and P4:B

Now they have a TCP connection established using P1:A and P3:B.
Suddenly, A detects that it has to retransmit and sends this hint to his own
shim layer.
Now what does the shim layer in A does?
It can change the Source locator and/or the destination locator.
Changing the source locator is not very useful since it will be rewritten by
the border router, and perhaps changing only the destination locator doesn't
solve the problem
Perhaps if the rewrite ok bit is not set, this would mean that the packets
has to be routed through the isp that is compatible with the source address
contained in the packet, so that the host can force the isp selection.

I know this is just speculation, and that we should see the details with a
more concrete proposal.

My second concern is about ULP hints implementation.
Right now ULP protocols don't do this. so this would imply modification, not
only in all the hosts to support the new shim layer, but also in all the ULP
that want to obtain the improved service.

Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 28 Oct 2003 13:32:30 +0100 (CET)

> Well, i don't know. It is the closest to a consensus that we could get in

But if I don't misremember the discussion that lead to the transport
survivability there was a question whether it was considered sufficient
to select a working set of addresses/locators when new communication
was established (e.g., as part of the application + TCP trying all
the addresses returned from the DNS) and the concensus was
that this was insufficient. Thus the goal is to provide rehoming
of already established communication.

This is expressed in the document as "transport survivability" but I don't
think we discussed whether this also implied a failover in any particular
time. As we've discussed different transports have radically different
time constraints.

> We could reconsider it, i guess. But imho we should agree on what we are

Agreed.

> communications but not the general case.

But there is no useful general case; I can concoct an application + transport 
over UDP that requires failover in one microsecond. I don't think we can use
the existence of such a thing as a guage to say that multihoming should
try to accomplish failover in one microsecond.

And how useful it is to only assume that the time before TCP gives up
(a few minutes) matters when users in front of web browsers give up
a lot sooner (10 seconds?)?

In terms of the time to failover I think all we know is that the shorter
it can be the better, but we don't know how much we are willing to "pay" to
reach any particular time limit (for different types of failures which
occur with different probabilities).

> tools to allow to achieve this functionality, even if additional mechanisms

But I don't even know what it means to satisfy "transport survivability" - for
my microsecond transport? For common TCP implementations timeout (since the
TCP standard doesn't say when TCP should given up; the assumption by some
protocol designers was "never" - keep trying).

> to propagate it is significant for the application.

I don't think it is that black and white. The routing system could propagate 
bad news faster than good news; what needs to be limited is the amount
of flapping.
If bad news travel fast then multihoming can provide choices between
different paths taking the bad news into account.

> system will not be accurate during that period.

What do you mean by "transistorizes"? Dictionaries say:
Date: circa 1952
: to equip (a device) with transistors

but that doesn't make sense in this context.

> requirement, and i can't think any better solution

And I think you're just too quick to draw that conclusion which is why
I try to push back a bit.

> The routing system has a limited response time that may not be suitable for

Let me suggest that the routing system of today has evolved in an environment
where the endpoints are not agile (in switching between locators) hence there
hasn't been any incentive to extract and/or propagate information
that agile endpoints/sites can use.

> - Obtain hints from the routing system that something is wrong

The need might actually be weaker than that; being able to make some
qantitative comparison between routing (flaps, stability, etc) between
two possible ISPs/paths might be sufficient. Doesn't mean that one
of them have to make a statement that "something is wrong".

Another possibility to investigate is to make bad news travel faster than
good news through the routing system.

> the appropriate locators (both source and destination)

Conceptually it would make sense to view it as the host/ULP observing the need
to try a different locator, but the routing system (through feedback mechanisms
like locator rewriting and perhaps others) influence the order in which
locators are tried.

> and then the routing system that still haven't detected anything yet, just

Host would try a different destination locator and routing system would
rewrite the source locator hence they would be complimentary I think.

> that want to obtain the improved service.

FWIW RFC 2461 recommends ULP hints "all is good" to avoid performing
neighbor unreachability detection probes. Perhaps the same hints can be
used to have the multi6 layer stay with the same locator?
Or perhaps we need both those positive hints and negative hints for multi6?

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Tue, 28 Oct 2003 17:00:11 +0100

> occur with different probabilities).

Yes, agree, this is the question that we have to answer

I guess that the approach could be to see how much could be achieved with
the different solutions and then see if it is worth it.

> I don't think it is that black and white. The routing system
> could propagate bad news faster than good news

I am not sure if this is true...
I am no routing expert, but i think that withdrawing a route is inherently
slower than advertising a new route, at least in path vector algorithms.
The problem is that to detect that there is no route to a certain, all the
alternative routes with increasing AS path are tried, which is inherently
slow. This would be similar to the count to infinity problem, but with
different AS path. This implies that the time to converge is proportional to
the number of ASes in the network. At least that is what i understood from
the Labovitz paper, but i may be misunderstanding some issues here.

However, i guess that when the system is looking for a new route, (i.e.
reconverging) multiple routes with increasing AS path length are advertised,
which can be used as a hint to infer that something may wrong (which i guess
was you proposal about the churn rate detection, right?)

> : to equip (a device) with transistors

oops... sometimes the spell checker do some funny things. I meant
transitory, but never mind, i guess you already see what i meant.

[...]

> rewrite the source locator hence they would be complimentary I think.

Well, i still don't see this very clear.
Let's try with an example.

Recycling an old figure, we have:

            +----+
        ----|ISPA|_             +----+

We are using packet rewriting at site border routers. This means that the
site exit router connection a multihomed site with ISPX will rewrite source
address of packets and it will replace the contained prefix with  PX::. Is
this correct?

Suppose that mh1 has an established tcp connection with mh2. They are
currently using PA:mh1 and PC:mh2 as locators and the communication flows
through ISPA and ISPC.
The link between ISPC and the internet breaks
It takes 180 seconds to bgp to detect the outage, so routing remains
unchanged during that period i.e. packets flowing from the mhsite2 to an
address containing PA are forwarded through ISPC.
There are two cases to consider here, one is when the internal routing of
mh2 routes packets sent to PB:: through ISPC and when it routes them through
ISPD. I guess that the problematic case is the first one, so let's assume
that packets flowing from mh2 to PB: are carried also through ISPC.

Now when the outage occurs, one or both of the communicating nodes can start
to retransmit.
Let's consider that mh1 is the one whose TCP timeouts and that starts to do
some retransmissions. Then TCP at mh1 communicates the shim layer at mh1
that something wrong is happening.
Then the shim layer changes the destination address that is being used in
the communication from PC::mh2 to PD::mh2. In this way the communication is
rerouted to the alternative ISP. Good!

Now, suppose that mh2 also obtains a hint that something is wrong (this
could be because TCP at mh2 timeout or because packets start arriving with a
different source address (this last case cannot be guaranteed because it
depends on the internal routing in mh1)). So, mh2 shim layer start using an
alternative destination address, so it switches from PA:mh1 to PB:mh1.
However, this does not solve the problem, since packets addressed to PB::
are also routed through ISPC.
The problem here is that TCP detects the problem but it has no means to
communicate the problem to the mh2's routing system, who still believes that
a route through ISPC is still available.

I guess that if we decide to place a failure detection mechanism in the
hosts themselves, we have to provide means to let the host force the routing
system to select the path, or at least the exit router.

This could be achieved with source address based routing.. the problem here
is that this is not compatible with source address rewriting by exit
routers, i mean or whether the source address is fixed and the exit isp is
determined by it or the exit isp is fixed and the source address is
rewritten to make it coherent with the isp selected, but not both.

One option would be that for packets with the rewrite ok bit not set, source
address based routing is applied and when the rewrite ok bit is not set then
destination address based routing is used.

(Another hint can be the reception of ICMP error messages.)

This is simple example, since no interaction with routing based mechanism
(churn level detector and so) are supposed. maybe we can try with a more
complex the next round :-)

regards, marcelo

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Wed, 29 Oct 2003 10:39:25 +0100

On tisdag, okt 28, 2003, at 12:39 Europe/Stockholm, marcelo bagnulo 

> tools to allow to achieve this functionality, even if additional 

Isn't this simply expectation management? The reason we changed the 
requirements into goals was because the items in the document where 
either contradictory; we realized that we couldn't get it all so make 
it all requirements wasn't an option; and we couldn't decide on what 
issues where needed more than others; ?

I think that connection survivability is a nice to have, but wether a 
connection will survive or not will also be highly dependent on the ULP 
as Erik points out. I think this is best handled by simply documenting 
what this solution offers, and what the alternatives for the ULPs are.

- - kurtis -

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Wed, 29 Oct 2003 10:49:32 +0100

> time. As we've discussed different transports have radically different

To some extent I wonder how much of this is really a multi6 problem. As 
you point out, we will never know how long ULP will continue to try 
before giving up. This might render ULP hints useless in that 
particular case. We also don't know what failure detection is in use 
for the ULPs. More, if there is also a rerouting event causing 
distribution of bad news, and perhaps a new set of locators, or the 
signaling to move to new locators, this might trigger a "race-like" 
condition, depending in what order the priorities are set and in which 
order they arrive where, and then actually making the case worse - no? 
Would it not be better to simply say that survivability and 
"restoration" time is due to these X factors, they can be influenced by 
this - but they are not an issue per se for the multi6 layer. They 
might depend on ULP signaling, BGP convergence, etc. But that is 
another issue.

Am I making my life to easy here?

- - kurtis -

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 29 Oct 2003 06:53:51 -0600

This is actually a pretty helpful discussion. A couple of points...

Spencer

> The routing system has a limited response time that may not be
suitable for

Agreed, but I'm not sure everyone else in the IETF agrees. It Would Be
Lovely to pick a target (TCP survivability, HTTP survivability, VoIP
survivability being three rough possibilities) and agree on it.

> contained in the packet, so that the host can force the isp
selection.

And it seems to me that the extent to which this happens has a
profound effect on mobile/multihomed applications and users in the
coming years.

I was the one standing up in the IAB workshop saying (I am
summarizing) if we get a lot of applications that WANT to handle their
own path selection, while we "solve the multihoming problem", they'll
STILL WANT to handle their own path selection after we "solve the
multihoming problem". If we pick a target (see previous), we can at
least tell some applications developers that they can ignore path
selection (multihoming happens automatically for them).

Spencer

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 29 Oct 2003 15:40:36 +0100 (CET)

>        /    +----+ \_+------+  _|ISPC|_

Correct.

> a route through ISPC is still available.

I thought you expressed concern about potential destructive interference
between the endpoints trying different locators and the routing system
finding a new path. But in the above example I don't find such destructive
interference.
(Sure, the worst case performance might not be better than the worst case
routing convergence time, but that is a different topic.)

So do you see any destructive interference?

If not folks can work separately on the multi6 rehoming and improving the
convergence time as well as information (such as churn rate) in BGP.

> This could be achieved with source address based routing.. the problem here

Source address based routing seems like a fair amount of added
complications; impacts the hardware/firmware in the routers and what else?
It might be easier to work on BGP convergence time.

> (Another hint can be the reception of ICMP error messages.)

I'm concerned about the threats in that case since the host has no way
to assertain whether the ICMP error was generated by a router in the path
or some off-path spoofer.

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 29 Oct 2003 16:41:48 +0100 (CET)

> order they arrive where, and then actually making the case worse - no? 

I think it makes sense to look for potential destructive interference.
It might be that we have to restrict some mechanisms (such as ULP
retransmit hints) from affecting the destination locator and others
to affect the source locator, and perhaps also specify a preference
between the different hints (for instance, the source locator in a
received packet might be a stronger indication of what to use
in destination packets than the ULP suggesting to use a particular
destination locator, or it might be the other way around).

But these types of policy considerations for selecting locators is
independent of the actual rehoming mechanisms I think.

> Am I making my life to easy here?

Perhaps a bit; but the policies can be worked on separately from the
mechanisms and the policies can evolve at a different rate in the future.

  Erik

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Wed, 29 Oct 2003 22:02:40 +0100

Agreed.

> mechanisms and the policies can evolve at a different rate in the 

Would it make sense to add that into the next version of the draft?

Best regards,

- - kurtis -

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 29 Oct 2003 22:25:06 +0100

> complications; impacts the hardware/firmware in the routers and what 

Many routers already support policy routing in hardware. Source address 
based routing is a subset of full policy routing. I don't see much of a 
problem in the  long run.

> It might be easier to work on BGP convergence time.

I actually have an idea for avoiding the BGP version of the count to 
infinity problem... Is this on topic now for this wg?  :-)

---

From: "marcelo bagnulo" <[email-masked]>
Date: Thu, 30 Oct 2003 15:56:51 +0100

Hi Spencer,

> survivability being three rough possibilities) and agree on it.

What do you mean?
That the M6 layer should provide 3 types of service, TCP HTTP and VoIP like
services and that the app can select it? Or perhaps an automatic selection
based in the port/protocol combination that is being used?
Or just that we should have theses 3 services as reference?

Regards, marcelo

---

From: "marcelo bagnulo" <[email-masked]>
Date: Thu, 30 Oct 2003 15:56:50 +0100

Hi Kurtis,

I am including answers to both of your emails referring this topic in this
one.

> issues where needed more than others; ?

Well i am not sure this the case of session survivability, at least right
now. I mean, i don't think that providing transport session survivability is
contradictory with another of the requirements, it is just that providing it
is expensive.

> I think that connection survivability is a nice to have,

IMHO is more than that. It is the main problem that we have.
I mean, all other aspects of multihoming are simpler to solve than this one
i guess.
The preservation of established sessions is the only requirements that its
support imposes modifications of both sites of the communication. this
implies that hosts *outside* the multihomed site will need to be upgraded in
order to support this feature of the multi-homing solution. This is IMHO a
great effort with a really important cost. This cost is justified only if it
is an important feature, i guess.
If session preservation is a nce to have feature, we could just don't
provide it and provide a multi-homing solution that only imposes
modifications to the multi-homed site. Such a solution would be more easily
accepted and deployed, since only the ones that obtain a direct benefit has
to upgrade their systems.
But I would say that this is not the case, people have been saying that the
preservation of established communication is important.

> but wether a

I think that we can do a bit more than that. I guess that we can provide
some ways to allow M6 compatible ULP to obtain an enhaced service.
I mean:
Different ULP have different needs, so it seems expensive (and perhaps
useless) to provide a general solution that satisfies the more demanding
needs.
So the approach that is proposed would be:
The general M6 layer provides a general service, for instance based in
current bgp response times
Also, the M6 accepts hints from modified ULP that have more demanding
requirements. This would enable that if an ULP is capable of detecting
outages more quickly than the generic M6 layer it can signal it to the M6
layer so that it can react somehow

This presents some  potential difficulties like the ones you point out:

> More, if there is also a rerouting event causing

I guess that we should try to see if it is possible to design a solution
that a avoids that.

> this - but they are not an issue per se for the multi6 layer.

Yes, but IMHO we should define those potential hints and also define the
interaction among them. Do you agree?

Regards, marcelo

> They
> -----BEGIN PGP SIGNATURE-----
> Version: PGP 8.0.2
>
> iQA/AwUBP5+NMKarNKXTPFCVEQLuTQCg8NTn1V0mniEDuTYQCybs3cyLFnUAoIq+
> 6m1xmITLki2bShj/JcERogDm
> =DI3I
> -----END PGP SIGNATURE-----
>
>

---

From: "marcelo bagnulo" <[email-masked]>
Date: Thu, 30 Oct 2003 16:27:00 +0100

[...]

> > could be because TCP at mh2 timeout or because packets start
>
> > arriving with a different source address (this last case cannot
> > be guaranteed because it depends on the internal routing in mh1)).
>
> > So, mh2 shim layer start using an alternative destination address,
> > so it switches from PA:mh1 to PB:mh1. However, this does not solve
> > the problem, since packets addressed to PB:: are also routed through
ISPC.
> > communicate the problem to the mh2's routing system, who still
> > believes that a route through ISPC is still available.
>

Well, i think that the ULP hint mechanism is not working for this particular
case, so i would say that a different interaction than letting the ULP hints
set the destination locator and the routing system to determine the source
locator is needed.
In this case we may detect some desturctive interference
IMHO, we should try to define a ULP hints mechanism that properly uses all
hints and that improves the routing system response time.

> It might be easier to work on BGP convergence time.

Well, i am not really agree with the evolution of the reasoning here.
Let me see if i do a brief summary:
- In order to provide ULP established session, we propose a solution that
implies the modification of all the hosts in the IPv6 Internet. IMHO, it's
ok
- However, if we base our solution only in BGP recovery features, the
solution won't be usefull to preserve the established sessions of a number
of applications.
- One option to address this concern is to use keepalives, but a general
solution based in keepalives exchanges seems inefficient and expensive. IMHO
agree
- Since every ULP has different needs we can use ULP hints to detect if
something is wrong. IMHO good
- Now the problem is that the host cannot force the routing system when it
detects a problem based in the ULP hint
- A proposed solution is to use source address based routing
- But now you say that it would be better to work in BGP convergence, which
is clearly out of the scope of the wg
The result is that the solution than will not provide ULP session
surviavility for a number of cases.
Is this the evolution of the discussion?
If so, i don't agree with the last steps.
IMHO we will propose an expensive solution and it should better provide good
capabilities and this inlcudes the preservation of the ULP sessions.
I don't think it is a good approach to tell that someone else will solve the
BGP convergence problem, so that we will just wait for that.
I think that the ULP hint approach has possibilties and we should try to
work it out, if it is not source address based routing, it may a routing hea
der or something else.
So my proposal is let's try to solve the problem above.
What do you think?

> or some off-path spoofer.

Agree, however, this could be a low priority hint perhaps?

Regards, marcelo

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 30 Oct 2003 17:19:53 +0100

> slower than advertising a new route, at least in path vector 

There was a time that interdomain routing was under threat from 
instability caused by frequent up-down and down-up transitions. The 
solution was to implement "flap dampening": if you flap (down-up 
transition) too many times within a certain timespan, your route is 
ignored for some time, saving some expensive BGP and route table 
operations on the router in question, but more importantly in routers 
further upstream.

Note that not propagating good news, so that a route seems to be down 
when in fact it is up, isn't too harmful as there are presumably other 
paths available so the destination is still reachable. Failing to 
propagate bad news on the other hand would be catastrophic, as packets 
continue to flow to a place where they can't be delivered. Avoiding 
this situation is the number one concern when running BGP.

> slow.

Only if there are no alternatives. If there are, the path for those is 
probably not that long so they show up pretty quickly. Only when the 
last/only path goes down you'll see the following problem:

> This would be similar to the count to infinity problem, but with
> different AS path.

But we usually don't care since when our last/only path goes down we're 
unreachable anyway.

> This implies that the time to converge is proportional to
> the number of ASes in the network.

No, the number of possible paths (= interconnection exists) where the 
ASes to the left provide transit for our viewpoint and the ASes to the 
right provide transit for the source AS. This number increases 
exponentially with the number of transit links ASes have. So more 
multihoming actually hurts here.  :-)

---

From: masataka ohta <[email-masked]>
Date: Fri, 31 Oct 2003 05:11:58 +0900

Erik;

> need to be concerned about.

The threat ID is an abstract nonsense.

The only thing to be secured w.r.t. multi6 is a set of
locators of a peer.

That is, all that is required is to share a cookie at the
start of the communication, to be used later to authenticate
locator set changes.

Note that when the set is stable, no new mechanism is necessary.
For example, that SMTP clients today try all the addresses
of all the servers of a domain is a form of end to end
multihoming, which is no security threat.

In addition, it is, of course, necessary, not to make a
protocol a DOS amplifier, but it is a generic requirement
to all the protocols and is not multi6 specific.

> is not the most efficient and expedient way to design secure enough protocols.

Security is a minor issue to be taken care of easily from the
beginning.

> I honestly disagree.

Protocol is defined by the packet content on the wire, not
by API.

That a packet of a protocol is modified within a host is a
host internal implementation issue having nothing to do with
the protocol.

You preserve API, not the protocol.

> Can we please have a civil discussion!!!

I wonder what you want to accomplish in this working group.

It is no useful to have a harmonized discussion to reach a
peaceful compromise on a set of useless drafts.

> easier to perform and harder to detect than in today's Internet.

I'm afraid you merely underestimate the security threats in
today's Internet.

In your threat ID, you wrote:

: Any system that is along the path from the source to the destination
: host can be compromised and used to redirect traffic.  Systems may be
: added to the best path to accomplish this.  Further, even systems
: that are on multi-access links that do not provide security can also
: be used to redirect traffic off of the normal path.

and the cookie approach do not give protection against attacks
from hosts on the path, which is no worse.

The Internet today is not so secure. Under certain assumption,
redirection attacks are easy and hard to detect.

							Masataka Ohta

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Thu, 30 Oct 2003 22:07:00 +0100

On onsdag, okt 29, 2003, at 22:25 Europe/Stockholm, Iljitsch van 
Beijnum wrote:

> Is this on topic now for this wg?  :-)
>
Maybe we should solve the M6 problem first...:-)

- - kurtis -

---

From: "Spencer Dawkins" <[email-masked]>
Date: Thu, 30 Oct 2003 21:46:24 -0600

My apologies for not being clearer. What I meant was, multi6 should
pick a targer for survivability, whether measured in tens of minutes
(TCP), tens of seconds (HTTP with a human in the loop) or hundreds of
milliseconds (VoIP).

By doing so, communities can consider their needs vis-a-vis the
target. If your needs are met by the target, you can rely on multi6
for survivability, and if not, you can start working on
special-purpose survivability mechanisms.

Spencer

----- Original Message ----- 
From: "marcelo bagnulo" <[email-masked]>
To: "Spencer Dawkins" <[email-masked]>; <[email-masked]>
Sent: Thursday, October 30, 2003 8:56 AM
Subject: RE: Preserving established communications (was RE: about
draft-nordmark-multi6-noid-00)

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 13:41:34 +0100 (CET)

> Would it make sense to add that into the next version of the draft?

I can add that into 02 (since while nobody was looking a generated
01 versions of NOID and SIM - I'd forgotten to discuss at least
the premeditated redirection attacks in 00).

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 13:44:23 +0100 (CET)

> based routing is a subset of full policy routing. I don't see much of a 

Is it really a subset?

Can existing routers and routing protocols convey, for different source
prefixes, different nexthops for the same destination?

I thought source routing merely implied that a route might only be applicable
for certain source prefixes, not that there could be multiple routes
for a given destination address/prefix that get applied based on the source
address in the packet.

  Erik

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 31 Oct 2003 14:17:52 +0100

> Is it really a subset?

Source address based routing basically boils down to a set of "if 
source address == a then next hop = x" clauses. On a Cisco router, this 
can be implemented using the more general policy routing mechanism that 
can also look at many other fields in the headers and some other stuff. 
I gather that Cisco and some other vendors support doing this in 
hardware and/or the fast path.

> prefixes, different nexthops for the same destination?

It is possible that IDPR supports this, but since IDPR has been dead 
for a decade we can safely assume "no". However, modifying routing 
protocols, while not exactly a trivial excercise, isn't as bad as 
modifying ASICs. Also, I'm not sure if routing protocols are really 
required here. A static setup should be enough.

> for a given destination address/prefix that get applied based on the 

Please don't use "source routing" in this context as the confusion with 
actual source routing will never end.

There are different ways to approach this. A very general one would be 
to combine header fields (lets assume source and destination for now, 
but it could be more) and the next hop and attach a preference value to 
every such combination. Then routing becomes the process of looking up 
all header/next hop combinations where the header fields match for the 
packet in questgion, select the one with the highest preference and 
forward the packet according to the listed next hop. There is the 
slight issue that the cartesion product of the full sets of destination 
prefixes, source prefixes and next hops could get out of hand, but 
that's a question of optimization.

But we don't have to go this far. We know that if a source address tied 
to ISP A is used, there is no use in forwarding the packet to ISP B, 
and the other way around. So there is no need to even check whether the 
destination is even reachable through the ISP indicated by the source 
address as there is no alternative course of action to forwarding the 
packet to that ISP and hope for the best. So essentially we're just 
taking default routes.

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 14:38:22 +0100

> special-purpose survivability mechanisms.

My reading of the thread is that the approach is somehow different:

The solution will provide a basic capability based on routing system path
outage detection.
Such mechanism won't provide a defined response time, since its response
time depends on the situation and the worst case can be very bad.

Additionally, the solution would accept hints from the ULP that can be used
as failure detection mechanism. This is better than leting the apps start
working on their own special purpose survivability mechanism, since they
don't need to discover alternative locators and provide proper security and
so on, they just need to provide hints to facilitate outage detection

The general mechanism would be used by ULP that don't require such a good
response time and also by legacy ULP that don't implement the hint
mechanism.
Does this sound good to you?

Thanks, marcelo

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 14:38:54 +0100

> last/only path goes down you'll see the following problem:

Ok, but this is the case that we care, right?
From ISPA prespective, there is no longer a route to the destiantion, so the
multi-homed site detects that and tries the alternative ISP (ISPB).
So the case that we care is the reconvergence of BGP within the portin of
the network where a route to the destiantion address no longer exist

> But we usually don't care since when our last/only path goes down we're

I don't see it that way, i mean, the destiantion *is* unreachable through
one of the providers. The route is withdrawn from that part of the network.
Then once that the route has been withdrwan from that part of the network,
the multihomed site detect the problem and tries with the alternative ISP.

Anyway, i guess that the reconvergence time is only part of the problem.
Another part of the problem is the time required to detect the outage. Only
this factor would render the solution inacceptable for many applications.
Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 15:46:32 +0100 (CET)

> is clearly out of the scope of the wg

FWIW I didn't suggest that this WG should work on improving BGP convergence
time.
But it is something that perhaps an other WG could do in combination
with operators and vendors.

> surviavility for a number of cases.

Since we haven't defined what session surviability actually means there
is a simple approach; use a multi6 solution with existing BGP and
set the time when TCP gives up to inifinite i.e. TCP will keep on
trying forever. That provides session suvivability.

While that might not be that useful, it does illustrate that the discussion
is really about the timelyness of failure recovery and not about whether
failure recovery is possible.

Perhaps having optional M6-layer heartbeat messages that
the ULP can enable by stating "I'd like failure recovery in 17 seconds"
would make sense.
As your examples point out this might still run into BGP convergence time
issues if all of the peer's locator prefixes are routed out the same
failed link/ISP.
But it might still be a step in the right direction in terms of abstracting
the "failure recovery" service offered by an M6 layer.

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 16:04:20 +0100 (CET)

> packet to that ISP and hope for the best. So essentially we're just 

But to solve Macelo's example where packets to destination PC:X and PD:X
both exit Y's site over the same link/ISP (and it takes a long time to
discover that that link/ISP has failed), you would actually
need something stronger. For instance something along the lines
of being able to configure the routers so
that packets with the source prefix PA all-other-things-being-equal(?) exit 
via ISP A and source prefix PB exit via ISP B. That
way the sender can explore the two ISPs by changing the source locator.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 18:00:25 +0100

> way the sender can explore the two ISPs by changing the source locator.

IMHO You would want to use the source address based routing when you don't
want to use source address rewriting.
There are some situations where you don't want the rewrite to take place and
you want source address based routing. I can think of the following:
- The external host don't support M6
- The M6 context has not been established yet
- An ULP hint made the M6 layer to select a different path, so you want to
let the host select the ISP and not the routing system

Note that source address routing means that the host selects the exit ISP
and source address rewriting means that the routing system selects the exit
ISP.

So a possible approach would be that if the rewrite ok bit is set, then you
perform destination address routing and if the rewrite ok bit is not set you
do source address based routing.
Does this makes sense?

Regards, marcelo

---

From: Eliot Lear <[email-masked]>
Date: Fri, 31 Oct 2003 09:25:30 -0800

> for a given destination address/prefix that get applied based on the source

There are ways to do this, particularly with MPLS-TE.  This allows for 
redundant paths based on all sorts of criteria.  But I don't think this 
is what you had in mind.

Eliot