Subject: survivability, rewriting

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 30 Oct 2003 21:32:44 +0100

About the session survivability: I think Kurtis is right that we must 
not let ourselves get caught up in unrealistic expectations. I believe 
a useful lower limit would be five seconds. This gives just over two 
round trips when both parties are using GSM/GPRS and/or satellite, a 
few more for more reasonable link technologies. And two missed acks is 
the absolute minimum we must require before even considering a rehoming 
event, as a single missed ack can very easily happen for many reasons 
that don't warrant rehoming. So if an application can't handle a 5 
second gap in the communication, it shouldn't count on general 
multihoming mechanisms to provide failover.

I don't think this is too unreasonable. Many people have mentioned 
VoIP, often in the same sentence with extremely unrealistic failover 
expectations. I believe 5 seconds is workable for VoIP: this is 
certainly close to the time a user will continue to shout "hello, are 
you still there?" when using a cell phone with less than perfect 
reception.

I maintain that having the transport layer provide hints to the 
multihoming layer about when a rehoming would be desired is the right 
approach. Yes, this will give us some trouble in the beginning, as 
existing upper layer protocols don't provide these hints yet. So we 
implement additional heuristics so the multihoming layer can rehome on 
its own. But this is never going to be as efficient as having the 
transport layer do it, as transport protocols have very good knowledge 
about what's happening end-to-end. TCP for instance goes through great 
lengths to be able to determine when to retransmit and whether only a 
single packet was lost or more. Streaming protocols on the other hand 
have a pretty good idea when new data should be arriving, so here the 
receiver is in a good position to send nacks or hints.

One thing we haven't discussed so far: if upper layers provide us with 
a hint, what exactly does the multihoming layer do after receiving such 
a hint? It would make sense to peform some kind of check to see what 
kind of reachability exists, but this means we need some kind of 
ping-like functionality. Is it reasonable to depend on such a mechanism 
in this age of ICMP paranoia?

About the rewriting: why again are we making life difficult for 
ourselves? The obvious place to put an indication that the address may 
be rewritten is... in the address. Is there any reason why we can't 
have one or more special prefixes that indicate that a router should 
fill in the source address?

However, this doesn't solve what we should do when rewriting isn't 
permitted. Obviously we could come up with a multihoming mechanism 
where rewriting is always allowed, trading off complexity in this area 
against complexity in recognizing a correspondent and accepting some 
types of spoofed packets. But "legacy" IPv6 also doesn't permit 
rewriting, so we must be prepared to handle this. The obvious solution 
is source address based routing, but it doesn't seem like everyone is 
convinced.

I don't really see any workable alternatives, though. Even if we can do 
ICMP or NAROS magic to make sure that new sessions magically use the 
right source address so initially they're able to pass ingress 
filtering, it's always possible that halfway through the session is 
rerouted over another ISP and the source address is filtered. This 
means we can't reroute traffic based on changes in BGP the way we're 
used to, the ultimate consequence of which is that we must hardcode all 
routing decisions. In practice this probably means using one ISP as a 
primary and the other one only as a backup.

Another problem is that if we depend on BGP to determine which ISP 
provides the shortest path to a destination, this effectively blocks us 
from using the other ISP to reach the same destination. For instance, 
if X has ISPs A and B, and Y has ISPs C and D, then it's entirely 
possible that X will use ISP A to reach both Y(C) and Y(D), so that 
when something bad happens with ISP A both of Y's addresses become 
unreachable. With source address based routing this isn't an issue as X 
can reach each of Y(C) and Y(D) over both A and B by simply using a 
different source address.

I do agree that source address based routing (which is in effect a 
limited form of source routing) doesn't mesh well with our hop by hop 
forwarding paradigm, but again: I don't see any alternatives.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Oct 2003 11:51:17 +0100

I agree with this, and I'd add that many applications can survive much
longer glitches than 5 seconds, and even TCP resets, by putting some fairly
trivial retry logic in the right place. In the VoIP case, the limit is indeed
the social one; there is no technical reason a VoIP solution can't
keep trying to reconnect indefinitely.

However, I suspect that this whole discussion belongs in a future
working group called multi6-ops. Once we get a basic mechanism agreed,
there will be many operational and implementation issues to be
followed up. Should we concentrate now on the basic mechanism?

   Brian

---

From: Pekka Nikander <[email-masked]>
Date: Fri, 31 Oct 2003 13:05:35 +0200

Iljitsch van Beijnum wrote:
> be rewritten is... in the address. Is there any reason why we can't have 
> one or more special prefixes that indicate that a router should fill in 
> the source address?

I think this is an excellent (but by no means new :-) idea.

In general, I think that we need a generic way of expressing
that the site border routers may rewrite the source address
in the packet.  There are cases where that would work already
now.  For example, the revised IPsec specifications allow IPsec
SAs to be set up so that the source address is ignored when the
packet is received.

If I understand correctly, the main reason why this idea was
rejected earlier was that it breaks upper layer pseudo header
checksums.  However, in the cases that we are dealing with now,
that isn't true any more (IPsec tunnel mode, the proposed IPsec
BEET mode, HIP, NOID, SIM, and probably others).

Hence, I really think that we should go for standardizing
a prefix that asks the first capable/so configured router
to fill it in.

--Pekka Nikander

---

From: Pekka Savola <[email-masked]>
Date: Fri, 31 Oct 2003 13:16:46 +0200 (EET)

On Fri, 31 Oct 2003, Brian E Carpenter wrote:
> trivial retry logic in the right place. [...]

(I'm pretty sure you agree here, but playing the devil's advocate to bring 
up an important point here..)

Is it the business of the applications to put in this retry logic?

No.

If *every* application has to do this, we've failed.  If such adding such
logic is deemed the best approach, it needs to be put somewhere else.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "Spencer Dawkins" <[email-masked]>
Date: Fri, 31 Oct 2003 06:43:19 -0600

Ummm, exactly.

While I would *never* utter the word "r_quirement" on the multi6
mailing list, it might be useful to think about success criteria.

It's blindingly obvious to me that multi6 can't meet the needs of
every application (for the same reason general-purpose security
mechanisms might not meet the needs of the most extremely paranoid),
and it's blindingly obvious to Pekka that multi6 has to meet the needs
of at least some applications (and I agree). Neither of these are
controversial positions. What's the success criteria, really, between
these two extremes?

Spencer

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Oct 2003 13:51:30 +0100

> logic is deemed the best approach, it needs to be put somewhere else.

That is what I would have said a few years ago. But the fact of life is
that TCP resets do occur, and if you are building a business class
application you will *not* allow that to cause an applications level
failure. So all the business class applications that I know already
have retry logic, and it was put there by programmers who wouldn't know
a multihoming event if it hit them in the face.

Actually it's just an extension of the fate sharing argument. If the host
hasn't actually crashed and burned, it should try again at successively
higher levels of the stack until things work again.

That's why I've always rated transport survivability as only "nice to have"
in multihoming.

   Brian

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 14:38:19 +0100

Hi Iljitsch,

I very much agree with you here.
Some comments below...

> approach.

At least we should explore what we can achieve following this path.

> Yes, this will give us some trouble in the beginning, as
> about what's happening end-to-end.

Agree

> a hint?

I thought this was what we were discussing with Erik in the Preserving
established sessiones thread :-)

 It would make sense to peform some kind of check to see what
> in this age of ICMP paranoia?

IMHO, i would like to explore alternative approaches first.
What about when recieving a ULP hint that there is a problem, change both
source and destiantion locators and send the packet with rewrite ok bit not
set?
A more aggresive option would be to send multiple packets in paralel, each
one with a different combination of source and destiantion address, and see
which reply comes first.

> where rewriting is always allowed,

I don't see how.

I mean, we are going to have two mechanisms, one based on bgp + rewriting
and the other one is based on ULP hints.
Now it is expected that the ULP hints mechanism will be have a better
response time i.e. it will detect problems faster, right?
However, the ULP hints mechanism works in the host itself and the
bgp+rewrite mechanism is in the routers. This means that even if the ULP
hint mechanism has detected a problem and changed the locator, if the final
word about this belongs to the routing system, the bgp+rewriting mechanism
will overrule the decisions made by the ULP hint mechanism which is exaclty
what we don?t want, since the ULP hint mech is faster)
So we need a mechanism to force the routing system to accept choices made by
the host, since the host is better detecting problems in the path.
Such a mechanism seems to require the capability to express that rewriting
is not permited.

Note that only the ULP hints mechanism is not enough, since for a while we
will have ULP that don`t provide hints, so we need a fallback mechanism

 trading off complexity in this area
> I don't really see any workable alternatives, though.

Well, we could use a routing header. Even we could define a routing headers
that is elimated by the exit routers, in order to addresss the MTU reduction
problem.

> Even if we can do

agree

> different source address.

agree (again)

> forwarding paradigm, but again: I don't see any alternatives.

An obvious comment about source address based routing is that source address
based routing only has to be implemented within the multihomed site.

Thanks, marcelo

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 15:02:22 +0100

There is middle ground between letting the application to deal with the
preservation of the established communications (handling locator discovery,
security) and letting all this to be performed by the multi-homing layer.
Some parts seem to be pretty common to all apps, locator discovery, security
Some parts seem to be very different such as failure detection.
So the common parts should be provided by the common multi-homing layer and
the different parts should be ULP specific.
IMHO, the issue is how to make the interaction of those mechanisms
Regards, marcelo

> -----Mensaje original-----
> De: [email-masked] [mailto:[email-masked]]En
> nombre de Brian E Carpenter
> Enviado el: viernes, 31 de octubre de 2003 13:52
> Para: Multi6 Mailing List
> Asunto: Re: survivability, rewriting
>
>
> Pekka Savola wrote:
> >

---

From: Cedric de Launois <[email-masked]>
Date: Fri, 31 Oct 2003 15:25:45 +0100

On Fri, 2003-10-31 at 14:38, marcelo bagnulo wrote:
> IVB> I do agree that source address based routing (which is in effect a
> IVB> limited form of source routing) doesn't mesh well with our hop by hop
> IVB> forwarding paradigm, but again: I don't see any alternatives.
> 

Note that source address based routing also provides these advantages :
- since the path (and the ISP) taken by an outgoing packet is determined
  by its source address, a host can easily select its preferred path by
  simply choosing the corresponding source address.
  We can thus imagine a p2p application that could explicitly
  load-balance its traffic on all its available paths. This is
  an interesting feature (I think).
- routing inside the multihomed site can be simpler since it doesn't
  longer depends on the destination address but only on the few 
  site prefixes delegated to the site.
- More importantly : source address based routing enables better
  failure detection and recovery that the use of bgp since all this
  stuff can be performed end-to-end (I agree with Marcelo)

Cedric

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 15:55:51 +0100 (CET)

> fill in the source address?

Two answers:
1. The ULPs need to know the addresses of the node itself to be able
   to perform certain forms of referal. But that doesn't prevent
   the node to issue IP packets with a source locator that doesn't
   include the upper bits of the address *as long as* the host knows
   that there is a rewriting router along the path.

2. Having a designated prefix in the source locator to indicate "rewrite ok"
   prevents hiearchical rewriting; this is when e.g. your border router 
   rewrites the prefix and your ISPs border towards its upstream can also 
   rewrite.

  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 15:02:17 +0100

Hi Brian,

> followed up. Should we concentrate now on the basic mechanism?

The problem is that perhaps the basic mechanism (for example one based on
source address rewriting) won't allow future optimizations if they are not
considered in the initial design.
So i guess that we should decide which type of future optimizations will be
supported and then ensure compatibility with the basic solution and let the
tunning of the optimizations for later.

Regards, marcelo

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 31 Oct 2003 16:08:15 +0100 (CET)

> based routing only has to be implemented within the multihomed site.

Is that true when you have a tree of reseller ISPs?

For instance

		DFZ
	    /     |    \
	ISP1	ISP2	ISP3
	/  \    /   \  /   \
     ISPA  ISPB      ISPC   ISPD
	      \      /
	       my site
	
  Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 18:00:44 +0100

> > An obvious comment about source address based routing is that
> >  source address based routing only has to be implemented
> > within the multihomed site.
>

Good question IMHO...

I guess this depends on how do you expect that provider multi-homing will
be.

I assume that in your example above you are considering the case where ISPA
obtains addresses form ISP1 and ISPB from (ISP1 and ISP2), and ISPC from
(ISP2 and ISP3), right?

This implies that my site has 4 prefixes and in that case source address
based routing is also required in ISPB and ISPC, correct?

But IMHO this is not how provider multihoming will be. IMHO ISPs will be
able to get a /32 each one, especially those ISP that are multi-homed to
multilpe upstream providers.
I guess that sub allocation will only make sense to isps that have a single
upstream provider.

I mean, i don't see signs that multihomed isps will have to obtain their
IPv6 address range from their upstream providerS, implying that those isps
will have several prefixes in their networks. (reducing the need of source
address based routing to the end sites only)
What is your view about this provider multihoming issue?

Regards, marcelo

   Erik

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 31 Oct 2003 18:00:45 +0100

>    that there is a rewriting router along the path.
I am sorry, Erik, but i don't understand this argument.
Those rewritable addresses wouldn't be passed for referrals purposes, they
are just an artifact to let the M6 layer to notify the border router that
they should complete with a real prefix. The application would only be aware
of the AID. The referals would contain real locators. I guess i am missing
your point here :-(

>    rewrites the prefix and your ISPs border towards its upstream can also

In what situation do you think that hierarchical rewriting would be needed?

I guess you are considering the small multi-homed provider that obtains
multiple prefixes, right?
Again, i don't believe this would be the way provider multi-homing will be

Regards, marcelo