Subject: ISP failures and site multihoming [Re: Enforcing unreachability of site local addresses]

From: Pekka Savola <[email-masked]>
Date: Thu, 20 Feb 2003 11:32:55 +0200 (EET)

Hello,

I'll take one particular issue, and Cc: to multi6 as I believe it is a 
very important thing to consider.
 
On Fri, 14 Feb 2003, Alan E. Beard wrote:
> > > Most of the end-user-network managers among my clients now multihome,
> > > and
> > > will continue to require multihomed service in future.  In every case
> > > where the user's network is multihomed, the multiple independent
> > > connections are seen as crucial for maintenance of high availability of
> >
> > [Kurtis:]
> > I find this funny. A number of studies have shown that if this is what
> > you are after, multihoming and BGP is the wrong way to go - but never
> > mind.
> >
> Your comment may be true, but my clients are nonetheless unwilling to risk
> the possibility of an extended network outage on a single ISP (while not
> frequent, these events are far from unprecedented) rendering their online
> customer-support environment unavailable for several hours, much less for
> a day.  Shorter outages (on the order of minutes in the single digits) are
> tolerated, provided that such outages are infrequent.

This is a very problematic approach IMO.

Need more resiliency?  Network outages unacceptable?

The right place to fix this is the network service provider, period.  
Nothing else seems like a scalable approach.

Consider a case when many companies _phone_ services would have been
changed to VoIP.  IP would be a critical service.  Do the enterprises
protect against failures by getting more ISP's?  Unscalable.  No, the
ISP's _must_ get better.  Pick one well when choosing them.

When ISP's have SLA's, a lot of customers for which continued service is
of utmost importance, the networks *will* work.  There is just no other
choice.  If the mobile phone of CTO, CEO or whatever rings after (1)5
minutes of network outage, things _will_ happen.

It just seems the mentality in some networks is that network outages are 
ok, networks don't have to be designed with multiple connections, etc.etc.

That must change if we want to build a mission-critical IP infrastructure.  
Instead of making every site try to deal with the problems themselves.

This is my view as ISP and an end-user.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 20 Feb 2003 11:52:14 +0100 (CET)

> Nothing else seems like a scalable approach.

There is no technical reason why a single service provider network can
do better than a similar network that consists of several smaller
service provider networks. Sure, BGP as-is doesn't provide the seamless
failorver some people would like. It annoys me to no end that Cisco uses
a 180 second default hold time for BGP, twice the already too
conservative value that is is suggested in the RFC. This means that when
a circuit goes down BGP takes two or three minutes to notice this. I
always recommend configuring a hold time of 15 seconds, but it seems
some vendors have designed their stuff in such a way that sessions can
fail with this value when the box is busy. But IGPs have the same
fundamental problem (although the details may differ). OSPF for instance
takes 40 seconds to detect a dead circuit.

> ISP's _must_ get better.  Pick one well when choosing them.

We are _very_ far from a situation where even the best ISP provides a
service level that is better then the one you get from multihoming even
if you consider failover delays.

Also, these approaches aren't mutually exclusive. ISPs should get
better. Multihoming should get better.

At the same time, we should recognize that it is simply impossible to
have the same failover delays at layer 3 as at layer 1.

> minutes of network outage, things _will_ happen.

My experience with SLAs is that they are a marketing tool and job
security for bureaucrats. They don't make the worst case any better,
they only make the worst case slightly less frequent.

(What makes you think this mobile phone will ring anyway? Speaking of
unreliable networks...)

And the single service provider thing doesn't scale anyway: the end
result would have to be a single global ISP.

> Instead of making every site try to deal with the problems themselves.

Has the end-to-end principle failed to teach us anything? Reliability
begins and ends in the end hosts. If each host is connected over two
service providers there are four possible paths the hosts can switch
between on a per-packet basis. Then the only problem becomes detecting
failure. The end hosts are in an excellent position to do this without
having to generate keepalive messages; a well designed protocol could
switch to an alternate path within a few round trip times when a path
failure occurs.

Multi6 has been gravitating towards multi-address multihoming solutions
for a while now, but unfortunately it seems impossible to move foward.

---

From: Lars Erik Gullerud <[email-masked]>
Date: 20 Feb 2003 16:06:08 +0100

> Nothing else seems like a scalable approach.

In a perfect world I'm sure I'd agree with you. In real life however,
the fact of the matter is that customers want multihoming, and it
doesn't matter to the customers if that is a problematic approach that
doesn't scale for the SPs. Doesn't even matter if it's technically the
best solution for THEM, customers are not known for picking the best
solutions, nor listening to their providers suggestions for better ones,
half the time. And, the simple fact of the market economy is that what
the customers want, the providers are going to sell.

Making the IP backbones more resilient costs money. You get that money
from your customers. You get the customers by selling them what they are
asking for. What they are asking for is multihoming. If they can't
multihome with IPv6, well, then they won't use IPv6 until they can. If
the customers don't want to use IPv6, the providers won't spend the
money to support it.

...and this is yet another rehash of a discussion that has already been
had so many times by so many people that I'm sure we can just copy &
paste from here on out.

The fact of the matter is, whether it's the best approach or not, we
need a solution for customers to multihome. So let's bring our attention
back to that, shall we?

/leg

---

From: Pekka Savola <[email-masked]>
Date: Thu, 20 Feb 2003 17:24:00 +0200 (EET)

On Thu, 20 Feb 2003, Iljitsch van Beijnum wrote:
> service provider networks.  [...]

Of course not -- but it's *much* easier.

> if you consider failover delays.

In some cases, this may be better.  In some others, not.

It's not IMO necessary to get significantly better but "roughly equal".
 
> (What makes you think this mobile phone will ring anyway? Speaking of

Some means of contacting will always exist, or the monitoring system of 
ISP's has developed so far the problems in the IP service are extremely 
rare.

> result would have to be a single global ISP.

It does scale, pretty well actually.  I'm not talking about your average
neighborhood ISP's with 100 customers, though.  Currently in DFZ, there
are about 3500 (ONLY!) AS numbers which transit at least one other AS
number.

Even multiplying that with 10, we would not have a problem.
 
> switch to an alternate path within a few round trip times when a path

Compare this to a solution where the site has two connections to the same 
ISP, and you're left with major ISP backbone failure or upstream failure 
(any relevant ISP's have only one upstream)?

Doesn't sound that difficult to me, particularly as these problems affect 
the whole (or majority) of the ISP -- and hence, are fixed quickly.

A solution without multi-connecting, ie. only one L1 connection to one 
ISP, is naturally out of question.
 
> for a while now, but unfortunately it seems impossible to move foward.

Multi-address solutions solve certain problems well, but leave some 
unsolved.  Coupling multi-addressing with multi-connecting, you have a 
very comprehensive solution, IMO.

---

From: marcelo bagnulo <[email-masked]>
Date: 20 Feb 2003 20:03:18 +0100

I haven't included ipng since i guess this is out of scope for them.

Hi Iljitsch,  

> having to generate keepalive messages; 

How?

Thanks, marcelo

> a well designed protocol could
-- 
marcelo bagnulo <[email-masked]>
uc3m

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 20 Feb 2003 20:13:03 +0100 (CET)

> It's not IMO necessary to get significantly better but "roughly equal".

Don't forget that failover is just one of the benefits of multihoming.
Two important others are protection against losing service when ISPs
go out of business and more optimal traffic flow.

> are about 3500 (ONLY!) AS numbers which transit at least one other AS

I don't see your point. What you were saying is that using a single
reliable ISP would be better than multihoming. Now obviously an ISP can
only control the QoS parameters inside its own network so if you want to
do reliable and high quality VoIP you need to use the same ISP
end-to-end. In other words: just one ISP.

> (any relevant ISP's have only one upstream)?

So ISPs get to multihome but not end-users?

There are many ways in which an ISP network can fail, as the large scale
Worldcom and AT&T outages six months ago illustrate. More common is the
situation that an ISP network has trouble reaching a certain destination
because the only link to that destination has failed (which in itself
may not be their fault) or there is congestion somewhere. And don't
forget maintenance windows.

> ISP, is naturally out of question.

Ok, so why is multihoming to a single ISP better than multihoming to
several ISPs?

> unsolved.

Like what?

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 20 Feb 2003 20:19:42 +0100 (CET)

On 20 Feb 2003, marcelo bagnulo wrote:

> > If each host is connected over two

> How?

With TCP you simply monitor incoming ACKs. If there aren't any for a
while, when they are expected, something must have gone wrong -> switch
to an alternate path. UDP-based protocols can use a similar mechanism,
but they often already know how to fall back to a secondary address.

The main problem is not switching to an alternate path too aggressively
when some packets are lost or there is an increase in delay.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 20 Feb 2003 21:53:47 +0200 (EET)

On Thu, 20 Feb 2003, Iljitsch van Beijnum wrote:
> go out of business 

Sure, independence is another issue -- not related to ISP failures, and 
there are ways to protect against this.

> and more optimal traffic flow.

I agree, but other than really simple traffic flow balancing (read: 
multiple addresses) is a difficult thing to do, and people don't often 
really do it.
 
> reliable ISP would be better than multihoming.

Considering the tradeoffs, yes.

> Now obviously an ISP can

This seems like a ridiculous argument or I'm missing something.  You could
just use VoIP up to the ISP, have ISP's coordinate the parameters (if you
need any), etc.  This is no different than multihomed scenario.
 
> So ISPs get to multihome but not end-users?

Yes.
 
> Worldcom and AT&T outages six months ago illustrate. 

I'm not aware of the case, so I can't really comment.  Pointers?

> More common is the
> may not be their fault) or there is congestion somewhere. 

This seems no different than the case when using BGP site multihoming -- 
unless you want the fine-tune your policy per destination -- a 
non-starter.

> And don't

No real ISP has maintenance windows that seriously affect all
communications at once.
 
> Ok, so why is multihoming to a single ISP better than multihoming to

Fewer tradeoffs: you can protect against most failure modes, while not 
having to hae AS, your own IP addresses etc -- that is, it doesn't require 
bloating the DFZ!
 
> > Multi-address solutions solve certain problems well, but leave some

Solves: operator independence

Doesn't solve: connection survivability, short-term failures, more than 
basic TE [mostly a non-issue IMO]

Mixing multiconnecting to one ISP and having a backup to the second one
gives you mostly everything.

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Fri, 21 Feb 2003 16:19:36 +0100

> This is my view as ISP and an end-user.

This highlights another issue with the solution to the multi6 problem - 
convergence time. We need a solution that also improves this, besides 
scales in the DFZ.

- kurtis -

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Fri, 21 Feb 2003 16:29:22 +0100

> do better than a similar network that consists of several smaller

See Abha and Craigs paper on convergence of BGP. Personally I would go 
for a large provider with multiple connections.

Last fall I was invited to a conference in Sweden to debate multihoming 
and the enterprise. Before me was this enterprise IT manager who showed 
how much more resilient his network was with two BGP sessions. While he 
talked I checked his announcements just to find that one of the 
providers bought transit from the other. You can't buy clue.

> takes 40 seconds to detect a dead circuit.

there was a fix proposed in San Diego (although for IS-IS) but that was 
voted down. There was pros and cons.

> if you consider failover delays.

I would like to see numbers for this. My experience says otherwise. 
Perhaps CAIDA have something on this? Or the RIPE RIS boxes could give 
some insight.

> better. Multihoming should get better.

Yes, but not for the same reasons.

> have the same failover delays at layer 3 as at layer 1.

 From my experience you can get very close, but I am very close to a NDA 
I have signed.

> they only make the worst case slightly less frequent.

To some extent I agree. There was inflation in SLAs some year ago. This 
was bad for the entire industry.

- kurtis -

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Fri, 21 Feb 2003 16:35:15 +0100

> the customers want, the providers are going to sell.

....which explains a lot of bad stuff being deployed, but that is 
another WG..:=)

> the customers don't want to use IPv6, the providers won't spend the

Agreed!!!

> need a solution for customers to multihome. So let's bring our 

Agreed.

I have a suggestion....:)

- kurtis -

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 22 Feb 2003 10:31:04 +0100 (CET)

> for a large provider with multiple connections.

Based on this paper? What I see is rarely as bad as what they describe.
However, I had the chance to experiment a little with revoking a
longer prefix and then see how soon the shorter prefix would "catch" the
traffic a while back, and this was certainly interesting: the state goes
back and forth between "working" and "not working" several times over
the course of two minutes. But simple failover is usually pretty fast.

If failover times when a circuit goes down are your main concern,
connecting to the same ISP twice makes a lot of sense. (Use a
non-switched network for this, though, or you'll be at the mercy of the
hold time.) On the other hand your failover time when in case of an ISP
failure is much worse this way. Personally, I'd rather eat the two
minute failover and be protected against the two week one than the other
way around, even if the former happens once every few months and the
latter may not happen at all.

> providers bought transit from the other. You can't buy clue.

You can buy a good book that explains it all.  :-)

Did you check to see if the second ISPs also had additional upstreams?

> > But IGPs have the same
> > fundamental problem (although the details may differ). OSPF for
> > instance takes 40 seconds to detect a dead circuit.

> voted down. There was pros and cons.

Just type:

 ip ospf hello-interval 1
 ip ospf dead-interval 3

But do it on ALL your boxes in the subnet or you'll live to regret it.

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Mon, 24 Feb 2003 09:27:21 +0100

> the course of two minutes. But simple failover is usually pretty fast.

My experience is different, and I believe many others share that. But 
this will always be different for everyone.

> Did you check to see if the second ISPs also had additional upstreams?

Yes they did. And they bought transit from the first provider.

> But do it on ALL your boxes in the subnet or you'll live to regret it.

This I thought was more or less standard. I was talking about less than 
100ms convergence.

- kurtis -