Subject: [lisp] Questions about draft-saucez-lisp-itr-graceful-03

From: Marc Binderberger <[email-masked]>
Date: Tue, 18 Feb 2014 14:48:25 -0800

Hello Damien/Olivier/Luigi/Clarence & LISP experts,

had a look at draft-saucez-lisp-itr-graceful-03. And wonder if there is 
more to come? Somehow section 4 feels a bit "short".

What I mean: if you try to solve the problem of the _two_ cache-miss 
storms - first on the 2nd ITR (ITR2) when your restarting ITR (ITR1) 
goes down, then on the restarting ITR1 when it picks up traffic again - 
then section 4 would probably need to talk about a permanent cache 
synchronization (?). Unless you want to solve a planned restart only. 
But for a failure of the ITR1 I don't see how the solution you describe 
would work

   o  ITR cache synchronization: upon startup, the ITR synchronizes its
      cache with the other ITRs in its synchronization set.  The ITR is
      marked as available only after the cache is synchronized.

as ITR2 would trigger the cache-miss storm for the traffic after ITR1 
failure.

Or if you want to solve only the cache-miss storm when ITR1 comes back 
into the traffic stream then the ITR deflection has the advantage to 
not require any cache-synchronization protocol, IMHO. The rate of 
Map-Requests could be throttled to turn the storm into a breeze. The 
method how to transport traffic to ITR2 could be one of many - a direct 
LAN, GRE, Lisp.

So my question in short: are you planning to add some words about a 
permanent cache synchronization?

Thanks & Regards,
Marc

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 18 Feb 2014 15:45:53 -0800

> method how to transport traffic to ITR2 could be one of many - a direct 

Or just make it a local matter and have ITR1 read its checkpoint file that it had written the last time before it crashed. These sort of problems could be solved better with implementation design and not protocol design.

Dino

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 09:38:54 +0100

Hello Marc,

On 18 Feb 2014, at 23:48, Marc Binderberger <[email-masked]> wrote:

> more to come?

Thank you for the interest.  We are indeed thinking on ways to extend
the document and provide more details on the ways the solutions could
be implemented.

> Somehow section 4 feels a bit "short".
> 

For now we don't have acceptable techniques to keep caches
synchronised in a permanent way but I don't think it is a big issue as
the purpose of the document is to deal with planned restart of routers
meaning that we know exactly when the routeur will get down then up
(it is controlled by the operator).

If you have a solution to continuously synchronise ITRs caches, we
would be very happy to look at them and integrate them in the proposed
solution.

Thank you,

Damien Saucez

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 09:40:51 +0100

> Or just make it a local matter and have ITR1 read its checkpoint file that it had written the last time before it crashed. These sort of problems could be solved better with implementation design a…

As a matter of fact, this is probably the simplest solution.  However
that implies that routers are down for period of time shorter than the
lifetime of entries in the cache.  Unfortunately, this solution only
prevents storms for the startup, not for the shutdown.

Damien Saucez

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 08:39:47 -0800

> prevents storms for the startup, not for the shutdown.

I am not following your logic.

Dino

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 17:58:04 +0100

> I am not following your logic.

1.  if you store the cache for next reboot, you will not experience
the miss storm when the traffic will come back to you.

2.  if you shutdown an ITR, packets are forwarded to another ITR and
there is a miss storm as long as the prefixes in the backup ITR do not
cover those that where in the "down" ITR.

Damien Saucez

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 09:18:38 -0800

> cover those that where in the "down" ITR.

LISP promotes multi-homing for inbound traffic to a LISP site, as well as outbound packets from a LISP site. You deploy LISP so you can get active-active multihoming.

So this storm will typically be academic.

Dino

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 18:26:34 +0100

> So this storm will typically be academic.

Well it is not just theoretical, it can happen as soon as you have two
egress points, even in active/active mode.

Imagine you have two identical egress points in your network (let say
that they can both reach the whole Internet) depending on the IGP the
traffic from a part of the network will go to one and the traffic from
the other part of the network will go to the second router.
Unfortunately the set of destination is not the same in both part of
the network so when you fallback to one router after the outage of the
other one, you will have misses.

We have evaluated that on our network (that is primary/backup) and
simulated it as active/active and noticed that the storm would not
negligible.

Damien Saucez

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Wed, 19 Feb 2014 17:29:54 +0000

> So this storm will typically be academic.

+1 

I've always thought that the mapping system will have to have capacity to deal with broken clients, random LIG bots, and DoS attacks (just like, say, DNS does).  This leads me to believe that over-engineering the management of sending map-requests on a given ITR will likely end up creating more complexity, and therefor fragility, in the implementation.

-Darrel

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 18:31:31 +0100

> I've always thought that the mapping system will have to have capacity to deal with broken clients, random LIG bots, and DoS attacks (just like, say, DNS does).  This leads me to believe that over-…

That's a point we can discuss, but are you ready to accept packet loss?
At the end, a miss = a packet drop...

Damien Saucez

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 09:32:58 -0800

> simulated it as active/active and noticed that the storm would not

Yes, understand Damien. One has to decide how much machinery one puts in a protocol for events that don't happen often.

Dino

---

From: "Darrel Lewis (darlewis)" <[email-masked]>
Date: Wed, 19 Feb 2014 17:38:10 +0000

On Feb 19, 2014, at 9:31 AM, Damien Saucez <[email-masked]> wrote:

> At the end, a miss = a packet drop…

"Screws fall out all the time, the world is an imperfect place." --John Bender

Seriously though, I'm perfectly willing to accept packet loss, but I recognize that there may well be people who won't. 

-Darrel

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 18:38:49 +0100

> Yes, understand Damien. One has to decide how much machinery one puts in a protocol for events that don't happen often.

I could offer you a solution to avoid packet drops, but you might not
like.  It starts with a "b" and finishes with a "uffer" :-)

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 19 Feb 2014 11:17:47 -0800

Hello Damien,

thanks for the reply!

> would be very happy to look at them and integrate them in the proposed

And I was curious to see a light-weight protocol extension from you :-)
Seriously, was wondering if you see an elegant, light way to implement 
this in the LISP protocol (?). 

> (it is controlled by the operator).

but then the "Traffic deflection to other ITRs (or a PxTR)" could be 
used to fill the cache of the 2nd ITR (the one that is not reloaded). 
You turn it on on ITR2 (off on ITR1), change your IGP to send all LISP 
data to remote sites to ITR2, "wait a bit", then ITR2 should be ready, 
you turn off deflection on ITR2 and reload ITR1. Then turning on 
deflection on ITR1 and bring the IGP routing back to active-active (or 
whatever the setup was before).

Regards, Marc

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 11:41:19 -0800

> this in the LISP protocol (?). 

Light-weight reads as non-robust and scalable. If you want those things, you have to do it right. And you then implemented BGP. 

One reason people like LISP is because it is reasonably easy to understand and employs *less protocol machinery* rather than more.

> used to fill the cache of the 2nd ITR (the one that is not reloaded). 

Then you get sub-optimal routing.

> data to remote sites to ITR2, "wait a bit", then ITR2 should be ready, 

This is easier said then done. That means you have to inject *all remote EID-prefixes* into your IGP. That is a non-starter.

Dino

> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 19 Feb 2014 11:53:05 -0800

Hello Dino et al.,

> This is easier said then done. That means you have to inject *all 
> remote EID-prefixes* into your IGP. That is a non-starter.

maybe I think too simple. Assuming you have two xTRs to connect your 
site to the LISP cloud. They both originate a default route into your 
site IGP. You then e.g. increase the metric of ITR1's default route or 
remove the default originated into the site IGP. Routing out of the 
site (to another EID) then moves to ITR2.

Ingress is a different story, probably you need to reduce TTL for 
registrations sent from ITR1, so you end up traffic ingress will use 
ITR2 only (?).

Then you are ready to reload ITR1.

Long story short: using the "Traffic deflection to other ITRs" plus the 
right operational procedure may solve the problem?

Regards, Marc

On Wed, 19 Feb 2014 11:41:19 -0800, Dino Farinacci wrote:
> Light-weight reads as non-robust and scalable. If you want those 
> things, you have to do it right. And you then implemented BGP. 
> 
> One reason people like LISP is because it is reasonably easy to 
> understand and employs *less protocol machinery* rather than more.
> 
>> 
> This is easier said then done. That means you have to inject *all 
> remote EID-prefixes* into your IGP. That is a non-starter.
>

---

From: Damien Saucez <[email-masked]>
Date: Wed, 19 Feb 2014 21:29:05 +0100

> On 19 Feb 2014, at 20:17, Marc Binderberger <[email-masked]> wrote:
> 
> Hello Damien,
> 
> thanks for the reply!
> 

Well directly using LISP maybe we could imagine something with map-notify and multicast to keep caches synchronized but I have to think more about that.

Damien Saucez

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 13:46:39 -0800

Yes, what you describe can work. But once you deflect, the other ITR still needs to send Map-Requests for all the new EIDs that are not cached in the map-cache.

Dino

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 19 Feb 2014 16:40:03 -0800

Hello Dino et al.,

> Yes, what you describe can work. But once you deflect, the other ITR 
> still needs to send Map-Requests for all the new EIDs that are not 
> cached in the map-cache.

True. Two options I see

(a) rate-limit the map-requests from the just-reloaded ITR. All this 
does is some EIDs are a bit longer deflected

(b) as Darrel explained it to me: if the MR/MS/mapping system cannot 
handle this from a single site then it's probably too weak and not fit 
for the job :-)

Regards, Marc

---

From: Florin Coras <[email-masked]>
Date: Thu, 20 Feb 2014 03:13:51 +0100

> handle this from a single site then it's probably too weak and not fit

What Dino meant, I think, is that all active, egressing flows passing 
through the ITR to be reloaded (ITR1 in your example) will cache miss in 
the backup ITR. I don't know if the problem you try to solve is is 
theoretical or practical, but in the latter case, maybe it would be 
easier just to provision a local caching Map-Resolver, close to the two 
ITRs.

Florin

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 19 Feb 2014 18:24:50 -0800

Hello Florin,

> is that all active, egressing flows passing 
> through the ITR to be reloaded (ITR1 in your example) will cache miss 
> in the backup ITR.

no, we covered the operational steps to make sure the backup 
(new-backup, actually active during reload) has learned "everything".

But the storm from the reloaded ITR does not go away, I agree. All the 
procedure fixes is the packet loss. And as I mentioned one could 
throttle the map requests from the reloading ITR as the delay in 
resolving EIDs does not hurt, due to the "deflection".

Regards, Marc

On Thu, 20 Feb 2014 03:13:51 +0100, Florin Coras wrote:
> through the ITR to be reloaded (ITR1 in your example) will cache miss 
> in the backup ITR. I don't know if the problem you try to solve is is 
> easier just to provision a local caching Map-Resolver, close to the 
> two ITRs.
> 
> Florin
>> 
>> Regards, Marc
>> 
>> 
>> 
>>> Dino
>>>

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 19 Feb 2014 19:17:17 -0800

Hello Dino et al.,
> does is some EIDs are a bit longer deflected

Already speced.

> handle this from a single site then it's probably too weak and not fit 

Already speced.

Dino

---

From: Luigi Iannone <[email-masked]>
Date: Thu, 20 Feb 2014 10:34:59 +0100

Hi,

>> So this storm will typically be academic.

As Damien pointed this is not just an academic exercise but something that may happen very easily.

How bad are the consequences is just a matter of deployment/reliability requirements. For some people it may represent a big deal, for other it is not even an issue. 

> I've always thought that the mapping system will have to have capacity to deal with broken clients, random LIG bots, and DoS attacks (just like, say, DNS does).  This leads me to believe that over-…

I agree that we do not need (and hopefully will not) over-engineer, but may be we can find very simple solutions for people that want to use it.

L.