Subject: [lisp] Adding a "Distance Server" to the Map Resolver, to support scalable "anycast" and "disaster recovery"

From: Robin Whittle <[email-masked]>
Date: Thu, 23 Apr 2009 19:02:09 +1000

On the RRG list:

    http://www.ietf.org/mail-archive/web/rrg/current/msg04907.html

As part of an RRG discussion with Bill Herrin, I created the idea of
a "Distance Server" by which an ITR could choose (or have already
chosen by something else) the closest (in BGP terms) ETR of multiple
ETRs in the mapping.

Bill suggested that an ITR can already do this, but it can't.

LISP or Bill's TRRP could have this as a separate server, but it
could also be integrated into LISP-ALT's Map Resolver.

For Ivip, I would integrate it into the full database Query Server
QSD.  For APT, the logical thing is to integrate it into the Default
Mapper.  The same thing could be used for a core-edge elimination
scheme, but I think core-edge separation schemes are the best approach.

Without a "Distance Server" or similar, neither a core-edge
separation scheme nor a core-edge elimination scheme can perform the
equivalent of the existing, conventional BGP-based "anycast" in a way
which scales better than the existing BGP approach.  This BGP
approach scales very poorly because it relies on a separate BGP
prefix for every such set of "anycast" servers.  See:

  http://www.ietf.org/mail-archive/web/rrg/current/msg04897.html
  http://www.ietf.org/mail-archive/web/rrg/current/msg04904.html

With a "Distance Server", I think LISP, APT, Ivip and TRRP etc. could
do this work very nicely, in a highly scalable fashion.

This goes beyond conventional anycast and into high reliability and
"disaster recovery" systems, which Bill gave an example of and which
I quote in my RRG message.

    - Robin

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 23 Apr 2009 11:48:06 +0200

Robin,
> chosen by something else) the closest (in BGP terms) ETR of multiple

This kind of idea is similar to what ALTO is supposed to develop. See
 http://www.ietf.org/html.charters/alto-charter.html

Olivier

-- 
http://inl.info.ucl.ac.be , UCLouvain, Belgium

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 10:46:56 -0700

Why not just pick an ETR, watch for return traffic and try to monitor  
RTTs between the sites for the data flow. Topological distance isn't  
as important as packet latency.

Dino

On Apr 23, 2009, at 2:02 AM, Robin Whittle wrote:

> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: Sam Hartman <[email-masked]>
Date: Thu, 23 Apr 2009 14:39:33 -0400

>>>>> "Dino" == Dino Farinacci <[email-masked]> writes:

    Dino> Why not just pick an ETR, watch for return traffic and try
    Dino> to monitor RTTs between the sites for the data
    Dino> flow. 
When would you transition to another etr?

So, its sounds like we're discussing anycast here.  In that space, the following concerns seem to be worth thinking about:

* Some anycast protocols only have one round trip
  ** balancing across all the possible destinatinos is important
  ** To the extent that closeness for some closeness metric matters, you need to get there on the first packet
* Some anycast protocols care a lot about stability: once you start with one destination you have to keep that destination
* TCP anycast is one limit of caring about stability

---

From: Sam Hartman <[email-masked]>
Date: Thu, 23 Apr 2009 14:42:51 -0400

Without speaking to whethre having some way to choose a best ETR for
the first packet is good or not, putting it in the map resolver seems
like the wrong place.  At least as I understand things it is a goal
that the map resolver be an optimization for people not connected to
the alternate topology and thus it should not have capabilities that
someone directly connected to the alternate topology.

So, it seems like you should either be saying: there is some
capability in the alternate topology I'd like to expose through the
map resolver.

or: There is some capability I'd like to add to the alternate topology
and then expose through the resolver.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 23 Apr 2009 15:00:20 -0400

There is quite a bit of information available, or ways to gather 
information, for ETR selection.  It is unlikely that a distance server 
(or an ALTO oracle) is either necessary or particularly useful.

If we are going to support anycast, doing so in the "id" space (the 
domain side of the mapping, instead of the range side of the mapping) 
does make sense to me.

However, I have a concern, related to the comments below about 
trnasaction.  The assumption up till now has been that the device 
picking the ETR can use any ETR it wants.  (Whether the chooser is the 
ITR or host software or whatever.)  And that it can even change that 
choice during the lifetime of a communication, with no impact on that 
communication.  (That's one of the benefits from the end users perspective.)

Anycast mapping entries (IDs, or whatever you want to call them) do not 
have that mutability property.  If you are talking to an entity using 
one entry mapped from an anycast, you can not expect to be able to 
continue the conversation with a different mapped entry.
It seems to me that having a conceptual entity in the architecture 
(whether it appears in the protocol(s) on the wire(s) or not is a 
different question) that represents the thing that you are talking to, 
with a way to relate that thing to the set of locators you can use 
interchangably to talk with that, seems a very useful construct.

If we want to handle anycast in the mapping, I would like to handle it 
in such a way (additional indirection?) that it does not obfuscate the 
notion of which locators for an interchangeable set for a communication 
session.

Yours,
Joel M. Halpern

---

From: Scott Brim <[email-masked]>
Date: Thu, 23 Apr 2009 15:28:06 -0400

> When would you transition to another etr?

What would the issue be?  Packets might get out of sequence but that
happens with every routing change and somehow applications do okay.

> So, its sounds like we're discussing anycast here.  In that space,
> the following concerns seem to be worth thinking about:

I don't think this has anything to do with anycast.  Anycast is about
a single address being used by multiple endpoints.  This is about a
single end prefix reachable via two different intermediate points.  It
has to do with choosing between alternative routes.  If a different
ETR is used, that will not change the endpoints that are being
communicated with.

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 12:41:00 -0700

> When would you transition to another etr?

I don't know. It would have to be local policy. But we have had to  
make these decision in fast-reroute algorithms as well. That is, if  
it's working and it's not broke, don't touch it.

I think you'll find from sites and ISPs alike that what they want with  
traffic engineering is a smooth usage of all their links. Rather than  
have hot spots on some links when other links are idle.

I really believe we don't have to be perfect and get this right. Best  
effort is really good enough.

> the following concerns seem to be worth thinking about:

Well, anycast RLOCs are interesting because it's like p2p protocols.  
That is, you just use an address and someone has to be up. So you just  
use it. Of course, your quality may vary.

With LISP, anycast RLOCs can be used by you have to not use the loc- 
reach-bits. Because once someone says the RLOC is down, you obviously  
stop using any ETR that is assigned the anycast address. But you do  
want to switch from an anycast RLOC to another *when the last anycast  
RLOC goes down*. And sine the anycast ETRs really may not talk to each  
other (they can in LISP when the ETRs are deployed at the site), this  
might be more trouble then it is worth.

>  ** balancing across all the possible destinatinos is important

Yes, and with the increase in the number of flows at a site, that will  
happen naturally.

>  ** To the extent that closeness for some closeness metric matters,  
> you need to get there on the first packet

If a destination site has RLOCs in the US and Europe and a LISP source  
site in Europe sends a map-request to the destination site, the  
destination site can return a map-reply with the Euro RLOCs with  
better priority then the US ones. So the Euro LISP site uses only the  
Euro RLOCs until they go down, in which case, the Euro site switches  
to the US RLOCs.

This is explained in draft-fuller-lisp-alt.

> * Some anycast protocols care a lot about stability: once you start  
> with one destination you have to keep that destination

I worked on AMT where anycast is used for discovery, but once you  
discover the server you latch on to a non-anycast address that you  
continue to use. This works as well when you use a transport protocol.  
Because the discovery with anycast addressing is done via packets and  
then when you get the latched real address (that doesn't move), you  
open the connection to the latched address.

Dino

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 23 Apr 2009 21:41:44 +0200

Dino,

> RTTs between the sites for the data flow. 

This is one possibility. Damien and Luigi demonstrated a technique to
pick ETRs based on additional information in
http://inl.info.ucl.ac.be/publications/interdomain-traffic-engineering-locatoridentifier-separation-context

Olivier

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 12:44:52 -0700

> the first packet is good or not, putting it in the map resolver seems

 From a LISP perspective, the best ETR is determined by the site and  
not by the cacher who sends packet to the site. And like I said in the  
previous email, the ETRs at the site can decide how to map-reply.

Remember, it's the site that pays for it's links. It should be the  
final decision maker on how packets arrive on those links.

> someone directly connected to the alternate topology.

It is more than an optimization, it will be the rule so we can deliver  
the ultimate in low opex multi-homing to stub sites.

> capability in the alternate topology I'd like to expose through the

The ALT just moves bits. And those bits are Map-Requests which are  
addressed to EIDs. That is the service it provides. And since it uses  
BGP, you can use all your BGP policy and security tools being  
developed for the underlying BGP infrastructure.

> and then expose through the resolver.

I would say probably not.

Dino

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 23 Apr 2009 21:45:04 +0200

Sam,

> So, its sounds like we're discussing anycast here.  

I don't think so. With LISP, one EID can be reached via several ETRs and
thus via different paths, but they all end at the same host for a given EID.

>In that space, the following concerns seem to be worth thinking about:
> 

Stability issues are likely to apply to LISP as well. Once an ITR has
started to use a given ETR to forward flows towards a given EID, it
should continue using this ETR for some time and not switch frequently
from one ETR to another for a given (Src EID - DST EID - port) flow.
It may of course load balance flows towards different ETRs.

Olivier

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 12:46:45 -0700

> If we want to handle anycast in the mapping, I would like to handle  
> it in such a way (additional indirection?) that it does not  
> obfuscate the notion of which locators for an interchangeable set  
> for a communication session.

Really good point Joel.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 12:48:18 -0700

> It may of course load balance flows towards different ETRs.

100% agree.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 12:52:06 -0700

Yes, I have read it. A find paper. I think there is potential for it  
but the Path Selection Mechanism needs to reside at the site. No one  
will allow a third party to decide how it's links should be used.

But if there is a box, which can be co-located with the xTRs that want  
to change the ranking, they could either start advertising them in Map- 
Replies or advertise those rankings to the map-server via Map-Register  
messages.

But at this point, we are not committing to having the map-server be a  
*proxy* map-replier for the site.

Dino

---

From: Scott Brim <[email-masked]>
Date: Thu, 23 Apr 2009 16:24:01 -0400

Excerpts from Joel M. Halpern on Thu, Apr 23, 2009 03:00:20PM -0400:
> Anycast mapping entries (IDs, or whatever you want to call them) do
> not  have that mutability property.  If you are talking to an entity
> using  one entry mapped from an anycast, you can not expect to be
> able to  continue the conversation with a different mapped entry.

LISP is encapsulation not tunnels.  If RLOCs are anycast addresses,
then once a packet leaves an ITR, the packet is forwarded based on how
intermediate routers reach that anycast address.  The ITR doesn't
control switching between ETRs if it's given an anycast RLOC to send
to.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 23 Apr 2009 16:27:40 -0400

>>>>> "Olivier" == Olivier Bonaventure <[email-masked]> writes:

    Olivier> I don't think so. With LISP, one EID can be reached via
    Olivier> several ETRs and thus via different paths, but they all
    Olivier> end at the same host for a given EID.

While that's all true, I thought I read in Robin's original message
that he was talking about scalable anycast over lisp.

I do agree that stability applies to situations outside of anycast.  I
think it is more critical in some anycast situations than in
non-anycast, but obviously there are reasons to.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 23 Apr 2009 16:34:54 -0400

>> Without speaking to whethre having some way to choose a best
    >> ETR for the first packet is good or not, putting it in the map
    >> resolver seems

    Dino> From a LISP perspective, the best ETR is determined by the
    Dino> site and not by the cacher who sends packet to the site. And
    Dino> like I said in the previous email, the ETRs at the site can
    Dino> decide how to map-reply.

Do you agree that Robin was proposing changing this in the case of map
resolvers?  If not, do you understand what he was proposing?

    >> like the wrong place.  At least as I understand things it is a
    >> goal that the map resolver be an optimization for people not
    >> connected to the alternate topology and thus it should not have
    >> capabilities that someone directly connected to the alternate
    >> topology.

    Dino> It is more than an optimization, it will be the rule so we
    Dino> can deliver the ultimate in low opex multi-homing to stub
    Dino> sites.

That sounds like an optimization to me.  A critical optimization, used
by most sites, but do you agree with my point that we don't expect
additional capibilities to be present in the mapping resolver not
present in the alternate mapping system.

    >> capability in the alternate topology I'd like to expose through
    >> the map resolver.

    Dino> The ALT just moves bits. And those bits are Map-Requests
    Dino> which are addressed to EIDs. That is the service it
    Dino> provides. And since it uses BGP, you can use all your BGP
    Dino> policy and security tools being developed for the underlying
    Dino> BGP infrastructure.

I agree with your statement, but do not understand how it is
responsive to mine.

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 23 Apr 2009 15:34:24 -0700

> resolvers?  If not, do you understand what he was proposing?

I do not understand what he is proposing.

> present in the alternate mapping system.

Is using /etc/hosts the DNS and using name-resolvers and name-servers  
an optimization? I think this is the analogue.

> I agree with your statement, but do not understand how it is

I'm saying you don't want to add more capability to the ALT. Because  
it has this benefit of being generic enough to use existing equipment  
from any host or router vendor.

If you had say, ".. there is some capability of the mapping system I'd  
like to expose ..." then I wouldn't have responded.

Dino

---

From: Sam Hartman <[email-masked]>
Date: Fri, 24 Apr 2009 11:05:58 -0400

>>>>> "Scott" == Scott Brim <[email-masked]> writes:

    Scott> Excerpts from Joel M. Halpern on Thu, Apr 23, 2009
    Scott> 03:00:20PM -0400:
    >> Anycast mapping entries (IDs, or whatever you want to call
    >> them) do not have that mutability property.  If you are talking
    >> to an entity using one entry mapped from an anycast, you can
    >> not expect to be able to continue the conversation with a
    >> different mapped entry.

    Scott> LISP is encapsulation not tunnels.  If RLOCs are anycast
    Scott> addresses, then once a packet leaves an ITR, the packet is
    Scott> forwarded based on how intermediate routers reach that
    Scott> anycast address.  The ITR doesn't control switching between
    Scott> ETRs if it's given an anycast RLOC to send to.

Joel, and I think Robin were talking about EID anycast, not RLOC
anycast.  I'll admit that I'm not really seeing the point in the
result of the mapping being an anycast address, although I agree that
if that happens, it should behave as you describe.  However I'm fairly
sure that Joel made it clear he was talking about providing anycast
service using LISP.  In that case, the EID would be an anycast
address, and the RLOC would reach one instance of that service.

Currently, I think the only person who may be proposing working on
this is Robin; and I may be misreading even his message.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 24 Apr 2009 11:39:36 -0400

Just to confirm, I do not think this is a topic that the LISP WG needs 
to spend any energy on.
Joel

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 24 Apr 2009 10:07:17 -0700

> Joel, and I think Robin were talking about EID anycast, not RLOC

Right, well my opinion about an EID being used as an anycast address  
basically has the same usage it does today. But with LISP the scope is  
within a site that assigns EIDs out of the EID-prefix it owns.

> result of the mapping being an anycast address, although I agree that

Well, there is an important point. You don't have to worry about RLOC  
liveness when you have anycast. You just pray and send packets and  
they will get to the destination site through one of the anycast ETRs.  
And if any of them go down or become unreachable, the core will  
reroute without any involvement from the ITRs.

> address, and the RLOC would reach one instance of that service.

There actually should be one more level of indirection. The EID indeed  
describes a service, but the EID comes out of an EID-prefix, and the  
RLOCs are associated with the EID-prefix.

Having said that, the EID-prefix could be a /32 or /128, but need to  
be extremely careful here that the mapping database doesn't become  
flat with host based entries.

> this is Robin; and I may be misreading even his message.

Understand. We will wait for a Robin reply for clarification.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 24 Apr 2009 10:07:31 -0700

I'll second that.

Dino

On Apr 24, 2009, at 8:39 AM, Joel M. Halpern wrote:

> Just to confirm, I do not think this is a topic that the LISP WG  
> needs to spend any energy on.
> Joel
>
> Sam Hartman wrote:

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2009 15:19:17 -0700

In addition to errors that may occur along the path from
the ITE to the ETE *before* decapsulation and errors that
may occur along the path from the ETE to the final
destination *after* decapsulation, errors may also occur
*during* decapsulation. An example is what if the ETR
decapsulates the outer IP header, but the inner IP stack
is not configured? (Answer is that the ITR should receive
an error by which it can ascertain that this ETR is having
trouble, and should start using a different ETR instead.)

This gives rise to a type of error message that is neither
fish nor fowl, i.e., it is neither an RLOC-based nor an
EID-based ICMP error - it is a *LISP* error instead. One
approach is to have the ETR send such a LISP error back
to the ITR with enough identifying credentials such that
the ITR can be reasonably sure the message is authentic.
The message should further not be dropped by packet
filters on the path from the ETR to the ITR that filter
"naked" ICMP messages.

Such LISP errors would need to be identified by a bit in
the LISP header (call it the 'E' bit) preferably adjacent
to the existing 'S' bit. The ETR sets the 'E' bit to tell
the ITR that this LISP packet contains an error, and not
an IPv4 nor IPv6 packet. The error could be formatted as
per an ICMPv4 or ICMPv6 error, but it need not be so.
For example, the packet could be constructed as:

 +-----------------+
 | Outer IP header |
 +-----------------+
 |    UDP header   |
 +-----------------+
 |  LISP hdr (E=1) |
 +-----------------+
 | ICMP header (?) |
 +-----------------+
 |                 |
 ~ Packet-in-error ~
 ~(up to 512 bytes)~
 |                 |
 +-----------------+

Again, this would be the ETR's way of telling the ITR
that it is having trouble and perhaps a different ETR
should be chosen. An exception would be a "packet too
big" error, in which the ITR would simply adjust its
packet sizes as long as the MTU reported by the ETR
is of a reasonable size.

As with any network-based error message feedback, this
is susceptible to on-path attacks. It is resilient
against off-path attacks, however.

Comments?

Fred
[email-masked]

---

From: Robin Whittle <[email-masked]>
Date: Sat, 25 Apr 2009 22:06:45 +1000

Short version:  I am working on a better description of the uses of
                the "distance server" idea I proposes.  I will call
                this system "nearcast".  In the future I will write
                this up on the Ivip site.  For now, I am writing
                about it to the LISP and RRG lists, because it seems
                to be novel and useful - and because I think it can
                be applied just as well to LISP, APT and TRRP as to
                Ivip.  (Actually, I have to change Ivip significantly
                to do it - the changes to LISP etc. would be less.)

                I think this "nearcast" approach has many of the
                benefits of conventional BGP-based anycast, but it
                will be entirely scalable as part of LISP etc.

                Conventional BGP-anycast is highly unscalable: one
                DFZ prefix must be devoted to each set of anycast
                servers.

                LISP, APT, Ivip etc, could be made to do much the
                same job as conventional BGP-based anycast, by
                anycasting the ETRs.  However this cannot be done
                in a scalable fashion.  It would be just as
                unscalable as conventional BGP-based anycast, and
                be more complex - with no obvious benefits.  Each
                set of anycast servers would still need a route
                in the DFZ.

                The "nearcast" system I am working on would have
                significant advantages, in addition to being
                scalable, compared to conventional BGP-based
                anycast.  I think the most important novel function
                would be, its much greater suitability for
                session-based protocols.

Hi Sam and Dino,

This "nearcast" "Distance Server" system addition to LISP or other
core-edge separation schemes should function as a scalable
replacement for conventional BGP-based anycast.  However, it would
not have precisely the same functionality.  In many scenarios its
functionality would probably be just as good as conventional
BGP-based anycast.  However, I think it would be also much more
suitable for the session-based protocols which are generally not used
for conventional BGP-based anycast.

I will write more about this in a forthcoming message on the RRG
list.  I am going to use the term "nearcast" for this system, since
it looks useful and since it is different from anycast and unicast.

> Understand. We will wait for a Robin reply for clarification.

I wouldn't "propose" the LISP team embark on any work or addition to
LISP, since I am not part of the team.   I try to contribute to the
development of LISP project by way of constructive critiques and by
suggesting new ideas which I think would improve LISP.

I just thought of this a day ago and as far as I know it is novel and
would be useful in some important and unique ways.

Amongst other things, it would be a way that with a single EID
address, an end-user network could field multiple servers around the
world - with full support for routing scalability.  Neither the
current BGP-based anycast technique, or its functional equivalent in
LISP etc.(anycast ETRs) is scalable.  Both are *less* scalable than
ordinary PI prefixes for end-user networks since in both cases, a
complete DFZ prefix needs to be dedicated to each set of anycast
servers.

There are many potential purposes for the "nearcasting" system I am
working on.  One of these is providing different DNS responses to
requesters from topologically separate parts of the Net.  I
understand this is currently done via a different technique, loosely
called "geo-DNS": matching the requesting IP address against some
kind of database to determine the country of origin.  I think there
are paid-for services for such maps, but a free one is
http://countries.nerd.dk/more.html .  This is a tricky business, to
say the least - because it requires a constantly updated database as
the usage of 0.25M DFZ prefixes change.   Another approach is
pgeodns, a perl script which is used by ntp.org at least to enable a
DNS server to respond according to the approximate "location"
(topological? geographical?) of the requester.

Both these and I guess any other approaches to geo-DNS are going to
work less and less once a core-edge separation scheme is operational,
assuming more and more requesters to the geo-DNS equipped server come
from EID addresses.  There will be little or no direct relationship
between an EID address and geographical or topological location,
since one of the purposes of a scalable routing solution is to enable
these EID addresses to be portable and used with any ISP,
irrespective of location.

So it seems that LISP or any other core-edge separation scheme is
incompatible with geo-DNS.  Unless we provide an alternative, this
would be a reason for people to argue against the adoption of LISP
etc.  If we can provide a better alternative, to geo-DNS and to
conventional BGP-based anycast, or for other uses of "anycast"-like
techniques, then this would be a reason people would want to adopt
LISP or whatever.

I think the "distance server" "nearcasting" system I am working on
would be useful for many things, including placing multiple
nameservers in different topological locations, all on the same IP
address - and then letting each one of those nameservers return
different IP addresses to their requesters.  No IP-to-location
database or any fancy computer programming is needed for this
approach.  So it doesn't work exactly on countries, but it would
automatically provide each separate DNS with a catchment area of
requesters according to how the BGP routers consider each "nearcast"
DNS server is "nearest" to the requester.  This will work fine with
requesters which are on EID addresses, as well as requesters on
conventional RLOC addresses.  A geo-DNS will not work as well, or in
principle, at all, with EID addresses.

>> anycast.

I am using a new term now: "nearcast", because what I am suggesting
is not precisely "EID anycast" or anything else.

I haven't found a good use for "RLOC anycast" == "anycast ETRs":
putting two or more ETRs on the same global unicast RLOC address, but
each with a separate border router in topologically separate parts of
the Net.

> within a site that assigns EIDs out of the EID-prefix it owns.

I understand this as there there being one or more ETRs at a site (as
usual, with each one having its own RLOC address) and within the
site's internal routing system there being multiple hosts anycast
with the same IP address.  So the area (scope) in which the anycast
system operates (the distinctive "closest alive destination" way of
choosing where to send packets) is entirely within the one "site" -
however defined.  Therefore, assuming the ITR's choice of ETR has
nothing to do with how close the ETR is to the ITR (how could the ITR
known this, without something like a "distance server"), then this
"anycast EID within a site" has no influence on the path taken by
packets in the DFZ.  Therefore, the scope of the anycast arrangement
is within the end-user network, not in the DFZ.

This would be scalable and may have its uses.  The only one I can
think of is load-sharing and/or redundancy with multiple locally
anycast hosts handling packets sent to them from one or more ETRs.
However, this "anycast within the end-user network" does not does not
at all achieve the benefits of conventional BGP-based anycast in the DFZ.

The benefits of what I call "conventional BGP-based anycast" - as
used for some root nameservers - rely on the way packets take the
"shortest" path within the DFZ to the nearest operational "anycast
host".  "Shortest" is in terms of each BGP router's choice of
shortest path - based on the number of ASNs in the path offered by
its neighbours.

The BGP system naturally transports the packet to the the nearest of
typically multiple border routers advertising the same prefix.  In
the case of a single conventional ISP or end-user network advertising
the same prefix from multiple border routers, there is only one
destination host for each IP address (unless there is local anycast
within that network, which is not at all conventional).

What makes two or more border routers advertising the same prefix a
true "anycast" arrangement is that each border router sends the
packet to its own host.  Typically, the border routers are widely
geographically and topologically separated.

It is impossible to discern from the flow of packets in the DFZ
whether this arrangement of two or more border routers advertising
the same prefix is anycast or not.  The "anycast" nature of it
depends on each router sending packets to its own host, not all such
routers sending packets to the one host.   With conventional
BGP-based anycast, the scope of the anycast system is the entire
Internet, including the DFZ.

>> I'll admit that I'm not really seeing the point in the
>> if that happens, it should behave as you describe.

What Dino refers to below is something different: "RLOC anycast" AKA
"anycast ETRs": multiple ETRs having the same, globally (with BGP),
anycast address - and each ETR having a separate border router, if
the ETR itself is not a border router.  This is simply a case of
using otherwise conventional BGP-based anycast for ETRs.

So these multiple border routers (one for each ETR) advertise the
same prefix, or at least prefixes which all encompass the one anycast
address used by the multiple ETRs.  These ETRs are physically
separate and located in different parts of the world.   This could
work, but I didn't find any good use for it.  Below, I explain why
anycast ETRs is at odds with LISP's (or APT's, or Ivip's etc.)
intention to help with the routing scaling problem

While I didn't find any use for it, I didn't consider this:

> Well, there is an important point. You don't have to worry about
> RLOC liveness when you have anycast.  You just pray and send
> packets and they will get to the destination site through one of
> the anycast ETRs.  And if any of them go down or become
> unreachable, the core will reroute without any involvement from
> the ITRs.

This is not an issue in Ivip. (Not counting how I intend to alter
Ivip with this "distance server", "nearcast" stuff) Ivip ITRs do not
make decisions about which ETR to use.  The mapping for each micronet
(EID prefix) has a single ETR address - since with real-time (a few
seconds) mapping distribution to all the ITRs which need it, the
single ETR address which each micronet is mapped to can be assumed to
be alive.

With LISP, APT and TRRP, you don't have real-time mapping so you need
 the mapping to include multiple possible ETR addresses - and let the
ITRs (or for APT, the Default Mappers) decide which ETR address to
used, based on tests or assumptions about which ETRs are currently
reachable.

In this context, anycast ETRs ("RLOC anycast") looks like it has an
an advantage for LISP, because as you wrote, it solves a problem of
ITRs needing to decide whether one ETR or another is reachable.
Instead, you can assume that at at least one ETR is alive and allow
the BGP system to take care of sending the tunneled packet to the
nearest ETR which is alive.

However . . . there is a serious gotcha with this "anycast ETR" use
of LISP (or any other core-edge separation scheme): it is of no use
to scalable routing.

Let's say you have an ETR in LA and another in NY, both on the same
address 44.55.66.77.  Each one decapsulates packets and sends them to
its own server, both of which are on the same EID address, for
example 7.6.5.4.

In this example, we use the BGP system's natural behavior to direct
tunneled packets to the nearest ETR which is alive.  This depends
entirely on the router at each site (which may be the ETR, or may be
some other router which the ETR is using to connect to the DFZ)
withdrawing the route which encompasses the ETR's address the moment
that ETR dies, or becomes unreachable - or the moment the server
behind the ETR (on 7.6.5.4) dies or becomes unreachable.

So for instance, both routers RLA and RNY would advertise
44.55.66.0/24 - which encompasses the address of the two ETRs.   The
mapping for the EID prefix encompassing 7.6.5.4 has a single ETR
address: 44.55.66.77.

Because each router RLA or RNY must withdraw the route when its ETR
dies, or when the server behind the ETR dies, you can't use this
44.55.66.0/24 prefix for any other purpose.

The scalable benefits of core-edge separation schemes such as LISP,
APT, Ivip and TRRP depend entirely on an ISP being able to have the
ETRs used by many end-user networks - thousands or hundreds of
thousands or whatever - all covered by a single BGP-advertised prefix
of an ISP.  Anycast ETRs violate this principle completely.  I
haven't yet found a practical use for them.

>> However I'm fairly

You can do local EID anycast as I initially described, where the
anycast scope is entirely within a single end-user network.  That
does not have the benefits of conventional, global scope, BGP-based
anycast: within the DFZ the packets automatically being forwarded to
the border router of the nearest alive host.

The closest way you can achieve the same benefit with LISP etc. is to
anycast the ETRs instead as I just described.  That has similar
benefits to conventional BGP-based anycast, and it uses EID space for
the destination host, as with all LISP etc. systems.  While it neatly
solves the problem of LISP ITRs having to figure out which of
multiple ETR addresses to use, the system does not seem to be useful
because:

  1 - It is no more scalable than conventional BGP-anycast: a single
      DFZ prefix is required for every set of hosts.  (This is worse
      scalability than conventional PI prefixes for end-user networks
      which are typically used to support large numbers of hosts.)

  2 - It is more complex than conventional BGP-anycast.

> Having said that, the EID-prefix could be a /32 or /128, but need to be
> extremely careful here that the mapping database doesn't become flat
> with host based entries.

This is a question which I think is unrelated to anycast.  I will
take it up in a separate thread.

  - Robin

---

From: Dino Farinacci <[email-masked]>
Date: Sat, 25 Apr 2009 16:57:29 -0700

> the ITE to the ETE *before* decapsulation and errors that

You mean from ITR to ETR (since this is a LISP mailing list post).

> trouble, and should start using a different ETR instead.)

This is unlikely in most implementations. If the IP stack is not  
configured the decapsulation wouldn't have happened. And that's  
because the IP packet would arrive on an interface and IP wouldn't  
process it.

So Fred, you really don't want to send per-packet errors. You don't  
want to send errors at all actually.

What if a regular IP packet had a source address of say 127.0.0.1,  
224.1.1.1, or 0.0.0.0? Do we send errors for these situations? First,  
you don't want hardware forwarding engines to test for all these  
invalid addresses, and second, you don't want the control-plane to  
have to deal with per packet errors.

IP is best-effort and drops packets, that's one of the best features  
the Internet architecture has.

> EID-based ICMP error - it is a *LISP* error instead. One

Not true. What if a NAT in the middle swapped a source address from a  
good address to 224.1.1.1?

Putting the blame to which protocol component is in error and trying  
to take action is not fruitful at all. In fact, it is rather damaging.

> the ITR can be reasonably sure the message is authentic.

Ah man, you are making the solution even harder. So you want pair-wise  
Security Associations among all ITR->ETR conversations?

> For example, the packet could be constructed as:

And since the ITR is so buggy, what makes you think it will listen to  
what the ETR has to say? What happens if the ITR is malicious?

> against off-path attacks, however.

With all due respect Fred, when I first read this, I checked the date  
on my calendar to make sure it wasn't April 1st.

The only reason I make this statement above is because you asked for  
comments.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Sat, 25 Apr 2009 17:00:42 -0700

Okay Robin, if you want us to take you seriously, and I really want  
to, can you please stop saying what the idea is 1) named, 2) who is  
going to work on it, 3) what it's benefits are, and 4) what protoocols  
can use it?

Rather, tell us what the idea is before you marketed it to death?

Give us the idea in a short description and the smart people on this  
list can conclude what the benefits and costs are.

Thanks in advance,
Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2009 07:01:27 -0700

Dino,

You can belittle all you want; just remember that you
heard it here first.

Fred

> -----Original Message-----
> From: Dino Farinacci [mailto:[email-masked]]
> Sent: Saturday, April 25, 2009 4:57 PM
> To: Templin, Fred L
> Cc: [email-masked]
> Subject: Re: [lisp] LISP errors
>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 27 Apr 2009 09:25:21 -0700

I'm not belittling you Fred. I did say "with all due respect" and I  
wasn't being sarcastic. I just can't see how this idea can scale.

Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2009 09:38:47 -0700

Dino,

> I just can't see how this idea can scale.

Then, I will give you just one more thought for now:
*rate limit*!

Fred

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 27 Apr 2009 09:44:30 -0700

But the point is, what would the ITR do with such error information if  
it could be scalably transmitted.

Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2009 09:59:29 -0700

Dino,

> Sent: Monday, April 27, 2009 9:45 AM
> To: Templin, Fred L
> Cc: [email-masked]

In order to answer this, we would have to begin by
enumerating the sorts of errors that could happen
at the LISP level. Examples:

- Parameter Problem (e.g., the version field following
  the LISP header contains an unknown value). Do not
  discount the importance of this one in particular,
  since it can be used by the ITR to determine whether
  or not this ETR implements a new function.

- Packet Too Big - packets sent by the ITR are being
  fragmented by the network. The ITR should begin
  limiting the size of the packets it sends.

- Destination Unreachable - no route. The FIB in the
  ETR is inconsistent with what appears in the mapping
  tables. The ITR should really try a different ETR.

- Destination Unreachable - protocol unreachable. The
  ETR somehow is in a funky state in which it can
  receive encapsulated packets and decapsulate them,
  but it can't forward them further, e.g. because
  the inner IP stack is down.

- Destination Unreachable - administratively prohibited.
  This is a case study unto itself

- Destination Unreachable - failed ingress/egress
  filtering. Ditto.

- others?

Fred

> >>>> Not true. What if a NAT in the middle swapped a source address
from
> > a
> >>>> Ah man, you are making the solution even harder. So you want
pair-
> >>>> wise
> >>>> And since the ITR is so buggy, what makes you think it will
listen
> > to

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 27 Apr 2009 10:09:49 -0700

> - others?

There are 100s more.

Dino

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2009 10:17:04 -0700

> There are 100s more.

Then, let's just focus on the few that *really* matter;
ICMPv4 and ICMPv6 were able to keep the lists bounded.

Fred

> Sent: Monday, April 27, 2009 10:10 AM
> To: Templin, Fred L
> Cc: [email-masked]

---

From: Sam Hartman <[email-masked]>
Date: Thu, 30 Apr 2009 12:14:49 -0400

>>>>> "Templin," == Templin, Fred L <[email-masked]> writes:

    >> There are 100s more.
    Templin,> Then, let's just focus on the few that *really* matter;
    Templin,> ICMPv4 and ICMPv6 were able to keep the lists bounded.

    Templin,> Fred [email-masked]

Fred has made a proposal that LISP needs a mechanism to indicate
errors.  Dino has raised scalability objections as well as an
objection that the ITR would not be able to do anything useful with
the information.

I'd appreciate comments from others.

I'd appreciate comments from Fred and Dino on how the motivations
behind this mechanism differ from ICMP.  My default presumption is
that Dino's concerns apply to ICMP, but so does the value
proposition--so that it would be valuable in many of the same ways
that ICMP is valuable but that many of the same constraints to make
sure it does not become unworkable would apply.
In particular, I believe we've already had to deal with Dino's concern about sending to multicast, local or other sources for both ICMPv4 and ICMPv6.

---

From: Olivier Bonaventure <[email-masked]>
Date: Thu, 30 Apr 2009 18:29:03 +0200

Sam,
> I'd appreciate comments from others.

I think that once we'll have a stable definition of the lisp protocol,
we'll need to define the error cases that will trigger the transmission
of an ICMP message. Although ICMP has clear drawbacks, I'm not convinced
that we can simply discard all packets in error. There are some errors
that may lead to the generation of an ICMP message. The protocol spec
will also need to discuss how an xTR will behave upon reception of an
ICMP message.

Olivier

---

From: Scott Brim <[email-masked]>
Date: Thu, 30 Apr 2009 12:30:21 -0400

> I'd appreciate comments from others.

If I understand Fred's scenario correctly, I would not put the error
handling in LISP.  If I'm correct: Site A encapsulates a packet and
sends it to Site B.  The packet is correctly delivered to Site B's
ETR.  The ETR correctly decapsulates it.  AFTER THAT, a forwarding
function looks at the unencapsulated packet and discovers it doesn't
understand the packet (wrong protocol version?), or there is something
else wrong with it.  This is not a LISP error -- LISP did its job
correctly.  It is either a configuration error or a forwarding error
and it should be handled as such errors are handled today, outside the
scope of LISP.  If an ordinary router, today, sees an IP packet with a
bad version number, what does it do?  What do the endpoints do in
response?  Etc.

Scott

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 30 Apr 2009 10:00:57 -0700

Hi Scott,

I think if I read Fred's proposal correctly he is talking about the 
errors at the ETR.

ETR will most likely be a router so we can not say that once the packet 
is decapsulated it is no longer, a LISP business. LISP is part of end to 
end architecture and will be in this experiment used as a transport. 
Even if we say this is not part of LISP ... it is clearly in the ITRs 
interest to know as much as possible about health of ETRs he is sending 
data to and to gather as much as possible hints on what good or bad is 
happening with those packets. Remember it is almost given that ITRs & 
ETRs will be sitting in different administrative domains.

I am in favour of defining finite set of error messages even if those 
happen after decapsulation and triggering ICMP to ITR to take some 
action on it. At min it could be just a syslog action, for more 
intelligent ITRs that could be skipping given ETR from the dst points 
:). Otherwise how would you propose to detect malfunctioning ETR if ITR 
uses N of them simultaneously ? You ping all they all respond it is just 
that one has a crashed RIB/FIB.

Cheers,
R.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 30 Apr 2009 15:02:50 -0400

>>>>> "Scott" == Scott Brim <[email-masked]> writes:

    Scott> Excerpts from Sam Hartman on Thu, Apr 30, 2009 12:14:49PM
    Scott> -0400:
    >> >> There are 100s more.  Templin,> Then, let's just focus on
    >> the few that *really* matter; Templin,> ICMPv4 and ICMPv6 were
    >> able to keep the lists bounded.
    >> 
    >> Fred has made a proposal that LISP needs a mechanism to
    >> indicate errors.  Dino has raised scalability objections as
    >> well as an objection that the ITR would not be able to do
    >> anything useful with the information.
    >> 

    Scott> If I understand Fred's scenario correctly, I would not put
    Scott> the error handling in LISP.  If I'm correct: Site A
    Scott> encapsulates a packet and sends it to Site B.  The packet
    Scott> is correctly delivered to Site B's ETR.  The ETR correctly
    Scott> decapsulates it.  

Scott, I believe as others have said that we're talking about problems
decapsulating the packet.  However I suspect you're quite familiar
with a similar situation from MPLS.  If I get an MPLS error trying to
pop a label--for example I didn't distribute that label over the incoming interface, what do I do?
What about TTL handling in the MPLS network?
When I generate errors do I send them towards the original source of the packet or do I send them to the entity who sent an MPLS packet to me?

Is this different with TE tunnels?