Subject: host-centric draft

From: Iljitsch van Beijnum <[email-masked]>
Date: Fri, 20 Feb 2004 01:05:06 +0100

Hi,

I've been catching up on drafts, reaching draft-huitema-multi6-hosts-03 
just now. A few questions:

   "A strong assumption of the IPv6 architecture is that all prefixes of
    a site will have the same length; it is thus possible to derive a
    prefix from the source address of a "misdirected" packet, by
    combining this prefix with a conventional suffix."

This is very unclear as the reason to have such an anycast address in 
the first place is only given 30 or so pages later.

About ICMP/ingress filtering: if it is actual ingress filtering we're 
talking about (ie, an ISP filters packets it receives from a customer) 
ICMP error messages aren't likely to be forthcoming when an improper 
source address was used by the customer, as the ISP in question doesn't 
know that the addresses which are filtered out should go back to the 
customer in question. So in order for this to work reliably customers 
must install filters that perform the same function, but then for 
egress, as the customer border router has a good return path for the 
ICMP message.

A general problem that I have with this draft is that it explores some 
very specific mechanisms, but at the same time doesn't explore some 
other mechanisms that could lead to similar results. (For instance, a 
situation where two distinct layer 3 networks exist in parallel isn't 
explored. For source address rewriting using a special "rewrite this" 
prefix would be a possibility.) It would be better to either be brief 
or complete.

The problem that rerouting could invalidate an earlier source address 
choice isn't mentioned.

I'm unsure why we would want to inject BGP information into hosts. 
Reachability yes/no isn't very useful as the actual reachability status 
of the other side is in almost all cases hidden by aggregation. So 
basically this only indicates whether the ISP in question is available. 
Determining preference based on BGP information is also a relatively 
futile endeavor as path lengths are very often the same and relevant 
differences in topology typically don't show up in BGP. (Many large 
networks use a single AS for an entire continent or even the world, 
while obviously small networks don't.)

The 6 level deep heading numbering is somewhat excessive.

9.1.4.19 What new information should applications be aware of?

    None

I don't really agree. Applications need to try all addresses for a 
correspondent. Some do this today, most don't. And this needs to be 
done in a smart way, a four minute timout between trying successive 
addresses isn't acceptable.

---

From: "marcelo bagnulo" <[email-masked]>
Date: Fri, 20 Feb 2004 18:37:50 +0100

Hi Iljitsch,

thanks for commenting the draft

I include my own answers to your comments, Christian and Richard perhaps
think different

> the first place is only given 30 or so pages later.

Ok.
We could add a pointer to the explanation 30 pages later, or even we could
remove some unclear details at this point and leave all the explanation for
later.
Just mentioning that it would be good if the prefix can be derived
automatically and leave for later the explanation on how to do this.

> egress, as the customer border router has a good return path for the

Yes, this is what i think is described in the draft...

I mean, we assume that the isp is performing ingress filtering, so the idea
is not to send packets with a incompatible source address.
We then explore different forms to do this, one of them involves sending an
ICMP error back to the host, indicating the correct source address. Now,
this ICMP error message is not generated by the ISP border routers but it is
generated by the site exit routers, since it is the site itself that is
interested in providing feedback to the hosts. All this is an intrasite
mechanism.
So, i think that i agree with you and the draft states what you are
saying...

> prefix would be a possibility.) It would be better to either be brief

Well, i think that the idea of the draft is to analyze all the options that
seemed reasonable to us. The problem may be that we didn't came up with all
the possible solutions, so we didn't include them in the draft.
The draft starts with an analysis of all the options that we had identified
(not so much detail in this part) and then selects and describes the
preferred solution (much more detail in this part)

IMHO, the analysis is as important as the solution proposed since it is
aimed to discuss different approaches.

So if there are some points missing in the analysis, the selected solution
may well change

I would be glad to try to include another options in the analysis.

for instance, the options that you were mentioning that is missing, could
you expand on this?

The two layer 3 network that you mention, would this be like a multihomed
host?

> The problem that rerouting could invalidate an earlier source address

Could expand this too? perhaps an example?

> I'm unsure why we would want to inject BGP information into hosts.

I don't think that injecting BGP routing table has been considered in the
draft (but i may be forgetting something)

> while obviously small networks don't.)

Well,
as i see it, there are two systems that can be aware of reachability: the
routing system or the end host itself

The end host can discover reachability by itself, simply trying to send
packets to the other end. This is more reliable way to discover
reachability, since the host is actually reaching the target.

The other option is to use the routing system. As you mention, the
information provided by the routing system is aggregated, so it hides some
information.
However, the routing system has the information in advance, while the host
has to discover it when it needs it.

So the host obtained by the host is more accurate, but the routing system
provides it faster, i guess.

both options are explored in the draft and some of the drawbacks are
analyzed, perhaps this particular pov in really included, though

The other problem with bgp is that it seems unsuitable for small sites,
where the is not likely to be bgp expertise available, so the host option is
considered more suitable for this scenario

Large sites that have networks manager are likely to be capable of running
bgp. Additionally, it seems that the bgp option provides more centralized
policy management, so perhaps it would be preferred by large sites.

But one of the goal of the draft is to foster discussion on this topics...

> The 6 level deep heading numbering is somewhat excessive.

Yes, but i guess that merging sections would render the draft even more
unreadable

> addresses isn't acceptable.

Well, rfc 3484 already states that
"Well-behaved applications SHOULD iterate through the list of
addresses returned from getaddrinfo() until they find a working
address."

But i agree with you, something about this can be included here.

The other point about this is when considering retrying with different
source addresses.
If you do this sequentially, i think that it would be up to the app to
retry.

However if you do it in parallel, i think that the ip layer could simply
generate multiple packets and send them in parallel, so no need to change
the apps.

Reagrds, marcelo

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 25 Feb 2004 10:05:35 +0100

> The two layer 3 network that you mention, would this be like a 

The idea is that you build two separate layer 3 networks so that the 
host gets to make a choice in which network they inject the packet and 
then there is no complexity moving packets from the part that talks to 
ISP A to the part that talks to ISP B. The simple way to do this would 
be to have two network interfaces on each host, where one NIC has an 
address from ISP A and talks to routers that eventually connect to ISP 
A, while the other NIC has an address from ISP B and only talks to 
routers connected to ISP B. A slightly more complex way to do this 
would be to have the host use different virtual interfaces over a 
single NIC (using VLANs or some such).

> Could expand this too? perhaps an example?

Host X wants to talk to remote host Y. X sends packets with source 
addresses compatible with ISP A to a router that delivers the packet to 
ISP A and life is good, despite the fact that the router doesn't do 
source address based routing. Now the link to ISP A fails, so the 
router reroutes the packets to Y over ISP B. But now the source 
addresses used by this session are incompatbile with the way the packet 
is routed. Life is not so good, especially if this was a legacy session 
that doesn't support changing addresses during the session.

Now in this example the problem isn't too bad because the session would 
have failed also if the site was single homed to ISP A. But if the site 
uses BGP to determine the best path, it may also reroute traffic over a 
different ISP if the first ISP is still available. So then we're 
breaking sessions that would have worked without multihoming. That 
can't be what we want. So any kind of dynamic best path selection can't 
be used unless ALL sessions can change addresses.

> draft (but i may be forgetting something)

What's all the talk about BGP in the draft about then?

>> futile endeavor

> reachability, since the host is actually reaching the target.

Right.

> information. However, the routing system has the information in 
> advance, while the host has to discover it when it needs it.

Our current interdomain routing system doesn't deliver actual 
reachability information. It is _extremely_ rare for an entire ISP / 
address block from an ISP to be down. And it's well-established that 
BGP is very slow to propagate the fact that a prefix is no longer 
reachable. So I would strongly advice against taking "reachable yes/no" 
information from BGP. And as explained above, doing path selection 
based on BGP properties is also problematic.

> provides it faster, i guess.

So the question is: can we detect the actual reachability using probes 
fast enough that we don't feel we also need to look in the routing 
tables? If end-to-end is 30 seconds while routing table is 30 
microseconds, then sure, I agree that the latter is useful. If 
end-to-end is 3 seconds, maybe having to wait this long some of the 
time is better than importing all this BGP complexity. If it's 300 
milliseconds I'm sure it is.

> source addresses. If you do this sequentially, i think that it would 
> be up to the app to retry.

> generate multiple packets and send them in parallel, so no need to 

Agree. But then, we probably want to try multiple destination addresses 
at the same time too, which can't be done without changes. And we 
probably want to have a way to signal to the other side "these are all 
really the same session, don't be worried about the session setup being 
aborted for the alternative ones".

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 25 Feb 2004 09:06:36 -0800 (PST)

> milliseconds I'm sure it is.

That's on the benefit side, but also have too look at the cost side
in terms of scaling.
If all hosts perform e2e probing of all the local/peer locator pairs
this might add up to a lot of packets; might be more than the amount of
data packets that are exchanged in some cases (imagine hosts being sensors
which only send a data packet every 30 seconds and now you add probing 4 or
so local/peer locator pairs!).

If we care about *site* (and not *host*) multihoming one could try to
have hosts share the information they learn from probing with their
neighboring hosts. But that leads more or less to reinventing
a routing protocol.

  Erik

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 25 Feb 2004 18:43:51 +0100

On 25-feb-04, at 18:06, Erik Nordmark wrote:

>> tables?

> this might add up to a lot of packets;

I'm afraind looking at the routing table first has very little impact 
on the number of reachability probes that are sent. After all, any 
given prefix is present in BGP 99% or more of the time, but we still 
don't know whether a certain address within that prefix is reachable. 
So only in the 1% of all cases where the routing table tells us the 
prefix is unavailable we save something.

> might be more than the amount of

That's why I think we should only perform reachability checks when we 
already know or at least have a strong suspicion that something is 
wrong.

One possible exception is session establishment: it might be useful to 
try several setup attempts in parallel, as the one that completes the 
fastest is probably also the one that offers best peformance during the 
session.

> neighboring hosts. But that leads more or less to reinventing

I think this would be useful (how many routing protocols are there? 
still room for one more) but we have other things on our plate right 
now.

---

From: "marcelo bagnulo" <[email-masked]>
Date: Wed, 25 Feb 2004 20:21:30 +0100

> single NIC (using VLANs or some such).

Ok, i see what you mean now, but i am not sure that this approach is very
attractive though.

There are two possibilities, as you explain:

- different physical paths: this is very expensive, i guess and normally,
you don't have to build a parallel network when you add a new ISP. So, while
i can see that some networks, with extreme fault tolerance requirements may
find this attractive, i see it a bit too much for a general solution.

- VLANs approach. this would be an option. However, i am not so sure about
how simple to manage would this be. I mean you will have to create VLANs all
over the multihomed site, which AFAIK would be a manual process, right?. I
don't know, we can explore a bit more this, but i think that it would be
similar than using some sort of source address dependent routing, but even
more complex.

> be used unless ALL sessions can change addresses.

This would is addressed in the draft, since the rules for selecting the path
is first determined by the source address (no matter if the path is
preferred or not) and only if there is no path through the isp associated
with this source address the packet is discarded and the initiating host is
informed.
See the rules detailed in section 7.2.1

But anyway, i think that your concerns about bgp are more general than this
particular case, so let's consider this with a broader approach.

Remember that the draft only deals with the problem of discovering a
reachable locator for initial contact. It does not address the problem of
preserving established communications

There are two approaches proposed in the draft:

- A host based approach, where the host tries with all the possible
combinations of source and destination addresses (probably in parallel) and
it selects the combination that provides faster response.

- an hybrid approach, where the host tries with one combination of source
and destination address and instead of waiting until timeout, it is the
routing system that informs about reachability to the initiating host. That
means, routers close to the exit site run BGP and knows about which
addresses are reachable through which isps. Then the host generates a packet
with a given source and destination address, and when it reaches one of
these routers that run bgp, the routers knows if this packet will probably
make it or not, since it has knowledge about which destinations are
reachable through which providers.
So, if the router knows that the packet won't make it its final destination,
it just discards it and informs the host that it should try with a different
source address, or just abort, because the requested destination is
unreachable. This approach seems to provide a faster answer because the
router already has reachability information, as opposed to the host that has
to discover it through polling.

Now, your concern seems to be (as you have expanded in a following email)
that the information in the router is not really valuable, since most of the
time all the prefixes are in the global routing table, and that aggregation
hides most of the relevant information.

Do you have any data to back this up?

I don't really have a preference here about any of the two approaches, i
only think that running bgp would be too much for small sites, so host based
approach would suit better for this case.
But when considering medium/big sites, i get the feeling that people think
that BGP information is pretty valuable and that an hybrid approach would
provide some value. I mean, is what we use today, and packets make it most
of the time. But as you have mentioned already, as more aggregated the
routing tables are, the less valuable bgp information will be.

[...]

> milliseconds I'm sure it is.

Actually i think that if we accept that bgp will only provide hints (how
strong they are, we haven't agreed yet), we may still have to rely on end
host probes when using BGP hints.

> aborted for the alternative ones".

YEs but i am focussing only in discoverung a reachable locator for
initiating a communication :-)

Regards, marcelo