Subject: RE: [Forces-protocol] topic 2:: HA

From: [email-masked]
Date: Mon, 26 Apr 2004 09:26:51 -0400

Hello All,

We have three cases.

CE-to-CE(s) (common HA mechanism but outside the scope of charter)
CE(s)-to-FE (currently within the scope)
FE-to-FE (outside the scope).

More comments are inline.

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of ext Khosravi,
> Hormuzd M
> Sent: Monday, April 26, 2004 1:45 AM
> To: [email-masked]
> Cc: [email-masked]; Wang,Weiming; [email-masked]
> Subject: [Forces-protocol] topic 2:: HA
> 
> 
> Hi All
> 
> Here is my attempt to start some discussion on topic 2, we just have 1
> week to complete All discussions so we need to move fast!
> 
> Based on our discussions on TML and PL as well as the 3 
> schemes proposed
> by Jamal, here is my suggestion for HA scheme...
> 
> At any time there is one Primary CE controlling the FE and 
> there can be
> multiple secondary Ces. The FE PL is aware of the primary and 
> secondary
> Ces. Only the primary CE is sending Control messages to the 
> Fes. The FE
> may choose to send its events, redirection packets to all the Ces or
> only Primary CE (we need to decide one or keep this configurable ?). A
> CE-to-CE synchronization may be needed, however that is out of our
> current scope for ForCES PL. The association between the FE 
> and primary
> CE may change based on the Connection State (triggered by TML) or an
> explicit message (such as Move, described by Jamal) from the 
> primary CE.

As CE's are maste,we have two levels of issues. One is remote endpoint is dead and another 
is link failure(very rare to happen in single box scenarion, but possible in multi hop situation).
Consider a case CE1 & CE2 are master and slave respectively for FE-1. FE-2 is an standby for FE-1.
When CE-1 detects the link failure, it may will wake up alternative FE (say FE-2). But FE-1 informs CE-2 
that CE-1 has failed. But CE-1 and CE-2 are already in the loop. In this scenario, CE(s) is the master and 
CE-2 should inform the FE-1 to be pulled out.

There are other such scenario's we can derive. Since CE is the controlling authority and we are 
assuming that there is CE to CE synchornization taking place, link layer failure cases are different than the
remote peer detection.

I would request we should block set of ID's in CE and FE's space to have smooth accomudatio of CE-to-CE protocol
thro' our message strucuture also.

> 
> This scheme is similar to Scheme 2 proposed by Jamal with a few more
> details. Hope we can agree on this and move to the other 
> topics. We can
> discuss more details around the Protocol Messages related to 
> this scheme
> during topic:4. 
> 
> Let me know what you guys think ?

Regards
Ramg

> 
> _______________________________________________
> Forces-protocol mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/forces-protocol
> 

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Mon, 26 Apr 2004 09:42:38 -0700

Ram,

Thanks for your comments. My comments/questions inline below...

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
> explicit message (such as Move, described by Jamal) from the 

As CE's are maste,we have two levels of issues. One is remote endpoint
is dead and another 
is link failure(very rare to happen in single box scenarion, but
possible in multi hop situation).
Consider a case CE1 & CE2 are master and slave respectively for FE-1.
FE-2 is an standby for FE-1.
When CE-1 detects the link failure, it may will wake up alternative FE
(say FE-2). But FE-1 informs CE-2 
that CE-1 has failed. But CE-1 and CE-2 are already in the loop. In this
scenario, CE(s) is the master and 
CE-2 should inform the FE-1 to be pulled out.

There are other such scenario's we can derive. Since CE is the
controlling authority and we are 
assuming that there is CE to CE synchornization taking place, link layer
failure cases are different than the
remote peer detection.

[HK] So what was your suggestion based on this scenario ? Would you like
to see some change to the scheme that I described ? Sorry its not clear
to me.

I would request we should block set of ID's in CE and FE's space to have
smooth accomudatio of CE-to-CE protocol
thro' our message strucuture also.

[HK] This is a good idea, what do others think ?

Regards
Hormuzd

---

From: [email-masked]
Date: Mon, 26 Apr 2004 12:52:27 -0400

Hello Hormuzd,

comments are inline.

> From: ext Khosravi, Hormuzd M [mailto:[email-masked]]
> Sent: Monday, April 26, 2004 12:43 PM
> To: Gopal Ram (Nokia-NRC/Boston); [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: RE: [Forces-protocol] topic 2:: HA
> 
> 
> Ram,
> 

<Ramg>

What i wrote is an extension (or additions )to your proposed text. For example, where both primary 
CE and FE informing the standby CE and standy CE should inform the FE to take appropriate actions. As CE are 
considered masters and have control over the services.

<Ramg>

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Mon, 26 Apr 2004 10:09:36 -0700

Hi Weiming,

----- Original Message -----

Why does TML configuration affect the association between the Primary CE
and FE ? I must be missing something here. Pls do clarify this.

[Weiming] My understanding of the re-association process of CE-FE is
actually
to re-configure the TML in some way. Of course, you may call it just a
state
change  or a 'move' action,  or whatever, which actually does not matter
too
much.  What I actually mean is this process may then be able to be
controlled by
PL if we do not want CEM/FEM to do this (I'm actually still not clear if
CEM/FEM
may always be uprunning along with the protocol running, or just run for
one
time only when CE/FE starts). For e.g., we may setup some policies in
advance to
FE TML to let it know what to do when some CE fails. This is what I mean
TML
configuration here. My idea (I suppose also Ram's ) is such
configuration may be
necessary, not just for the HA policy.
[/Weiming]

[HK] Yes I understand this now, I posted another email to this effect.
Sorry I was confused before. I completely agree that this is something
that the PL should control in the TML, such as re-starting, initializing
the TML, etc. This does not belong to the CEM/FEM.

Regards
Hormuzd

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Mon, 26 Apr 2004 14:44:47 -0700

Hi Jamal,

Thanks for the clarification.
This looks good to me. I'll send out the updated text soon.

Regards
Hormuzd

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 

On Mon, 2004-04-26 at 17:03, Khosravi, Hormuzd M wrote:

> BTW, just wanted to clarify what you meant by 'backup path' above? I
am
> guessing that some TML might implement multiple paths, e.g. multiple
> TCP/UDP connections between CE & FE. Is that what you mean ? 

Multiple TCP/UDP connections could be one of the modes that the TML is
responsible for. It could also be multihomed, multinexthop reachable,
trunked in active/backup mode, ethernet bonded etc. Point is it is a
blackbox and the PL doesnt care about those details.
 
To provide an example:
Consider CE with id #102 which is currently considered master.
Consider that at the low layer is infact reachable as IP address a.b.c.d
and IP address z.x.y.v etc. 
Consider CE with id #103 and ip address w.e.r.t reachable via interface
3 with nexthop of s.d.f.g and a.w.d.r.
The Forces ids #102 and #103 is what PL cares about. The IP addresses is
what the TML cares about.
At some point when the TML has no way of reaching #102 via any of its
know addresses, then the PL needs to make a call to move to CE #103

cheers,
jamal

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 27 Apr 2004 10:03:41 -0700

Hi Weiming, Jamal,

Thanks for your comments. I'll add some more clarifications to address
your comments.

Regards
Hormuzd

-----Original Message-----

Hormuzd,

Thank you for your hard work. I'm quite fine with it, only with some
minute
comments as below.

Cheers,
Weiming

----- Original Message -----

Hi All

Here is the updated text for HA scheme, based on comments from Weiming,
Jamal, Ram...
(Added some more details/clarifications)

At any time there is one Primary CE controlling the FE and there can be
multiple secondary CEs. The FE PL is aware of the primary and secondary
CEs. This information (primary, secondary CEs) is configured in the FE
during pre-association by FEM.

[Weimng] Then the TML in FE knows the redundant CE id list, but how the
PL knows
this? And we should also let CEs (primary and redundant ) know all the
redundant
CEs' ids, so that CE can send 'move' command to FE.

Only the primary CE is sending Control
messages to the FEs. The FE sends its events, redirection packets to
only the Primary CE. A CE-to-CE synchronization may be needed to support
fast failover, however that is out of our current scope. During a
connection failure between the FE and CE, the TML on the FE will trigger
the FE PL regarding the failure. The FE PL will send a message to the
Secondary CEs to indicate this failure and one of the Secondary CE takes
over as the primary CE for the FE. An explicit message (e.g. 'Move')
from the primary CE, can also be used to change the Primary CE for an
FE.
In order to support fast failover, the FE will establish association
(joining) as well as complete the capability exchange with the Primary
as well as Secondary CEs.

[Weiming] FE joining is needed for secondary CE, but the capability
exchange
not, because we can assume at this time, the secondary CE has already
had the
information via the CE-CE syn. mechanism.

(I'll clarify the details of association
establishment, capability exchange during the Protocol Message
discussion:topic 4)

Responsibilities for HA:

1) TML level - Transport level:
a) The TML controls logical connection availability and failover.
b) The TML also controls peer HA managements.

At this level, control of all lower layers example transport level (such
as IP addresses, MAC addresses etc) and associated links going down are
the role of the TML or whatever the TML depends on.

[Weiming]That's ok.

2) PL Level:
The CEIDs are used to identify primary, secondary CEs. Protocol Messages
will be defined related to "Move" operation, etc.

[Weiming] This also means the CE should know the redundant CEids in
advance.

To put the two together, if a path to a primary CE is down, the TML
would take care of failing over to a backup path, if one is available.
Should the CE be totaly unreachable then would the PL be informed and it
may decide to move to a new CE.

[Weiming] Agreed.

Regards
Hormuzd

---

From: [email-masked]
Date: Tue, 27 Apr 2004 16:28:11 -0400

Hello Jamal,All,

> [mailto:[email-masked]]On Behalf Of ext Jamal 
> Hadi Salim
> Sent: Tuesday, April 27, 2004 3:00 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [Forces-protocol] topic 2:: HA
> 
> 
> On Tue, 2004-04-27 at 14:08, [email-masked] wrote:
> 
> > > [Weiming] If there is no CE-CE comm., the CE has to do 
> from scratch 
> > > (only with
> > > the FE providing some limited  state infomation). In this 
> case, it's 
> > > of very
> > > little help for the CE to even know who the FE used to talk with.
> > 
> > [ad] I agree that knowing who had been Primary (assuming 
> there is only 
> > a single primary) is not as useful in this case.  Still the 
> Fe should 
> > be able to supply a rather complete picture of its state if so 
> > required.
> 
> Ok, i think i finaly got what you are saying and i think it is
> a valid scheme. Since we are already putting responsibility to the
> FE to be aware of all CEs in multiCE setup, i dont think this 
> additional
> item will be that much of a cost either performancewise or protolwise
> (there will be some cost).
> OTOH, i think if a CE-CE setup decides it wants to likewise pass such
> information around, that too should be fine.
> I am not sure how to reach a compromise: Should the setup make the 
> decision?

This is a alternative scheme. But many vendors address basic HA mechanism thro'CE-to-CE.
But our ForCES should allow extensions if needed to accomudate those.

Regards
Ramg

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 27 Apr 2004 14:32:21 -0700

> > [Weiming] If there is no CE-CE comm., the CE has to do from scratch 
> > (only with
> > the FE providing some limited  state infomation). In this case, it's

> [ad] I agree that knowing who had been Primary (assuming there is only

> a single primary) is not as useful in this case.  Still the Fe should 

FE to be aware of all CEs in multiCE setup, i dont think this additional

[HK] I agree, we can have a compromise to keep both these schemes, i.e.
FE may send events, packets to all CEs or just primary CE as
configurable. This can be a behavior which is configurable by the CE. Is
this acceptable to Avri, Weiming, others ?

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 28 Apr 2004 14:22:03 -0700

Hi Avri,

I agree with you on the HA scheme. This would be a reasonable way to go
forward.
If Weiming, Ram agree with this we can close this issue for good.

Regarding, event messages, I have posted an email on the Protocol
Messages which addresses your comment I think. If you could review that
and let us know what you think, that will be helpful.

Thanks
Hormuzd

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, April 28, 2004 10:49 AM
To: [email-masked]
Subject: Re: [Forces-protocol] topic 2:: HA

Hi,

Yes, as long as the capability exists in the protocol, I don't mind if 
someone can turn it off.  I would argue, though, that having a FE 
notify its primary CE (or the CE group) of any new CE association (or 
loss of association) would be useful even if there is a CE-CE link - 
just as a safety check.  And obviously if everything is going well with 
CE coordination, then the CE probably will not need to request a state 
dump from the FE, but of course should be able to.

I also think that the ability to stifle event messages, by type 
including CE association announcements, is probably a general 
capability which is needed in any case.

a.

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 28 Apr 2004 14:25:23 -0700

Hi Robert

Sorry for the late reply.
Yes I think your comments regarding CE Group get addressed by having
this scheme.

Thanks
Hormuzd
-----Original Message-----
From: Robert Haas [mailto:[email-masked]] 

This seems to address the CE group issue I raised in my previous note, 
so I am fine with it. The CE group may consist of a single CE or 
multiple CEs. I suggest to consider symmetry, i.e., FE groups, as well.
-Robert

Khosravi, Hormuzd M wrote:

>Weiming, Avri, Ram,
>
>Is the compromise proposed by Jamal, myself below acceptable to you
guys
>?
additional

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 29 Apr 2004 12:51:59 -0700

Weiming, Thanks for this clarification. This helps us move forward on
this issue.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Wang,Weiming
Sent: Wednesday, April 28, 2004 10:54 PM
To: [email-masked]
Subject: Re: [Forces-protocol] topic 2:: HA

Hi Avri, Rob, Hormuzd,

I actually quite agree to retain the ability at the PL for FE able to
Broad/Mulitcast to CEs regarding its events, ability change, etc, and
also quite
agree with Avri's idea below. What I propose is just we may be careful
when we
recommend a CE failover (HA) scheme to customers. We should not let
customers
have a feeling that such B/M ability can solve HA very well. Customers
should
more rely on their CE-CE implementation for this purpose, with the B/M
ability
as only some asistance, therefore 'may assist fast CE failover recovery'
may be
the appropriate description.

Cheers,
Weiming

----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Thursday, April 29, 2004 1:48 AM
Subject: Re: [Forces-protocol] topic 2:: HA

> > configurable. This can be a behavior which is configurable by the
CE.
> > Is