Subject: RE: [Forces-protocol] topic 2:: HA

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Mon, 26 Apr 2004 18:41:30 -0700

Hi All

Here is the updated text for HA scheme, based on comments from Weiming,
Jamal, Ram...
(Added some more details/clarifications)

At any time there is one Primary CE controlling the FE and there can be
multiple secondary CEs. The FE PL is aware of the primary and secondary
CEs. This information (primary, secondary CEs) is configured in the FE
during pre-association by FEM. Only the primary CE is sending Control
messages to the FEs. The FE sends its events, redirection packets to
only the Primary CE. A CE-to-CE synchronization may be needed to support
fast failover, however that is out of our current scope. During a
connection failure between the FE and CE, the TML on the FE will trigger
the FE PL regarding the failure. The FE PL will send a message to the
Secondary CEs to indicate this failure and one of the Secondary CE takes
over as the primary CE for the FE. An explicit message (e.g. 'Move')
from the primary CE, can also be used to change the Primary CE for an
FE.
In order to support fast failover, the FE will establish association
(joining) as well as complete the capability exchange with the Primary
as well as Secondary CEs. (I'll clarify the details of association
establishment, capability exchange during the Protocol Message
discussion:topic 4)

Responsibilities for HA:

1) TML level - Transport level:
a) The TML controls logical connection availability and failover. 
b) The TML also controls peer HA managements.

At this level, control of all lower layers example transport level (such
as IP addresses, MAC addresses etc) and associated links going down are
the role of the TML or whatever the TML depends on.

2) PL Level:
The CEIDs are used to identify primary, secondary CEs. Protocol Messages
will be defined related to "Move" operation, etc.

To put the two together, if a path to a primary CE is down, the TML
would take care of failing over to a backup path, if one is available.
Should the CE be totaly unreachable then would the PL be informed and it
may decide to move to a new CE.

Let me know what you guys think ?

Regards
Hormuzd

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 27 Apr 2004 19:48:57 +0800

Hormuzd,

Thank you for your hard work. I'm quite fine with it, only with some minute
comments as below.

Cheers,
Weiming

----- Original Message -----
From: "Khosravi, Hormuzd M" <[email-masked]>

Hi All

At any time there is one Primary CE controlling the FE and there can be
multiple secondary CEs. The FE PL is aware of the primary and secondary
CEs. This information (primary, secondary CEs) is configured in the FE
during pre-association by FEM.

[Weimng] Then the TML in FE knows the redundant CE id list, but how the PL knows
this? And we should also let CEs (primary and redundant ) know all the redundant
CEs' ids, so that CE can send 'move' command to FE.

Only the primary CE is sending Control
messages to the FEs. The FE sends its events, redirection packets to
only the Primary CE. A CE-to-CE synchronization may be needed to support
fast failover, however that is out of our current scope. During a
connection failure between the FE and CE, the TML on the FE will trigger
the FE PL regarding the failure. The FE PL will send a message to the
Secondary CEs to indicate this failure and one of the Secondary CE takes
over as the primary CE for the FE. An explicit message (e.g. 'Move')
from the primary CE, can also be used to change the Primary CE for an FE.
In order to support fast failover, the FE will establish association
(joining) as well as complete the capability exchange with the Primary
as well as Secondary CEs.

[Weiming] FE joining is needed for secondary CE, but the capability exchange
not, because we can assume at this time, the secondary CE has already had the
information via the CE-CE syn. mechanism.

(I'll clarify the details of association
establishment, capability exchange during the Protocol Message
discussion:topic 4)

Responsibilities for HA:

[Weiming]That's ok.

2) PL Level:

[Weiming] This also means the CE should know the redundant CEids in advance.

[Weiming] Agreed.

Regards
Hormuzd

---

From: Jamal Hadi Salim <[email-masked]>
Date: 27 Apr 2004 08:39:10 -0400

> during pre-association by FEM.

Should be clear of "aware" - that it is "aware" of their IDs.
Does this restrict on setting the number of secondary CEs? Could you add
more CEs in post-association?
Do we also wanna talk about multiple FEs which are active backup?

> the FE PL regarding the failure. 

You are assuming the failure is only due to transport breakdown. This is
not always the case.

> The FE PL will send a message to the

There will be PL-PL heartbeats between CE and FE.
This is orthogonal to TML-TML heartbeats and TML link detection if it
exists. Probably shouldnt talk about TML-TML heartbeats, rather that the
TML will deploy some mechanisms which will provide the functionality.
The TML can detect the lack of connectivity to the PL Id being
monitored; i.e the cause of connectivity breakdown being transport.
Otoh, if theres some s/ware breakdown of some PL, then TML cant help
you, dependence then is in PL discovery of such a problem.

> establishment, capability exchange during the Protocol Message

Its probably to specific, but fine (dont wanna lag progress)

> the role of the TML or whatever the TML depends on.

Maybe add transport protocol ports in that list before etc

> may decide to move to a new CE.

I think the section on HA responsibilities above should come first
followed by the details you had at the beggining.

cheers,
jamal

PS: should we posting text on other topics - at the moment only Hormuzd is.
Whats next? What can i do?

---

From: Jamal Hadi Salim <[email-masked]>
Date: 27 Apr 2004 08:56:11 -0400

> CEs' ids, so that CE can send 'move' command to FE.

Maybe we can have information advertised in some protocol message?
Perhaps a "config" message is needed if we are not going to have an
always-on FEM-CEM.

> information via the CE-CE syn. mechanism.

Makes sense.

cheers,
jamal

---

From: [email-masked]
Date: Tue, 27 Apr 2004 10:09:39 -0400

>> during pre-association by FEM.

I am not sure I agree with this since there could be a load share or a 
specialization opportunity that doing this would prevent.

> Perhaps a "config" message is needed if we are not going to have an

I don't think the FE should, or even can, count on CE-CE 
communications.  I also think that secondary CE could establish 
adjacency with an FE at any time during the operation of the NE.

One way to deal with this is to have a FE announce any CE rlatioship it 
establishes to the CEs.  this way, even if there is no CE-CE 
communucation, at least the CE will know who else is talking to an Fe.

>> information via the CE-CE syn. mechanism.

I do not believe we can count on this.

a.

---

From: Robert Haas <[email-masked]>
Date: Tue, 27 Apr 2004 17:02:22 +0200

I note that the current proposal excludes schemes in which the FE sends 
its events and receives its commands from CEs located in a group of CEs 
identified by a particular group ID.

As a side note, and as an apetizer for the protocol message definition work:

A control-type LFB could be introduced named "HA LFB", which would 
contain on the FE side the list of primary and secondary CEs. This way, 
once somebody wants to introduce active-backup FEs, we don't need to 
change any of the protocol messages. Instead, only need to extend the HA 
LFB with additional attributes. This applies also to the current FE-CE 
relationships at an FE that a CE can query (or be informed about, just 
like some error messages).

-Robert

-- 
Robert Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: "Weiming Wang" <[email-masked]>
Date: Tue, 27 Apr 2004 23:29:47 +0800

Hi Avri,

I may have an opinion different  from yours on multi-CEs . I think the CE-CE
communication is fundamental for multi-CE parallelly working scenario to work.
As far as I can think of currently, I can not reach  other better means than
CE-CE direct communications for inter-CE state syncronization. I'm not sure if
you have a better approach for this issue.
Pls also see some more reply inline.
Thank you.

Weiming

----- Original Message -----
From: <[email-masked]>

> communucation, at least the CE will know who else is talking to an Fe.
[Weiming] If there is no CE-CE comm., the CE has to do from scratch (only with
the FE providing some limited  state infomation). In this case, it's of very
little help for the CE to even know who the FE used to talk with.
> I do not believe we can count on this.
[Weiming] I think the "copy" mode between CE-CE is the simplest mode for
syncronization. If we even can not count on this 'copy' mode,  we may also not
able to count on other modes to parallel the CE states. I think to do like to
let FE broadcast to other redundant CEs actually also only let the CE partially
know something instead of everything.

---

From: [email-masked]
Date: Tue, 27 Apr 2004 14:08:56 -0400

Hi Weiming,

> communication is fundamental for multi-CE parallelly working scenario 

I guess I don't see this as necessary, though it does make it easier.

> you have a better approach for this issue.

I am not going to argue it is better, but rather it is a scheme for 
making things work in the absence of CE-CE communications.

Basically an FE is responsible for informing all CEs it has association 
with of which other CEs it is in association with - sending 
notification every time a new association is created or is lost. This 
way all CEs know who else is talking

Additionally an FE needs the capability to report it state to an CE.  
So any CE in association can request a reporting of the FE state (this 
is needed in general, not just for this purpose - I think)

> little help for the CE to even know who the FE used to talk with.

[ad] I agree that knowing who had been Primary (assuming there is only 
a single primary) is not as useful in this case.  Still the Fe should 
be able to supply a rather complete picture of its state if so 
required.

>>>> not, because we can assume at this time, the secondary CE has 
>>>> already
>>>> had the

[ad] Yet even that mode is not fool proof since there may be a gap 
between the last CE copy state operation and the time of the failure.

[ad]Please note I am not saying that CE-CE coordination is not a good 
thing, I just don't think it should be required and even if required I 
don't see it as infallible.  There still needs to be a way to query the 
FE to show its complete state and it still has to be up to the FE to 
inform its CEs of new CEs in the mix..

a.

---

From: Jamal Hadi Salim <[email-masked]>
Date: 27 Apr 2004 15:00:08 -0400

> be able to supply a rather complete picture of its state if so 

Ok, i think i finaly got what you are saying and i think it is
a valid scheme. Since we are already putting responsibility to the
FE to be aware of all CEs in multiCE setup, i dont think this additional
item will be that much of a cost either performancewise or protolwise
(there will be some cost).
OTOH, i think if a CE-CE setup decides it wants to likewise pass such
information around, that too should be fine.
I am not sure how to reach a compromise: Should the setup make the 
decision?

cheers,
jamal