Subject: Re: [Forces-protocol] topic 7: HA

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 7 Apr 2004 22:35:12 +0800

Hi Jamal,

I have some ideas on the TML of its functions like HA, which I'm not sure if the
same as what you think. Let's just discuss it:

1) The interaction mode between PL and TML
Although I agree with you that TML itself controls the connections with their
state, I think such control should be under the control of PL as well as
pre-association protocol. On the whole, I think TML should work in a passive
mode, with its intelligence limited to a perceivable scope by PL or PAP. As
such, my idea is that the interaction between the PL and TML may proceed via two
items:

a) PL controls TML via some TML setup attributes, which may be from CE to FE via
ForCES protocol or directly inside CE from PL to TML.
b) TML may trigger some event reports to PL, which are at FE side as a kind of
FE events.

Then, what we need to discuss is just what attributes and events (actually apear
as FE attributes and FE events at FE side) are needed and how they are defined
in PL. This actually relates to the discussion of how much the TML should have
the intelligence, and which is better, more or less.

Coming back to the HA, I think we then can reduce our discussion on what kind of
FE attributes and FE events are needed for HA, and among them, what are to be
transmitted to TML(as you said, are transparent to PL) and what are needed to
take functions within the PL.

My some more comments are as inlines below.

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>

> This text does not talk about state synchronization or restarts
> or rollovers. Those details are considered solvable if the
> discussion below is resolvable.
>
> Two levels of HA  responsibilities that i see:
>
> 1) TML level:
>
> a) The TML controls logical connection availability and failover.
> b) The TML also controls peer HA managements.

[Weiming] I agree with such control. But seems we still have a depth selection
for the control:
(1)Control to only let TML able to detect the connection availability and
failover for already setup connections, then trigger event report to PL.
(2)Under some guidance from PL, the TML can reach the high availability as well
as detect the state.
(3)Can intelligently reach the HA without PL guidance.

For all the three controls, we may have different TML HA attribute contents for
the PL to setup it.

>
> 2) PL Level:
>
> Depending on the scheme implemented by the TML level, the PL
> level may have no role to play i.e HA being transparent to it.
>
[Weiming]Yes, if we choose to let TML have enough intelligence. But I think when
we designe the protocol, we may be better not to exclude the support for some
TMLs that may not have enough intelligence.

> The TML layer
> -------------
> To describe the detail,  a good example would be  to show multiple
> CEs trying to control an FE.
>
> For all three schemes, a heartbeat that terminates at the TML layer
> is needed for meeting requirement 1). The are three schemes i can
> think of which i will list here:
>
> Scheme 1) Le brute force:
> The FE TML is aware of the multiple CEs i.e knows their logical
> connections.
> It may receive commands from all the CEs but will process the one with
> highest priority only in given time instances.
>
> Protocol requirement: To have priorities in the headers. We already do.
>
> Advantages: KISS.
> Disadvatanges: too much traffic
>
> Scheme 2) the Tenor:
> The FE TML is aware of all CEs controlling it however responds to only
> one at any one point.
> Something akin to HA as achieved by SCTP multihoming or some
> mobility protocols. i.e at some point the current known master is
> obsoleted for example because of some missing heartbeats or physical
> disconnect or by explicit instructions to migrate.
>
> Protocol requirement:
> A "move" command to instruct to a migration to a new CE being issued
> to the FE.
>
> Advantages: less traffic
> Disadvantages: slower in recovery compared to 1).
>
> Scheme 3) the soprano:
> The FE is NOT aware of the CEs unlike 1 and 2 above.
> The election of which CE is controlling an FE is done elsewhere
> at the CE level.
> At any point the controlling CE will use a logical address.
> This makes it transparent to the FE how many CEs are behind the control
> point. Very easily doable with a broadcast or multicast protocol.
> Example protocol which does this is VRRP.
>
> Protocol requirement: None, the TML is burdened with providing either
> multicast or broadcast.
>
> In terms of compute requirements on the FE a) > b) > c)
> In terms of bandwidth requirements on the wire a > b > c
>
[Weiming]When I checked the three schemes, I just have a feeling that it's the
difference of control depth from PL(as I mentioned above), or from other point,
the difference of the TML intelligence. that is,

In terms of control depth from PL: a<b<c
In terms of the TML intelligence: a>b>c

Therefore, I just suppose what we need to discuss is how such control can be
efficiently described in the protocol layer. From your above text, you mentioned
the protocol priority may have something to do with it. I think we have much
more. My idea as described at the head is we need to define some HA (more widely
TML) attributes for such control policies.

> The PL Layer
> ------------
> The PL layer provides HA management of the LFBs and their availability.
>
> Scheme 1)
> The PL heartbeats terminate at the TML layer.
> The TML layer aggregates info on its PLs into its own heartbeats
> to its peers.
> The TML peer uses the knowledge to make calls on failover.

Best regards,

Weiming

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: [email-masked]
Date: Thu, 8 Apr 2004 23:41:56 +0800

Hi Jamal, 

I'm glad we are basically with the same viewpoint. Pls see more comments inline.

----- Original Message ----- 

> Weiming,
> The first portion of your email  touches on generality of the TML<->PL
> interface and the second (a response to my email) is on one specific
> thing - HA. I am separating the two issues with two emails for clarity.
> 
> Its good you brought this up at this point.
> 
> On Wed, 2004-04-07 at 10:35, Wang,Weiming wrote:
> > Hi Jamal,
> > 
> I am with you on this.
> We definitely need to have an interface between PL and TML. I realize
> this may be adding more complexity but it would be necessary if we have
> to tell people to write TMLs.
[Weiming]This maybe also that Hormuzd means for TML requirements, but I'm 
afraid they not just requirements, instead they are items that should be 
defined explicitly in the Protocol messages. 

> I think theres one more view. Each Local PL may instruct its local TML
> on certain things. Basically, theres an API/protocol between the PL and
> TML; with control being driven by the PL. For generality, a PL may need
> to instruct a remote TML. Is this going overboard?

[Weiming]Not overboard, I think. Actually by using attributes defined in the 
protocol, we have implied this control may be from remote, that is, may from CE 
to some FE. From your schemes presented last time on the priority for multi-CEs 
controlling a FE, I suppose you actually imply the TML control may be via 
metadata in the protocol messages for remote control. 
Talking about metadata, after I posted my last email, I have also noticed that 
we may need never able to exclude the control from PL to TML via a per packet 
metadata, because a broadcast/multicast ID actually acts like this kind of 
metadata. But what we need to discuss to find out is what attributes and 
metadata we on earch need. I have a basic thought that we may only use metadata 
when we are sure attributes are not feasible for a goal, for it seems 
attributes are more flexible, while some metadata may have to be defined in the 
protocol more strictly, but I'm still not sure of this. Do you have more ideas 
on how metadata may look like from protocol layer?

> Events from TML to PL are the other piece.
> 
> Theres another piece: Should interface between PL->TML be on a out of
> band ctrl or metadata carried with packet? Example in the case of
> reliability: I may want a specific message to be reliably delivered or i
> may request all packets be delivered reliably. In the later case, it
> makes sense to just send a attribute request to be reliable. In the
> former case, i could attach metadata to the message if it is an
> exception.
> 
[Weiming] Agreed. We may need to find more examples on the need for the 
metadata like control of TML.

> Yes, HA is one item which applies to such interface - although it doesnt
> have a per-packet semantics: others i can think of that will have per
> packet semantics are: Reliability, Security and priority. 
> 
[Weiming]One more important example for per-packet control is 
broadcast/multicast support.

Cheers,
Weiming

-------------------------------------------------
This mail sent through HUCNIC Webmail system.

---

From: Jamal Hadi Salim <[email-masked]>
Date: 08 Apr 2004 10:36:06 -0400

Weiming,

I am with you on this.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: 08 Apr 2004 11:45:34 -0400

BTW, Is the mailing list still broken? I look at the time stamps and it
is taking a looong time to get messages.

> (3)Can intelligently reach the HA without PL guidance.

The way i see it above is the TML may be capable of all the three, but
policy from PL may impose 3 for example. 

> TMLs that may not have enough intelligence.

Agreed. Again, some TMLs may not be capable of all three possibilities.

> In terms of the TML intelligence: a>b>c

Its both actually. If the TML is not capable of all three ways of
control, then the workload is on the PL. 
I would say the TML will have some way of conveying such capability to
PL and once the negotiation is complete, the decision on which HA scheme
would be made.

> TML) attributes for such control policies.

As i pointed out in the earlier email, i think theres two issues: first
being the interface between TML to PL and other being the specifics on
HA. 
To be honest when i posted the first text i was trying to exhaustively
list all the possibilities so we could choose one or a hybrid ; this
discussion leads me to believe we should be able to support all three.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: 08 Apr 2004 15:33:27 -0400

Weiming,

BTW, this mailing list latency is horrible; not to forget it has
swallowed at least 2 of my messages that i havent sent in the past. This
is not proper for a discussion. Maybe we should look for alternatives.

Greg, can you confirm that you received this message and that the
mailing list logs show it was received?

> defined explicitly in the Protocol messages. 

Ok, so we may be approaching this from a different angle. I am thinking
there may be some messages which are just local; eg via an API such as
(invoked by PL):

TML_ioctl(fd, USE_HA_MODE2)

You are thinking that infact the PL constructs a message addressed to
the TML which may be terminated localy. It could also be terminated at a
remote TML. Did i understand you correctly?

> controlling a FE, 

Ok this validates my earlier point that you are looking at this from a
purely messaging/protocol view. Which is fine with me but needs some
clarity in regards to addressing. 

> I suppose you actually imply the TML control may be via 

I am just thinking actually the metadata is mostly to instruct the TML
on how to deal with the message. Not to be passed to the PL layer. IOW,
you could send a message to TML to be forwarded to one or more FEs but
you want that message to be delivered reliably on some occasions but not
on others. In such a situation you will have something attached to the
message to instruct it to do so. I am thinking probably a TLV.

[..]

> broadcast/multicast support.

This one may be tricky. Do you want the PL layer to know about details
of how the message is transmitted? Shouldnt the addressing take care of
it?

cheers,
jamal