Subject: [Forces-protocol] Protocol Messages: topic 3 & 4a

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 27 Apr 2004 15:49:54 -0700

Hi All,

Since the discussion around the HA seems to be coming to a reasonable
conclusion, 
I am taking this opportunity to post some text around the Protocol
Messages (PL level).
This covers both topic 3 (capability exchange) & 4 a,b.

Couple of comments...I have categorized the messages in a certain way,
however that is Not important to me at this point. I am fine with
however, you guys would like to categorize this or Not categorize this
at all (?) Also, I have tried to incorporate some of the comments that I
received from Weiming, Jamal, others.

Note to Weiming: The reason for not categorizing these messages as "FE
messages" is so that they can be used by the CEs also in the future. For
e.g., the Leave message can be used by the CE also to leave a ForCES
association with a FE (actually this is a requirement as well but I wont
mention the RFC here, don't want to piss off my friend(Jamal) :-)).
Also, the State messages such as UP/DOWN can be used by CEs also thus
making the protocol future-proof in a way & simplifying some of the HA
schemes.

* Association Setup Messages
Join Request/Response - The Join Request is used by the FE to join or
start an association with the CE. The Join Response is sent by the CE
and indicates whether the Join Request was successful or not. The CE
also assigns the FE ID to the FE in this message.
Leave - This message can be used by an FE or CE to leave an association

* Capability Control Messages
Capability Req/Resp (only FE Caps which will be defined in the Model) -
I wanted to keep the Caps Query separate from all other Queries.
However, if folks wont to only use the Query Req/Resp (below) for this,
I am fine with that.
Query Req/Resp (including FE, LFB Properties as well as LFB topology &
Inter-FE Topology)
Configuration Req/Resp (FE, LFB attributes)
 
* Control Packet or Traffic Maintenance Messages
CP Redirect to CE - This message will contain the packet to be
redirected along with some control information such as the FE Port ID on
which the FE arrived or the LFB ID on which it arrived.
CP Forward to FE - This message also contains the packet along with
control information such as the FE Port ID or LFB ID on which the packet
is to be forwarded.

* Event Notification Messages
Event Register/De-register Req/Resp - This allows the CE to
register/de-register for FE events, LFB events, packets, etc.
Asynchronous FE Event (including FE events such as DoS events, LFB
events, etc)

* State Maintenance Messages or HA related messages
PE Up/Down - These indicate the state of the FE or CE.
PE Active/InActive - These are actually commands which can be used by
the CE to Activate or Inactivate an FE (change its state). By activating
an FE, I mean asking it to start forwarding packets. This is a useful
during initial startup of protocol operations, to take the FE into
steady state. These commands can also be very useful to handle HA
scenarios when there are multiple FEs connected to the CE, some are
active and some are standby (or inactive).
Move PrimaryCE - This command is used to change the Primary CE for an
FE.

TBD...(Not sure if this is needed, may be Config Req/Resp can be used
for this)
* Management Object or SNMP support Messages
MO Get/Set
MO Response
HeartBeat or Keep-alive - This message may Not be needed, depending on
whether we want this functionality to be provided by the TML

Changes since last post on Protocol Messages...
Removed - Leave Response
Removed - Vendor Specific messages
Added - Move message

Hope we can get agreement on this basic set, let me know what you guys
think.
I will send out a separate email, with some diagrams describing the
Protocol scenarios showing the different messages.

Regards
Hormuzd

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 27 Apr 2004 22:40:33 -0400

> This covers both topic 3 (capability exchange) & 4 a,b.

Non-tech question:
Where are we going with all the txt that so far you are the serial
interface to. From the size of it, it seem inadequate to be going into a
draft. I know sooner or later we are going to start editing xml pieces.
Is this text a lead/summary to that?

> messages" is so that they can be used by the CEs also in the future. 

I like this. the ID should clearly identify if it is a CE or FE, no?

> For

I am trying my best not to bring up how we did things in order to
expidite things(as an example, the way HA is being approached is totaly
different from how we did and i hope it makes Ram happy ;->). In some
cases, however, sharing the experience helps.
We implemented slightly differently. The FEM CEM interface provides the
FE its ID. The join/syn/association message could of course be used to
override this ID. Infact, in some of the h/ware we used the FE ID could
be associated by slot and chasis ID(sometimes referd to as geographical
information) which is known to the FE on bootup.

> Leave - This message can be used by an FE or CE to leave an association

nod.

> Configuration Req/Resp (FE, LFB attributes)

I think we may need to explicitly call out details. 
For one exhaustive listing of all sorts of capabilities that need to be
exchanged.

> control information such as the FE Port ID or LFB ID on which the packet

The operative term is "such as"; means that there could be more than
what you list. Otherwise above is fine.

> Asynchronous FE Event (including FE events such as DoS events, LFB

We did things differently, but above is something i can live with.

> * State Maintenance Messages or HA related messages

Call them maintainance messages.

> PE Up/Down - These indicate the state of the FE or CE.

Sounds like an event to me.

> active and some are standby (or inactive).

Could you remove "PE" from that? Should probably be activate/deactivate

> Move PrimaryCE - This command is used to change the Primary CE for an

Again, probably "move" is sufficient.

> * Management Object or SNMP support Messages

Not sure of need for this.

> whether we want this functionality to be provided by the TML

I think there is desire to have heartbeats at PL level. So a hearbeat
message is needed.

> Hope we can get agreement on this basic set, let me know what you guys

I am a little slow right now, seems like you covered almost everything.

> Protocol scenarios showing the different messages.

Hormuzd, can i accuse you of hogging this? ;->
What text can i write? I asked in the other email

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 28 Apr 2004 21:33:41 +0800

Hi Hormuzd, Jamal, and all,

----- Original Message -----
From: "Khosravi, Hormuzd M" <[email-masked]>

[Weiming] I suggest that we first list out all possible and necessary messages,
then decide if they should be categorized or not. Therefore, the most important
for current discussion is not to miss something (messages).

[Weiming] From my viewpoint, this (for future use) may actually become a reason
why we need to explicitly indicate the message functioning direction. I'm sure
in many cases, the operation toward FE is different from that toward CE even if
they have the same operation name, e.g., if we allow FE to query CE in the
future, the message formats for CE->FE and FE->CE are surely diffrent because CE
does not have sth like CE model, LFB. I'm also sure we will have messages that
can only be allowed functioning in one direction, e.g., config. msg. which is
only allowed  from CE->FE. Unless we decide that our model is purely
peer-to-peer instead  mainly Master-slave based. (Jamal, Avri:
we may need to lunch this master-slave issue discussion right now along with the
protocol message discussion).  Back to Hormuzd's question, another reason to use
the FE/CE prefix is it may be more clear in the meaning and may be easier for
customers to use.

Talking about the join message, I think the meaning FE uses is different for
that CE (in the future?) uses. A FE actually sends join request message to ask
to join, while a CE sends (if needed) only join command to inform the FE of its
joining.

Besides the join req/resp message, I also have more thinking on the proposed
Req/Resp message action mode:

1. I think in a mainly Master-Slave based model, the Req/Resp mode should mainly
be used by messages from FE->CE, while for CE->FE msgs, most of them are
commands, therefore I think of if it is proper to use word "Request" in this
case. They are just commands, therefore it may be more comfortable to say
'config message' instead of 'config request' message.

2. In many cases, Respose msgs are not so necessarily needed, too much use of
them may unnecessarily  consume more resources. Also, remember we even have ACK
mechanism to do the same thing. Even if in some cases we need to define response
messages, I have a suggestion that we use a flag in the header or body to allow
customers to have a choice whether or not to ask responses.
One more thing for response msg to use is we should note the "command result"
format (as used in FACT) in the resp msg will eventally
be defined by our protocols instead by FE model, I'm just afraid FE model will
not be able to define such things.

Jamal: I'm afraid what I here talked about the msg direction is a little
different from what you may mean. I just mean that we may need to tell customers
sth like 'this message can only be sent from CE to FE, and not allowed from FE
to CE'. I see what you mean may be for those msgs that can be sent in either
directions and you propose that the direction can be implicitly defined by
defining the Src ID and Dest ID. I agree that in this case, we may only need to
define one msg for both directions, while let the FEid and CEid to decide the
direction. And I even think that such direction is very obvious during
implementation, because we may never expect a FE will get messages from other
FEs.
Talking about this, I have to mention our old drafts again. I think the Netlink2
addressing scheme can tell wether the message is from CE or FE,  because the PID
is universal and the 'Src PID' "Dest. PID' can either be FE or CE, but FACT and
GRMP schemes can not. Therefore, I'm afraid here we also have to discuss the
common header and addressing mode first. I think we need now decide if we use
static field in the header for CE id or FE id, or we just use 'Src. ID' and
'Dest. ID'. In the latter mode, we need to define the address space separately
for FEid and CEid.
[/Weiming]

[Weiming] As I mentioned above, the FE join may be a little different from CE
join.  The former is a request, the latter is a command or inform.

[Weiming] As mentioned above, If we decide to use 'Src.ID' and 'Dest.ID' fields
instead of 'FEid' and 'CEid' field in the header and let FEid and CEid
universally defined, we then only need one message, or else, we may need 'FE
leave' and 'CE leave', two msgs.
[/Weiming]

[Weiming] Can we currently just skip the category name for these messages,
because I'm a little confused with the meaning of 'capability' here.

[Weiming] I agree to separate capability query from general query here, but I
just prefer to use the name '(FE) Capability Query/Resp'instead of 'Capability
Req/Resp' for this message, the reason is as I mentioned above. Here the 'resp'
is necessary.

Query Req/Resp (including FE, LFB Properties as well as LFB topology &
Inter-FE Topology)
[Weiming] Again, I prefer to use the name '(FE) Query/Query Resp' for this.
Here, the 'resp' is necessary.

[Weiming]Still prefer '(FE) Configuration message' instead of 'configuration
request message' as the name. Note that here the 'resp' is not always necessary.
For the contents,it may also include config of LFB topology.

[Weiming]Actually here I think we may only need to define one message the
'Packet Redirect msg', leaving addressing body (FEid CEid) to decide the
direction. I don't think the FE port ID or LFB ID here will take effect for the
FE to correctly get the packet to be redirected. This should highly rely on the
LFB topology to decide where to induce the packets, e.g., from a classifier or a
rate limiter (if used for DoS). I suppose that FE model should define a default
datapath ID, via which packets enter and are encapusulated as a protocol
message. The same is for packets redirected from CE to FE.

[Weiming]In my opinion, we truly do not need to use two specific msgs for event
register/de-register. It can be easily done by 'Configuration' msg. no?

[Weiming]Firstly, I strongly suggest we devide it into FE and CE. They are so
different. Then, I think we may only need two messages:

FE maintenance msg (CE->FE)
    -including FE Shutown/Activate/Deactivate/... Note that for FE state Up/Down
report, I'm with Jamal that they should be FE events.
CE maintenance msg
    -including move operation
To be hornest, I'm still not sure how the CE maintenance msg works, even not
sure if it is a CE->FE action or FE->CE. Jamal, could you give more details on
how such as Move action works?
[/Weiming]

[Weiming]I think here we may have missed something that are so important. How we
manage the LFB state, that is, how we lunch(mount), activate/deactivate all the
LFBs? Can they be included in the 'FE Config msg'?

[Weiming]We have never had a discussion on how SNMP can be supported from the
perspective of protocol. Can we have some on it? What I think is just a case
where the SNMP MO and the SNMP agent are located separately in CE and FE. E.g.,
if an agent is in CE while the set MO is in FE, how then we deal with this if we
do not have some protocol message support? A probable way is to define all MOs
as FE attribute then to use Config msg to do it. But is is possible to let all
MOs be FE attributes?

[Weiming]I more prefer to have it as events, therefore, then we can flexibly
register/de-register it.

Regards
Hormuzd

[Weiming] Sorry have made the mail too long.

Cheers,
Weiming

---

From: [email-masked]
Date: Wed, 28 Apr 2004 14:16:49 -0400

On 27 apr 2004, at 22.40, Jamal Hadi Salim wrote:

> I know sooner or later we are going to start editing xml pieces.

Just wanted to let people know that the shell of the draft with all the 
pieces is ready.  So whenever everyone is ready to start dealing with 
it, they can.

a.

---

From: Jamal Hadi Salim <[email-masked]>
Date: 28 Apr 2004 23:35:59 -0400

Weiming,

> the FE/CE prefix is it may be more clear in the meaning and may be easier for

To take from netlink2 experience: The important piece is the end
processor of the "command".
If direction is CE->FE thats a config. If direction is FE->CE typically
CE would intepret that to be an event. 

> to join, while a CE sends (if needed) only join command to inform the FE of its

I actually dont like the name "join". SYNchronize or boot are probably
more descriptive. 
Direction of CE->FE could be interpreted by FE to imply that a CE is
rebooted/restarting which would cast doubt on its (FE)state.

> 'config message' instead of 'config request' message.

Refer to my comments above.

> to CE'. 

Refer to my comment above. I think the message can be sent in either
direction; how its intepretted may be the issue.

> I see what you mean may be for those msgs that can be sent in either

Refer to my comments above and lets discuss

> 'Dest. ID'. In the latter mode, we need to define the address space separately

Refer to my comments on intepreting msgs instead of labeling them that
they are for a specific type of node.
Addressing should direct message to specific types of nodes.
When they get to the node, the execution should be the nodes decision.

> join.  The former is a request, the latter is a command or inform.

I think we can make great use of CEs sending SYNs to reset FE states.

> Leave - This message can be used by an FE or CE to leave an association

agreed.

> leave' and 'CE leave', two msgs.

Refer to my earlier comment. I will cut my response to here because i
think the rest of your message has mostly got thoughts based on
direction of message and CEid vs FEid and i will keep repeating the same
thing. Lets discuss above then we can continue with the rest.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 29 Apr 2004 13:23:29 +0800

Jamal,

Thank you for the reply.

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>

> CE would intepret that to be an event.
[Wweiming]I think this approach (we may also call it implicit approach) may be
quite different from that Hormuzd proposed. Could you try to list more msgs
based on this idea to ease our discussion? I think it is also a feasible
approach. My idea was actually moer based on Hormuzd's proposal, that is, the
config msg is only allowed from CE->FE. In this case, I think to mark the
direction that is to mark as 'FE config' is necessary.
Because the following discussion is more related to this base, therefore I also
just stop here to let us decide this first. Hormuzd, could you have some of your
idea? Thank you.

Cheers,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Thu, 29 Apr 2004 13:53:32 +0800

Thank you so much for the work.
Weiming
----- Original Message -----
From: <[email-masked]>
To: <[email-masked]>
Sent: Thursday, April 29, 2004 2:16 AM
Subject: Re: [Forces-protocol] Protocol Messages: topic 3 & 4a

> pieces is ready.  So whenever everyone is ready to start dealing with

--------------------------------------------------------------------------------

---

From: Jamal Hadi Salim <[email-masked]>
Date: 29 Apr 2004 08:07:00 -0400

Hi Weiming,

> just stop here to let us decide this first. Hormuzd, could you have some of your

Lets assume a node (CE or FE) receives a message destined to its ID;
such a node will have to intepret and may execute the message - its like
opening a letter with your name(ID) on the envelope.  
The execution could be noop such as silently discarding it or returning 
"please reset your brain" (as in the case of TCP RST) in a response. 

Defining the name of the address to be FEid or CEid doesnt add any
value. Defining the execution of a message when it is received by FE or
CE is valuable. And i think this is done in the protcol description
text.

If you want to go through all the possibilities of the treatment of such
a message then lets take a simple example of of what most of the configs
will be:
"update table X (for LFB Y) with config Z". Essentially all config
messages are really state updates from CE to FE. 

1)lets take any config message going from CE->FE; 
a) received by a FE which is capable of using it: it will be
interpretted and used to configure something. Response of execution
result (maybe a simple ACK) may be sent back to CE
b) received by a FE NOT capable of interpretting it - noop will happen
on the FE; maybe a NACK or reset sent back.

The hope is we dont have too many of #1b above. Even if we did this is a
problem that affects any protocol today.

Lets take the case of FE->CE in which case CE is receiving the same or
similar message

2a) It could be used as a subset (not full set) of Event Notification
Messages. This is what netlink2 did. i.e instead of viewing it the way
FE did as a command, you view it as something that has already happened.
A message of "update table X (for LFB Y) with config Z" as in #1a has a
different interpretation at the CE. CE now sees the same message as
"table X (on LFB Y) was updated with config X".  

2b) You could choose to ignore the message from CE unless it has a
speacial tag to it. Hormuzd refers to it as Event Notification Messages;
i am indiferent and would like to wait until i see his posting of the
Event notification message.

So summary of above:
- we dont need to have FEid/CEid - srcID and dstID are sufficient. In
otherwords there are no ambiguities that need to be resolved.
- #2a and #2b are not very different - the message is the same only
difference is in what color of envelope(header) you use to deliver the
Event Notification Messages; 

cheers,
jamal