Subject: RE: [Simple] Re: Draft-campbell-simple-im-session-01: Head of Li ne Blocking

From: Adam Roach <[email-masked]>
Date: Thu, 24 Apr 2003 14:14:59 -0500

> -----Original Message-----
> From: Paul Kyzivat [mailto:[email-masked]]
> >
> > The relays could trivially enforce ordering. In practice, 
> if a client
> > sends a message that requires using the "large messages" pipe,
> > it will usually be tied up sending the large content to the relay
> > so that  it can't send another message in the same session. If there
> > is a substantial difference in bandwidth, the relay could simply
> > quarantine subsequent messages until it was finished sending
> > the big message.
> 
> How? The messages headers don't contain anything that can be used to 
> order the messages. The only likely candidate is the 
> transaction id, but 
> it is only a token.

Umm... how about the order in which they arrived? FIFO?

/a
_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Ben Campbell <[email-masked]>
Date: 24 Apr 2003 14:32:52 -0500

> Umm... how about the order in which they arrived? FIFO?

That doesn't seem to work if you have 2 relays that send the messages
for a single session over 2 or more connections. For example, assume a
large message, followed by several small messages arrive on a given
session. Relay 1 opens a separate connection for to relay 2 on which to
send the large message. Does it still send the smaller message on the
first connection? If so, then they are all likely to complete deliver to
relay 2 before the large message. So from relay 2's perspective, the
large message arrived _after_ the short messages.

You could be around this by simply moving _all_ messages on the session
to the second connection after a message hits the size threshold.
Otherwise, you could base reception time on when you _started_ to
receive a message, but in the above example, that would require the
relay to buffer the short messages until it had completed the transfer
of the long message.

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 24 Apr 2003 15:43:38 -0400

Adam,

What you say makes no sense to me. Assume I have a client C, a server S, 
and a relays R1 and R2. Between R1 and R2 we have two connections, one 
for small messages and another for big messages. There is a single 
connection between C and R1, and one between R2 an S. Lets also assume 
that the connections C:R1 and R2:S are much faster than the two R1:R2 
connections.

Now C sends four messages M1...M4. M2 is big and the others are small. 
R1 receives M1 and sends it on the small connection, sends M2 on the 
large connection, then sends M3 and M4 on the small connection.

There is a fair chance that R2 will receive M1, M3, and M4 before it 
finishes receiving M2. Now what kind of FIFO ordering does it use to put 
these back into proper order?

	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 24 Apr 2003 21:12:53 -0400

Adam Roach wrote:
 > That's what I mean by quarantine: R1 does not send M3 or M4 until
 > it gets a response for M2.

Ahh, I see now. This is equivalent to saying that you can only move a 
stream from one connection to another when it has no messages in transit.

Your solution is quite similar to Ben's suggestion to move all the 
sender's traffic to the big-message connection when the first big 
message is seen. Yours is no harder and may provide somewhat better 
average latency.

But this will aggrevate a problem I realize we haven't addressed - flow 
control. (And lack of flow control in SIP is what led to MSRP in the 
first place.)

Messages can stack up in a relay waiting to get out of quarantine, or 
simply to get into a slower downstream connection. The relay has no way 
to put back pressure on the sender, so it is stuck.

There is indirect back pressure in the sense that no response has been 
received for the messages outstanding. But it isn't clear how much 
pressure that is. The client may be willing to wait forever for responses.

This is simply a flow control problem - just like in real routers.

We could impose fixed, or variable, windows for clients, in the form of 
either bytes outstanding or messages outstanding, or both. Of course 
fixed windows will be suboptimal and variable windows will be complex.

Now that I think about it, it seems like we need some kind of solution 
here even if we do chunking.

	Paul

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 24 Apr 2003 22:36:45 -0700

I'm not a big fan of the multiple TCP pipes solutions because of the problem
of too many connections already and TLS setup times but I think this does
bring up a good point. If we are going to allow the system to someday be
upgraded to SCTP, we might want to say that client is not allowed to overlap
messages to the same destination.

Cullen

---

From: Ben Campbell <[email-masked]>
Date: 25 Apr 2003 09:52:20 -0500

> here even if we do chunking.

Would it be sufficient to simply add the equivilant of a 503 response,
that would mean something to the effect of "quit sending stuff for a
while" with a retry-after header?

Although in all honesty, I am starting to feel more and more that we are
reinventing BEEP.

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 25 Apr 2003 15:36:15 -0400

>>This is simply a flow control problem - just like in real routers.
...
> Although in all honesty, I am starting to feel more and more that we are

I have felt this way for quite awhile. The trick is figuring out if this 
means we should be reusing something else (BEEP, XMPP, SCTP, ...), that 
we are doing too much, or that in spite of everything this reinvention 
is justified.

Regarding 503: while I work for Cisco I am not a router guy. But I know 
extra messages plus retransmission often does more harm than good for 
dealing with congestion. Whether that is true here requires more thought 
than I have given it, and probably more knowledge than I have.

	Paul

---

From: "James Undery" <[email-masked]>
Date: Fri, 25 Apr 2003 21:45:45 +0100

inline
----- Original Message ----- 
From: "Ben Campbell" <[email-masked]>

> Although in all honesty, I am starting to feel more and more that we are

Very true, and my gut feeling is that if we add chunking we are leaving are
ourselves open to state loading DOS attacks that can be better solved with
TCP (and the associated HOL problems) or SCTP. My fear with DOS attacks is
based on the fact that with Cullens' 5GB files we don't know when to drop
the state but at least with TCP/SCTP timeouts become more reasonable, as
opposed to is this stream halted because of other traffic questions. (But I
could be wrong here.)

James

---

From: Sean Olson <[email-masked]>
Date: Fri, 25 Apr 2003 13:51:31 -0700 (PDT)

> than I have given it, and probably more knowledge

I would second Paul's concerns here. The retry logic
must be crafted very carefully to avoid making a bad
situation worse, especially if the primary goal of
MSRP is to provide congestion control (vs. SIP)

__________________________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo
http://search.yahoo.com
_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: Ben Campbell <[email-masked]>
Date: 25 Apr 2003 16:29:07 -0500

> opposed to is this stream halted because of other traffic questions. (But I

I am not entirely sure I follow you. Even if we added chunking, it would
be over TCP or equivelant. Are you suggesting that with TCP, chunking is
not needed?

---

From: "James Undery" <[email-masked]>
Date: Fri, 25 Apr 2003 22:41:11 +0100

----- Original Message ----- 

> be over TCP or equivelant. Are you suggesting that with TCP, chunking is

My thought is that some HOL fixing mechanism is needed for large files, but
I am not sure chunking is a good idea. My problem is that you get a 5GB file
all but the last "chunk", how long do you wait to discard it, with a stream
based protocol (e.g. TCP without chunking or SCTP) a timeout is appropriate,
but when the data is arriving in datagrams (or chunks) you have to take in
to account other factors (e.g. network traffic conditions), however, as I
said it's a gut feeling, ignore at your leisure.

James

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 28 Apr 2003 13:18:41 -0400

> said it's a gut feeling, ignore at your leisure.

Any solution that presents the relays with arbitrarilly large messages 
and no way to chunk them presents a DoS opportunity. This gets worse if 
the relay may be forced to queue lots of these messages.

So I think either messages must be restricted to some reasonable bounded 
size, or else the relay must be able to chunk large messages on its own.

This seems to be a converse to your conclusion.

	Paul

---

From: "James Undery" <[email-masked]>
Date: Mon, 28 Apr 2003 20:23:21 +0100

----- Original Message ----- 
From: "Paul Kyzivat" <[email-masked]>

> the relay may be forced to queue lots of these messages.

I guess what I really meant to say is chunking requires another mechanism to
prevent state loading and deal with HOL effectively.

> size, or else the relay must be able to chunk large messages on its own.

No the whole point should be to not restrict sizes, relays handling chunking
is more complex than simply dividing up the messages, I guess what I really
mean is don't reinvent SCTP, use it or swallow the associated problems of
TCP (and use some Relay configured policies to restrict message sizes in
these inferior TCP using implementations).

James

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 28 Apr 2003 15:33:54 -0400

I haven't followed the 64K messages on this topic, but I would be very 
concerned about conclusions that SCTP actually helps with HOL blocking. 
SCTP cannot throttle one stream, so if you have 'forking', as in

                                       //-----\\
                                     ||         ||
                                      /         |
                                    // \\-----//
                                  //
                                //
                              //  slow link
                            //
               +-------+  //
               |       |//
               |   P   --
  -------------+       | ----
               |       |     ----
               +-------+         ---
                                    ----  ///-----\\\
                                        ++           ||
                           fast link     |           |
                                          \\\-----///

the left-most link, SCTP or not, will get blocked waiting for the 
slowest outbound link. Chunking doesn't seem to help all that much 
either - the basic problem is that P has no way to tell the previous 
node that sending messages for the fast link is ok, but messages for the 
slow link should not be sent.

Henning

---

From: Alex Audu <[email-masked]>
Date: Tue, 29 Apr 2003 10:12:58 -0500

I am just catching up with this thread,..but I felt I had to make a comment
here.

SCTP's design allows application to select whether its data is to be
delivered to it
in-order or not.  If you do not select in-order mode, then SCTP delivers the

packets as they become available. Thus packet N can be delivered before
packet N-1;
we don't have to block or wait for packet N-1 to arrive before delivering
packet N.
That resolves HOL blocking issue. Am I missing something here?

If you use different SCTP streams for the slow and fast flows, that should
prevent
events in the slow flow from affecting the fast flow.

Regards,
Alex.

---

From: Alex Audu <[email-masked]>
Date: Tue, 29 Apr 2003 10:25:27 -0500

Also, SCTP can ensure that messages are delivered to the SCTP user in
sequence whithin
a stream (in-sequence delivery mode). When a stream  is blocked waiting for
the next
in-sequence user message, deliverey from other streams can proceed.

Alex.

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 29 Apr 2003 11:32:08 -0400

The problem is that there is no separate flow or congestion control for 
each stream and you cannot selectively pull packets for a particular 
stream. Thus, in the figure that I drew, the node P can't just remove 
the 'fast-link' packets and tell the sender to slow down the 'slow-link' 
packets. The sender will inject packets as they arrive, and the poor 
receiver will get the packets, furiously stow the 'slow-link' packets 
into the ever-increasing queue for the slow link and forward the fast 
packets, until such time that it either
- needs to stop receiving packets altogether (block whole inbound link)
- or drop slow-link packets on the floor.

Thus, in the heterogeneous next-hop model, SCTP multiplexing does 
exactly nothing to prevent blocking or flooding. The use of the term 
HOLB, as others have alluded to, is thus somewhat misleading as it leads 
the SCTP advocates to jump to the wrong conclusion. SCTP only prevents a 
very limited form of HOLB, where buffering at the end system is not the 
problem, but waiting for a long segment is. It is roughly equivalent to 
what HTTP/1.1 chunking does.

Note that this behavior offers ample opportunity for mischief. A 
black-hat next hop can pretend to be slow and, if the node isn't 
careful, throttle or block all streams. Thus, the middle node has to 
drop packets. This means that we now not only have to re-invent flow 
control at the application layer, we also have to add another 
reliability mechanism to the whole chain. At that point, may I suggest 
we ask that SIMPLE be moved to the Transport area?