From owner-ecm@wyvern.aciri.org Sun Jul 2 12:17:40 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA08709 for ; Sun, 2 Jul 2000 12:17:40 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id JAA44081 for ecm-outgoing; Sun, 2 Jul 2000 09:16:58 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id JAA44076 for ; Sun, 2 Jul 2000 09:16:57 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id MAA28364; Sun, 2 Jul 2000 12:16:56 -0400 Message-Id: <200007021616.MAA28364@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 X-url: http://nms.lcs.mit.edu/~hari/ From: Hari Balakrishnan Reply-to: Hari Balakrishnan To: ecm@aciri.org cc: srini@seshan.org, hari@lcs.mit.edu Subject: draft-ietf-ecm-cm-00.txt submitted Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 02 Jul 2000 12:16:56 -0400 Sender: owner-ecm@aciri.org Precedence: bulk Srini and I just submitted a new draft of the Congestion Manager I-D (new draft name in subject line). It is also available from http://nms.lcs.mit.edu/papers/ draft-ietf-ecm-cm-00.txt We'd like to get some discussion on it before the Pittsburgh IETF meeting. To this end, I'll summarize the issues in two parts: (I) open issues and (II) changes from previous version. (These points will make more sense after reading or at least perusing the submitted draft.) (I) Open issues 1. Prioritization. In the current CM draft, ALL transmissions have to be OK'd by the CM, which implements a TCP-like window-based algorithm (this last part isn't a requirement, but we expect many implementations to do this). Howeve, this sometimes causes a small problem, which needs to be addressed by the group. It is best illustrated via an example, of TCP retransmissions (assume no SACK). When TCP (Section 6.1.1) recognizes that data has been lost in the network, it first uses cm_update() to inform CM of the loss and, as a result, CM performs congestion control (e.g., window-halving). At this point, CM believes that it has much more data in the "network pipe" than its congestion estimates allow. TCP then performs a cm_request() in order to schedule the *retransmission* of the lost data. However, due to the state of the CM, this request is unlikely to get served for quite some time. Since TCP does not get new ACKs, it passes little new information to CM and the whole process gets stalled. We handle this in our current TCP/CM implementation by bypassing the CM while performing TCP retransmissions---they do not wait for a cm_request() to be satisfied. Retransmissions are sent immediately and essentially ignore the CM. This sounds like a hack (it is) and we need a clean way to handle this, perhaps by extending the API. In general, some modifications need to be made to support any application that *really* needs to send a packet to make progress or obtain additional feedback. One possibility is to add a priority parameter to cm_request() and allow a single high priorty packet to be sent every RTT, independent of the current congestion state. This would allow applications like TCP to schedule retransmissions through the CM. It might even be necessary to restrict access to this to specific protocols (e.g., in-kernel apps). Your thoughts on this would be quite valuable. 2. Buffered transmissions. See Section 4.2. The previous CM draft had three types of transmission APIs. The callback API using cm_request/cmapp_send; the synchronous API using cmapp_update(); and a buffered cm_send API. In the current draft, we removed the third option. The reason is that we found it to be simpler and more elegant to implement buffered congestion-controlled transmissions as part of UDP rather than in the core CM. The advantage of this mechanism is that the CM is simple, and conformant CM implementations are easier to develop; the disadvantage is that the onus for providing congestion-controlled buffered transmissions is pushed to individual transports. As it turns out, it isn't too hard to implement this for UDP, providing congestion-controlled UDP sockets. Do you see any architectural or other drawbacks of this decision. 3. Default granularity. See Section 4.5 This issue has arisen several times on the list. The submitted draft has the following comment about default granularity: The default suggested aggregation method is to aggregate by destination; i.e., all streams to the same destination are aggregated to a single macroflow by default. The cm_getmacroflow() and cm_setmacroflow() calls can then be used to change this as needed. Does the group agree with this, or should it be replaced with a default macroflow granularity of single streams, with the API calls being used on a per-application basis to change the granularity? (II) Summary of changes 1. Many issues pointed out by Vern (message sent to the list on Fri, 16 Jun 2000 15:47:33 PDT (18:47 EDT)) addressed and his comments incorporated; other comments from private email incorporated. 2. cm_send() eliminated; added section on congestion-controlled UDP as a CM application. 3. Clarified TCP description. 4. Fixed many references. For the extremely curious, context diffs between the previous and current versions are at http://nms.lcs.mit.edu/papers/cm-draft-diffs.txt Also, there's a technical report on our implementation, performance and efficiency considerations at http://nms.lcs.mit.edu/papers/MIT-LCS-TR-808.ps.gz (this is an older version of a paper in preparation that will appear at OSDI 2000). Thanks, Hari & Srini From owner-ecm@wyvern.aciri.org Sun Jul 2 12:26:31 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA08751 for ; Sun, 2 Jul 2000 12:26:31 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id JAA44139 for ecm-outgoing; Sun, 2 Jul 2000 09:26:29 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id JAA44134 for ; Sun, 2 Jul 2000 09:26:28 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id MAA28399; Sun, 2 Jul 2000 12:26:27 -0400 Message-Id: <200007021626.MAA28399@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 To: Vern Paxson cc: ecm@aciri.org Subject: Re: comments on draft-balakrishnan-cm-02.txt In-Reply-To: Message from Vern Paxson of "Fri, 16 Jun 2000 15:47:33 PDT." <200006162247.e5GMlYf06115@daffy.ee.lbl.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 02 Jul 2000 12:26:27 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk Vern, Thanks very much for these detailed and useful comments; almost all of these have now been incorporated. > > FLOW > > A stream of packets that all share the same source and > > destination IP address, IP type-of-service, transport > > protocol, and source and destination transport port numbers. > > This term is too easily confused with "macroflow" (there are places in the > document where this happens). The document should just use "stream", which > in its definition is noted as being in one-to-one correspondence with the > notion of "flow", anyway. Good suggestion. Done. > > CONGESTION WINDOW (cwnd) > > A CM state variable that modulates the amount of outstanding > > data between sender and receiver. > > Should there also be a definition for ssthresh? Well, we didn't use it because we just alluded to RFC 2581 when necessary. > Citation to Stevens97 should be updated to RFC 2581. (This occurs > elsewhere, too.) Done. > It's important that all of the discussion of APIs in this document > emphasizes that we're referring to *abstract* APIs. Explicitly mentioned in two separate places now. > > > 1. Open: All applications MUST call cm_open(u32 dst) before using > > the CM API. dst is the 32-bit IPv4 address. This returns an > > There needs to be an IPv6 interface, too. Replaced it with an abstract "stream_info" argument. > Shouldn't doing PMTU-D be a SHOULD? Yes. > > does not transmit any data upon a cmapp_send() callback invocation, > > it SHOULD call cm_notify(dst, 0) to allow the CM to permit other > > flows in the macroflow to transmit data. > > Why give it any sort of leeway here? If the the app doesn't call > cm_notify(), what ensures that the bookkeeping will stay consistent? > (Or is the SHOULD to emphasize that correct operation of the controller > shouldn't rely on the didn't-send notification, because it's an easy > mistake for the app to make?) Notice that this is only for a non-transmission. If the app doesn't call cm_notify(), it isn't the end of the world. All other cm_notify()'s are done from the IP output routine. > > It uses application > > notifications (Section 4.3) from concurrent streams on the same > > macroflow to build up information about the congestion state of the > > different network paths. > > It's not clear what's meant by "different network paths" here. Different > paths taken by the elements of the same macroflow? Or is this simply a > comment that there will be multiple macroflows? If the latter, then I suggest > deleting it. Fixed. > > > The congestion control module MUST implement the following > > interfaces (these are not directly visible to applications; they > > are within the context of a macroflow): > > If they're not directly visible, then why *must* the congestion control > module implement them? If it's to provide for modular interoperability > with different, separately-developed schedulers, then that should be stated. This point is made explicit now. > > - void schedule(u32 num_bytes): When the congestion control module > > determines that data can be sent, the schedule() routine MUST be > > called with the number of bytes that can be sent. > > This MUST reads a little funny to me. Is the point that it shouldn't be > calle with *no more* bytes than the number that can be sent? If so, > then phrasing it "with no more than the number ..." would make it read > a little better. Done. > "control" should be "controller" > > > - void notify(u32 nsent): This interface is used to notify the > > scheduler module whenever data is sent by a CM application. The > > nsent parameter indicates the number of bytes just sent by the > > application. > > Shouldn't the call include the flowid (streamid)? That would simplify the > scheduler's state management. Yes, indeed! > > The IP output routines on the host calls cm_notify() when the data > > is actually sent out. > > How does it know which flowid/streamid is responsible? It doesn't. It calls cm_notify() with the stream_info (which includes src and dst addrs and ports and IP proto). > > Therefore, it calls > > cm_update(flowid, 4 * avg_pkt_size, 3 * avg_pkt_size, > > CM_TRANSIENT, rtt). > > This needs careful explanation. Also, why is it okay to use avg_pkt_size > instead of computing the actual sizes? If this is a general principle, > then there needs to be a section explaining it as such. We've addressed this. > > 5. Upon the arrival of a partial acknowledgment (one that does > > not exceed that last segment transmitted at the time the loss > > "that last segment" should be "the last segment". Or should this > be "the highest segment"? > > > When the source first starts, it uses the cm_query() call to get an > > initial estimate of network bandwidth and delay. > > But there won't actually be an initial estimate available, will there? > (unless some other streams have already been active) Fixed. If there is, it is used, else not. cm_query() returns a rate, rtt, rttdev of -1 if no info is available. We changed the abstract API to make these signed values. Thanks again for your comments! Hari From owner-ecm@wyvern.aciri.org Wed Jul 5 16:41:30 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16600 for ; Wed, 5 Jul 2000 16:41:29 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id NAA65730 for ecm-outgoing; Wed, 5 Jul 2000 13:39:17 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id NAA65725 for ; Wed, 5 Jul 2000 13:39:16 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e65KdGC08927; Wed, 5 Jul 2000 13:39:16 -0700 (PDT) Message-Id: <200007052039.e65KdGC08927@daffy.ee.lbl.gov> To: ecm@aciri.org Subject: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard In-reply-to: Your message of Fri, 16 Jun 2000 15:45:33 PDT. Date: Wed, 05 Jul 2000 13:39:16 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk Reminder, the last call closes this Friday, July 7th. So far there haven't been any comments. If you've already reviewed it and are satisfied with it, please send me a note to that effect (or post to the list), so I can gauge the degree of support for the document. Thanks, Vern From owner-ecm@wyvern.aciri.org Wed Jul 5 16:42:56 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16647 for ; Wed, 5 Jul 2000 16:42:55 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id NAA65760 for ecm-outgoing; Wed, 5 Jul 2000 13:42:55 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id NAA65755 for ; Wed, 5 Jul 2000 13:42:54 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e65Kgn508943; Wed, 5 Jul 2000 13:42:49 -0700 (PDT) Message-Id: <200007052042.e65Kgn508943@daffy.ee.lbl.gov> To: Hari Balakrishnan Cc: ecm@aciri.org Subject: Re: comments on draft-balakrishnan-cm-02.txt In-reply-to: Your message of Sun, 02 Jul 2000 12:26:27 PDT. Date: Wed, 05 Jul 2000 13:42:49 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk Thanks for addressing these points, looks good. One comment: > > > does not transmit any data upon a cmapp_send() callback invocation, > > > it SHOULD call cm_notify(dst, 0) to allow the CM to permit other > > > flows in the macroflow to transmit data. > > > > Why give it any sort of leeway here? If the the app doesn't call > > cm_notify(), what ensures that the bookkeeping will stay consistent? > > (Or is the SHOULD to emphasize that correct operation of the controller > > shouldn't rely on the didn't-send notification, because it's an easy > > mistake for the app to make?) > > Notice that this is only for a non-transmission. If the app doesn't call > cm_notify(), it isn't the end of the world. All other cm_notify()'s are done > from the IP output routine. My concern is that some bookkeeping could get inconsistent if the app doesn't call cm_notify() when it decides not to send. In particular, I'm wondering whether an app could "save up" several credits from cmapp_send() and then dump them all into the network later; I'd like to ensure that that doesn't happen. Is this a problem? Vern From owner-ecm@wyvern.aciri.org Wed Jul 5 23:26:41 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA23747 for ; Wed, 5 Jul 2000 23:26:40 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id UAA67687 for ecm-outgoing; Wed, 5 Jul 2000 20:26:04 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id UAA67682 for ; Wed, 5 Jul 2000 20:26:03 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id XAA12115; Wed, 5 Jul 2000 23:25:57 -0400 Message-Id: <200007060325.XAA12115@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Hari Balakrishnan X-url: http://nms.lcs.mit.edu/~hari/ To: Vern Paxson cc: ecm@aciri.org, sseshan@us.ibm.com Subject: cm_notify() semantics [was Re: comments on draft-balakrishnan-cm-02.txt] In-reply-to: Your message of "Wed, 05 Jul 2000 13:42:49 PDT" <200007052042.e65Kgn508943@daffy.ee.lbl.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 05 Jul 2000 23:25:57 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk Vern, You bring up a good point, which we ought to have brought up as a discussion issue. There are two cases: (i) A CM app hoards credits and eventually "cashes in" on them all in a burst. This is the case you bring up. (ii) One or more CM apps on a macroflow crashes/disappears/goes-AWOL. Case (i) is relatively easy to handle. We do not issue subsequent cmapp_send() calls to a stream while one is pending. In our implementation, this would be done by adding a "pending" flag on the cm_stream struct that is cleared by cm_notify(). Notice that this means that an app MUST do a cm_notify(0) is it decides not to send on a previous cmapp_send(). The current draft is incorrect (SHOULD instead of MUST) on this count. Case (ii) is trickier; crashes always are :}. This requires a timeout before that grant can be given to other streams on the macroflow, and before state can be cleaned up. It's hard to see what the correct timeout should be, but having said that, I don't think it really matters what the timeout is. I would reckon something like "if a stream hasn't used a grant in one RTT, pretend it's been used up." It's not clear to me when to clean up state for an application that may have crashed. Obviously if the process has crashed it's not an issue, but I don't know if that's enough of a check, especially for in-kernel protocols that may get wedged every so often. We'll think about this some more. If anyone on the list has any ideas, we'd appreciate hearing them! Thanks, Hari On Wed, 05 Jul 2000 13:42:49 PDT, you wrote: > Thanks for addressing these points, looks good. One comment: > > > > > does not transmit any data upon a cmapp_send() callback invocation, > > > > it SHOULD call cm_notify(dst, 0) to allow the CM to permit other > > > > flows in the macroflow to transmit data. > > > > > > Why give it any sort of leeway here? If the the app doesn't call > > > cm_notify(), what ensures that the bookkeeping will stay consistent? > > > (Or is the SHOULD to emphasize that correct operation of the controller > > > shouldn't rely on the didn't-send notification, because it's an easy > > > mistake for the app to make?) > > > > Notice that this is only for a non-transmission. If the app doesn't call > > cm_notify(), it isn't the end of the world. All other cm_notify()'s are done > > from the IP output routine. > > My concern is that some bookkeeping could get inconsistent if the app doesn't > call cm_notify() when it decides not to send. In particular, I'm wondering > whether an app could "save up" several credits from cmapp_send() and then > dump them all into the network later; I'd like to ensure that that doesn't > happen. Is this a problem? > > Vern From owner-ecm@wyvern.aciri.org Thu Jul 6 02:18:41 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA08401 for ; Thu, 6 Jul 2000 02:18:41 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id XAA68402 for ecm-outgoing; Wed, 5 Jul 2000 23:16:49 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id XAA68397 for ; Wed, 5 Jul 2000 23:16:49 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e666Gih10529; Wed, 5 Jul 2000 23:16:44 -0700 (PDT) Message-Id: <200007060616.e666Gih10529@daffy.ee.lbl.gov> To: Hari Balakrishnan Cc: ecm@aciri.org, sseshan@us.ibm.com Subject: Re: cm_notify() semantics [was Re: comments on draft-balakrishnan-cm-02.txt] In-reply-to: Your message of Wed, 05 Jul 2000 23:25:57 PDT. Date: Wed, 05 Jul 2000 23:16:44 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk > I would reckon something like "if a stream hasn't used a grant in one RTT, > pretend it's been used up. That sounds like a good approach to me. Vern From owner-ecm@wyvern.aciri.org Fri Jul 7 06:55:33 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA19980 for ; Fri, 7 Jul 2000 06:55:32 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id DAA77691 for ecm-outgoing; Fri, 7 Jul 2000 03:52:08 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id DAA77686 for ; Fri, 7 Jul 2000 03:52:06 -0700 (PDT) (envelope-from nsyracus@cnri.reston.va.us) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA19943; Fri, 7 Jul 2000 06:52:01 -0400 (EDT) Message-Id: <200007071052.GAA19943@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ecm@aciri.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ecm-cm-00.txt Date: Fri, 07 Jul 2000 06:52:00 -0400 Sender: owner-ecm@aciri.org Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Endpoint Congestion Management Working Group of the IETF. Title : The Congestion Manager Author(s) : H. Balakrishnan, S. Seshan Filename : draft-ietf-ecm-cm-00.txt Pages : Date : 06-Jul-00 This document describes the Congestion Manager (CM), an end-system module that (i) enables an ensemble of multiple concurrent streams from a sender destined to the same receiver and sharing the same congestion properties to perform proper congestion avoidance and control, and (ii) allows applications to easily adapt to network congestion. This CM framework integrates congestion management across all applications and transport protocols. The CM maintains congestion parameters (available aggregate and per-stream bandwidth, per-receiver round-trip times, etc.) and exports an API that enables applications to learn about network characteristics, pass information to the CM, share congestion information with each other, and schedule data transmissions. This document focuses on applications and transport protocols with their own independent per-byte or per-packet sequence number information, and does not require modifications to the receiver protocol stack. The receiving application must provide feedback to the sending application about received packets and losses, and the latter uses the CM API to update CM state. This document does not address networks with reservations or service discrimination. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ecm-cm-00.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ecm-cm-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ecm-cm-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000706151842.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ecm-cm-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ecm-cm-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000706151842.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ecm@wyvern.aciri.org Fri Jul 7 13:43:10 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06716 for ; Fri, 7 Jul 2000 13:43:09 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id KAA80532 for ecm-outgoing; Fri, 7 Jul 2000 10:41:45 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id KAA80527 for ; Fri, 7 Jul 2000 10:41:44 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id KAA12398; Fri, 7 Jul 2000 10:41:43 -0700 (PDT) Message-ID: <39661646.71981A41@isi.edu> Date: Fri, 07 Jul 2000 10:41:26 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Vern Paxson CC: ecm@aciri.org, touch@isi.edu Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard References: <200007052039.e65KdGC08927@daffy.ee.lbl.gov> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Vern Paxson wrote: > > Reminder, the last call closes this Friday, July 7th. So far there haven't > been any comments. If you've already reviewed it and are satisfied with it, > please send me a note to that effect (or post to the list), so I can gauge > the degree of support for the document. What is under last call- draft-balakirshnan draft-ietf-ecm-cm The latter appeared only today; given the differences, if it is the latter that is proceeding in the WG, perhaps it would be possible to extend the last call a week? Joe From owner-ecm@wyvern.aciri.org Fri Jul 7 15:59:47 2000 Received: from wyvern.aciri.org ([192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11695 for ; Fri, 7 Jul 2000 15:59:46 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id MAA81479 for ecm-outgoing; Fri, 7 Jul 2000 12:59:11 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id MAA81474 for ; Fri, 7 Jul 2000 12:59:10 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id PAA19750; Fri, 7 Jul 2000 15:59:05 -0400 Message-Id: <200007071959.PAA19750@breeze.lcs.mit.edu> To: Joe Touch cc: ecm@aciri.org Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard In-Reply-To: Message from Joe Touch of "Fri, 07 Jul 2000 10:41:26 PDT." <39661646.71981A41@isi.edu> Date: Fri, 07 Jul 2000 15:59:05 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk > > What is under last call- > > draft-balakirshnan > > draft-ietf-ecm-cm The latter. Hari > > The latter appeared only today; given the differences, if it is the > latter that is proceeding in the WG, perhaps it would be possible to > extend the last call a week? > > Joe From owner-ecm@wyvern.aciri.org Mon Jul 10 00:03:06 2000 Received: from wyvern.aciri.org ([128.32.201.124]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA16077 for ; Mon, 10 Jul 2000 00:03:06 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id BAA93489 for ecm-outgoing; Sun, 9 Jul 2000 01:22:46 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id BAA93484 for ; Sun, 9 Jul 2000 01:22:45 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e698MiX23751; Sun, 9 Jul 2000 01:22:44 -0700 (PDT) Message-Id: <200007090822.e698MiX23751@daffy.ee.lbl.gov> To: Joe Touch Cc: ecm@aciri.org Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard In-reply-to: Your message of Fri, 07 Jul 2000 10:41:26 PDT. Date: Sun, 09 Jul 2000 01:22:44 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk > > Reminder, the last call closes this Friday, July 7th. So far there haven't > > been any comments. If you've already reviewed it and are satisfied with it, > > please send me a note to that effect (or post to the list), so I can gauge > > the degree of support for the document. > > What is under last call- > > draft-balakirshnan > > draft-ietf-ecm-cm It's now the second of these (as Hari noted). But the second is just a revision of the first, renamed to reflect the fact that it's a WG document and not an individual submission. I asked Hari to do this when submitting a revision incorporating the earlier comments. > The latter appeared only today; given the differences, if it is the > latter that is proceeding in the WG, perhaps it would be possible to > extend the last call a week? Extending a week runs into the pre-Pittsburgh I-D IETF deadline. I think we can extend it until say Tuesday at noon PDT, though, seeing as how currently no further revisions are needed. Also, a reminder: as noted above, please send me a note if you've reviewed it and are satisfied with it, I've heard very little back on this. Vern From owner-ecm@wyvern.aciri.org Tue Jul 11 14:52:16 2000 Received: from wyvern.aciri.org ([128.32.201.124]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15611 for ; Tue, 11 Jul 2000 14:52:14 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id LAA14791 for ecm-outgoing; Tue, 11 Jul 2000 11:49:14 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id LAA14786 for ; Tue, 11 Jul 2000 11:49:13 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id LAA12756; Tue, 11 Jul 2000 11:49:12 -0700 (PDT) Message-ID: <396B6C15.50A9F970@isi.edu> Date: Tue, 11 Jul 2000 11:48:53 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Vern Paxson CC: ecm@aciri.org Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard References: <200007090822.e698MiX23751@daffy.ee.lbl.gov> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Some brief comments before last call closes: Sec 4.2 ALF != dynamic content although ALF can be dynamic and can affect content, there are other application dynamic content responses that aren't ALF-based Is it absolutely necessary that each MTU require a separate CM call or callback? This could severely affect performance. cm_request should be able to pass a requested number of bytes, either an upper bound or an estimate, e.g., for cases where the application is NOT sending an MTU. E.g - for realtime apps, it would prevent waiting for permission to send a full MTU when only a few bytes are being sent. (preferably, this could be cm_request(K,size) where K is the number of packets to be sent and size is the max size of each packets, again to allow amortization. the downcall (cm_data?) appears to be referred to but not defined in the discusion of synchronous calls As suggested at the BOF earlier, all reasonable parameters should include variances. This includes cmapp_update's rate value. Just as RTT includes a deviation, rate should be an upper bound or average with some variance as well. All variance information should have a way to control and read the averaging interval, (and possibly to get a token indicating the averaging algorithm). Some parameters of the algorithms (decay rate, averaging window) may need to be in the interface in the future. 4.3 - application notification the use of CM_PERSISTENT and CM_TRANSIENT requires the application to infer, rather than report. it would be safer to report explicit issues: timeout_loss sequence_loss reported_loss these should be a bit-vector, because there could be combinations of some kinds of loss (e.g., reported sequence loss from the other end, etc.) cm_notify should have a MUST to call cm_notify(stream,0) if the app has no immediate data to send. that would make it consistent with the subsequent claim that the controller be robust and fix apps that don't report it correctly (i.e., when the app has an error). I.e., if it's just a should, why is the CM doing cleanup? should the CM do cleanup at all? (It doesn't matter which - but the two appear inconsistent) 4.5 get/set macroflow these routines lack an interface to set sharing policy. the entire mechanism underspecifies this property, and currently assumes that streams are shared 1/N. that is only one choice in the second paragraph, destination = destination IP address Security considerations there are quite a few issues to consider, many of which were discussed in Touch97. in particular, the CM opens attacks to siblings of a vulnerable stream on a single machine, or (if macroflows span sets of hosts) across machines. I would very much like to see some of these issues addressed before this protocol goes to proposed standard. Joe From owner-ecm@wyvern.aciri.org Wed Jul 12 16:37:22 2000 Received: from wyvern.aciri.org ([128.32.201.124]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13262 for ; Wed, 12 Jul 2000 16:37:20 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id NAA24064 for ecm-outgoing; Wed, 12 Jul 2000 13:34:52 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id NAA24059 for ; Wed, 12 Jul 2000 13:34:51 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id NAA14669; Wed, 12 Jul 2000 13:34:49 -0700 (PDT) Message-ID: <396CD65B.A92C4F81@isi.edu> Date: Wed, 12 Jul 2000 13:34:35 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Vern Paxson CC: Hari Balakrishnan , ecm@aciri.org, sseshan@us.ibm.com, touch@isi.edu Subject: Re: cm_notify() semantics [was Re: comments on draft-balakrishnan-cm-02.txt] References: <200007060616.e666Gih10529@daffy.ee.lbl.gov> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Vern Paxson wrote: > > > I would reckon something like "if a stream hasn't used a grant in one RTT, > > pretend it's been used up. > > That sounds like a good approach to me. > > Vern Define 'used up' - does that mean that the stream has used the grant, that the grant returns to the pool for someone else, or that the grant just disappears and does not return to the pool? (they might have different effects on the CM's implementation of windowing) Joe From owner-ecm@wyvern.aciri.org Wed Jul 12 20:06:24 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18643 for ; Wed, 12 Jul 2000 20:06:23 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id RAA01072 for ecm-outgoing; Wed, 12 Jul 2000 17:05:45 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id RAA01067 for ; Wed, 12 Jul 2000 17:05:44 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id UAA15352; Wed, 12 Jul 2000 20:05:41 -0400 Message-Id: <200007130005.UAA15352@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 To: Joe Touch cc: ecm@aciri.org, sseshan@us.ibm.com Subject: Re: cm_notify() semantics [was Re: comments on draft-balakrishnan-cm-02.txt] In-Reply-To: Message from Joe Touch of "Wed, 12 Jul 2000 13:34:35 PDT." <396CD65B.A92C4F81@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 12 Jul 2000 20:05:41 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk > > > Vern Paxson wrote: > > > > > I would reckon something like "if a stream hasn't used a grant in one RTT, > > > pretend it's been used up. > > > > That sounds like a good approach to me. > > > > Vern > > Define 'used up' - does that mean that the stream has used the grant, > that the grant returns to the pool for someone else, or that the > grant just disappears and does not return to the pool? > > (they might have different effects on the CM's implementation of > windowing) > > Joe Joe, "used up" means that the stream has sent some data in response to the cmapp_send() grant. If a stream does not send a packet out within a certain time of receiving the grant from the CM, it returns to the macroflow pool for some other stream to use. The time constraint of one RTT is too low in general especially over low-RTT flows, as a couple of people have pointed out in private email. I think the right approach is for this to be something like max(RTT, 100ms). Hari From owner-ecm@wyvern.aciri.org Wed Jul 12 20:27:26 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18840 for ; Wed, 12 Jul 2000 20:27:25 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id RAA01217 for ecm-outgoing; Wed, 12 Jul 2000 17:27:15 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id RAA01212 for ; Wed, 12 Jul 2000 17:27:14 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id UAA15428 for ; Wed, 12 Jul 2000 20:27:14 -0400 Message-Id: <200007130027.UAA15428@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 X-url: http://nms.lcs.mit.edu/~hari/ From: Hari Balakrishnan Reply-to: Hari Balakrishnan To: ecm@aciri.org Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 12 Jul 2000 20:27:14 -0400 Sender: owner-ecm@aciri.org Precedence: bulk Joe, Thanks for these detailed comments---appreciate it! On Tue, 11 Jul 2000 11:48:53 PDT, you wrote: > Some brief comments before last call closes: > > Sec 4.2 > ALF != dynamic content > although ALF can be dynamic and can affect content, > there are other application dynamic content responses > that aren't ALF-based Yes, that's true; we certainly didn't mean to imply that. I've modified the text to make this clear. > Is it absolutely necessary that each MTU require a separate > CM call or callback? This could severely affect performance. > > cm_request should be able to pass a requested number of bytes, > either an upper bound or an estimate, e.g., for cases where > the application is NOT sending an MTU. E.g - for realtime apps, > it would prevent waiting for permission to send a full MTU > when only a few bytes are being sent. (preferably, this could be > cm_request(K,size) where K is the number of packets to be sent > and size is the max size of each packets, again to allow amortization. This has been a topic of several debates. In fact, the original design had K as a parameter, but various people suggested a simpler interface. In fact, the original proposal was of the form: cm_request(max_want, min_usable) There are a few tradeoffs here that need to be weighed. Perhaps these will convince you about the choice we made, or at least see where we're coming from! 1) If the concern is purely overhead, then the addition of K to the core CM API may not be the best way to go. (This document describes the API that the CM MUST implement to do the job of congestion control, and at the same time enabling application adaptation.) If a particular OS has high user/kernel crossing cost, it may provide the facility that enqueues multiple cm_request() calls in sequence and issues them to the CM (within the kernel) one at a time. The primary reason that the older design had a byte count was not overhead but application-related. The fear was that an application could not make a decision with cmapp_send() callback for a single MTU. For example, it may have needed information that it could send e.g., 4 packets during this RTT to make an application adaptation decision. In the end, it was felt (based on comments received from people who've built such apps and our own experience) that this longer-time-scale adaptation was better handled through the query and rate-based callback API. 2) Not having an explicit (potentially large) size to schedule makes the implementation of CM internals simpler. A lot simpler, in fact. 3) Notice that the cmapp_send() callback still occurs individually for each packet transmission, to avoid burst transmissions. Because of this, the number of user/kernel crossings or overhead is not reduced by more than a factor of 2 (not K). On the other hand, doing a callback with a passed parameter (of the number of packets to send) results in CM giving up any possibility of shaping the output traffic (i.e., the app may burst packets into the network). > the downcall (cm_data?) appears to be referred to but not defined in > the discusion of synchronous calls This paragraph in the draft may be misleading. The CM is not on the data path of such transmissions; there's no cm_data() or equivalent call. Transmissions occur using udp_output(), for example. I've added a note to this effect in the draft. > As suggested at the BOF earlier, all reasonable parameters should > include variances. > This includes cmapp_update's rate value. Just as RTT includes a > deviation, rate should be an > upper bound or average with some variance as well. All reports from the CM are instantaneous values. (I've added a note clarifying this.) Any calculation of averages or variances should be done by applications themselves. The reason for this is that it is hard to know a priori what the averaging mechanism should be that makes sense for each application. For example, rate-averaging can be done in several ways, with different applications preferring different ones. It is therefore better, a la the end-to-end argument, to simply report raw instantaneous values, empowering applications (e.g., TCP) to treat these samples any way they choose. > All variance information should have a way to control and read the > averaging interval, > (and possibly to get a token indicating the averaging algorithm). Some > parameters of > the algorithms (decay rate, averaging window) may need to be in the > interface in the future. Joe, this seems somewhat more complicated than desirable, and it is unclear that we'll get it all correct. The interface we've chosen allows any application to do all of this quite trivially. Do you agree? (I do agree that specific apps might want to do all this, and they should if it makes sense for them.) > 4.3 - application notification > > the use of CM_PERSISTENT and CM_TRANSIENT requires the application to > infer, > rather than report. > > it would be safer to report explicit issues: > > timeout_loss > > sequence_loss > > reported_loss > > these should be a bit-vector, because there could be combinations of > some kinds > of loss (e.g., reported sequence loss from the other end, etc.) Terms such as reported, sequence and timeout seem to be TCP-specific. We have used PERSISTENT and TRANSIENT to merely represent the concept of persistent heavy congestion loss/loss of ack clock/timer driven event/etc. vs. transient data loss etc. In addition, CM really requires applications to provide feedback on time periods of a RTT. We do agree that it may make sense to report a bit-vector, and I will change the document to reflect that. > cm_notify should have a MUST to call cm_notify(stream,0) if the app has > no immediate > data to send. that would make it consistent with the subsequent claim > that the controller > be robust and fix apps that don't report it correctly (i.e., when the > app has an error). > I.e., if it's just a should, why is the CM doing cleanup? should the CM > do cleanup at all? > (It doesn't matter which - but the two appear inconsistent) Absolutely. There are a couple of other reasons to do a "MUST," having to do with how long a cmapp_send() grant is valid for. This has been fixed now. However, the CM MUST still have support for cleanup to support applications that crash before cm_notify or are written incorrectly and do not do cm_notify(0). > 4.5 get/set macroflow > these routines lack an interface to set sharing policy. the entire > mechanism underspecifies > this property, and currently assumes that streams are shared 1/N. that > is only one choice The objective of this interface was only to set up sharing of groups not sharing policy. The scheduler must provide the interface to set sharing policy within a macroflow. Because we want to support many different schedulers (each of which may need different information to set policy), we chose not to specify an API to the scheduler. In a later guideline document, we plan to describe a few simple schedulers (at least a hierarchical scheduler and a weigted round robin) and the API they export to provide relative prioritization. > in the second paragraph, destination = destination IP address Yes, fixed now. > Security considerations > > there are quite a few issues to consider, many of which were discussed > in Touch97. > in particular, the CM opens attacks to siblings of a vulnerable stream > on a single machine, > or (if macroflows span sets of hosts) across machines. Agreed; added to the section. > I would very much like to see some of these issues addressed > before this protocol goes to proposed standard. > > Joe Hari & Srini From owner-ecm@wyvern.aciri.org Wed Jul 12 21:04:41 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19531 for ; Wed, 12 Jul 2000 21:04:40 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id SAA01424 for ecm-outgoing; Wed, 12 Jul 2000 18:04:28 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id SAA01419 for ; Wed, 12 Jul 2000 18:04:28 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id SAA20710; Wed, 12 Jul 2000 18:04:26 -0700 (PDT) Message-ID: <396D158B.CDE3F723@isi.edu> Date: Wed, 12 Jul 2000 18:04:11 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, sseshan@us.ibm.com, touch@isi.edu Subject: Re: cm_notify() semantics [was Re: comments on draft-balakrishnan-cm-02.txt] References: <200007130005.UAA15352@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > > > > > > Vern Paxson wrote: > > > > > > > I would reckon something like "if a stream hasn't used a grant in one RTT, > > > > pretend it's been used up. > > > > > > That sounds like a good approach to me. > > > > > > Vern > > > > Define 'used up' - does that mean that the stream has used the grant, > > that the grant returns to the pool for someone else, or that the > > grant just disappears and does not return to the pool? > > > > (they might have different effects on the CM's implementation of > > windowing) > > > > Joe > > Joe, > > "used up" means that the stream has sent some data in response to the > cmapp_send() grant. If a stream does not send a packet out within a certain > time of receiving the grant from the CM, it returns to the macroflow pool for > some other stream to use. If the stream hasn't 'sent some data', the pretend it has? That doesn't make sense - it seems like - grants should have expiration times the implication of expiry is that - the application MUST NOT send data based on that grant after time has expired - the application SHOULD return the grant if it is not going to use it, before the time has expired Once time has expired, the CM can issue a local "return" - reusing the permission locally. > The time constraint of one RTT is too low in general especially over low-RTT > flows, as a couple of people have pointed out in private email. I think the > right approach is for this to be something like max(RTT, 100ms). So the cmapp_send() grant would include an indication of 100ms as the upperbound of the grant. Joe From owner-ecm@wyvern.aciri.org Wed Jul 12 22:39:08 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA22749 for ; Wed, 12 Jul 2000 22:39:07 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id TAA01952 for ecm-outgoing; Wed, 12 Jul 2000 19:38:39 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id TAA01942 for ; Wed, 12 Jul 2000 19:38:38 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id WAA15957; Wed, 12 Jul 2000 22:38:36 -0400 Message-Id: <200007130238.WAA15957@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 To: Joe Touch cc: ecm@aciri.org, sseshan@us.ibm.com Subject: Re: cm_notify() semantics In-Reply-To: Message from Joe Touch of "Wed, 12 Jul 2000 18:04:11 PDT." <396D158B.CDE3F723@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 12 Jul 2000 22:38:36 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk > > > Define 'used up' - does that mean that the stream has used the grant, > > > that the grant returns to the pool for someone else, or that the > > > grant just disappears and does not return to the pool? > > > > > > (they might have different effects on the CM's implementation of > > > windowing) > > > > > > Joe > > > > Joe, > > > > "used up" means that the stream has sent some data in response to the > > cmapp_send() grant. If a stream does not send a packet out within a certain > > time of receiving the grant from the CM, it returns to the macroflow pool for > > some other stream to use. > > If the stream hasn't 'sent some data', the pretend it has? > That doesn't make sense - it seems like > > - grants should have expiration times > the implication of expiry is that > - the application MUST NOT send data based on that > grant after time has expired > - the application SHOULD return the grant if it is > not going to use it, before the time has expired > > Once time has expired, the CM can issue a local "return" - reusing the > permission locally. Joe, You're right. I was a little careless in one of my previous messages. The following rules must hold: - At most one grant outstanding per stream at any time. - Each grant have expiration time = max(RTT, CM_MIN_GRANT) where CM_MIN_GRANT is an implementation-dependent value (e.g., 100ms) - If expired, application MUST NOT send data without another explicit cm_request() - The application SHOULD call cm_notify(stream, 0) if said grant is not going to be used within expiration time. Let me know if you disagree. Should the cmapp_send() API include an explicit time for which the grant is valid? That's what Joe seems to be suggesting, and it seems like a good idea to me. Thoughts? Hari > > The time constraint of one RTT is too low in general especially over low-RTT > > flows, as a couple of people have pointed out in private email. I think the > > right approach is for this to be something like max(RTT, 100ms). > > So the cmapp_send() grant would include an indication of 100ms as the > upperbound > of the grant. > > Joe From owner-ecm@wyvern.aciri.org Wed Jul 12 23:33:59 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA24010 for ; Wed, 12 Jul 2000 23:33:59 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id UAA02221 for ecm-outgoing; Wed, 12 Jul 2000 20:33:47 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id UAA02216 for ; Wed, 12 Jul 2000 20:33:46 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (ink-i.isi.edu [128.9.102.4]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id UAA13822; Wed, 12 Jul 2000 20:33:44 -0700 (PDT) Message-ID: <396D36A3.1FED5E13@isi.edu> Date: Wed, 12 Jul 2000 20:25:23 -0700 From: Joe Touch X-Mailer: Mozilla 4.7 [en] (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, sseshan@us.ibm.com, touch@isi.edu Subject: Re: cm_notify() semantics References: <200007130238.WAA15957@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > The following rules must hold: > > - At most one grant outstanding per stream at any time. > - Each grant have expiration time = max(RTT, CM_MIN_GRANT) where > CM_MIN_GRANT is an implementation-dependent value (e.g., 100ms) > - If expired, application MUST NOT send data without another explicit > cm_request() > - The application SHOULD call cm_notify(stream, 0) if said grant is not > going to be used within expiration time. and - the CM MAY (SHOULD? / MUST?) act as if cm_notify(stream, 0) has been called at the end of the expiration time (I would say "SHOULD") > Should the cmapp_send() API include an explicit time for which the grant is > valid? That's what Joe seems to be suggesting, and it seems like a good idea > to me. (yes, that's what I am suggesting - also, it could be application-configurable, or bounded or set by the CM) Joe From owner-ecm@wyvern.aciri.org Thu Jul 13 13:15:40 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00076 for ; Thu, 13 Jul 2000 13:15:39 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id KAA07577 for ecm-outgoing; Thu, 13 Jul 2000 10:14:55 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id KAA07572 for ; Thu, 13 Jul 2000 10:14:54 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id KAA28429; Thu, 13 Jul 2000 10:14:52 -0700 (PDT) Message-ID: <396DF8FC.F26F04BC@isi.edu> Date: Thu, 13 Jul 2000 10:14:36 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, touch@isi.edu Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard References: <200007130027.UAA15428@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > > Is it absolutely necessary that each MTU require a separate > > CM call or callback? This could severely affect performance. > > > > cm_request should be able to pass a requested number of bytes, > > either an upper bound or an estimate, e.g., for cases where > > the application is NOT sending an MTU. E.g - for realtime apps, > > it would prevent waiting for permission to send a full MTU > > when only a few bytes are being sent. (preferably, this could be > > cm_request(K,size) where K is the number of packets to be sent > > and size is the max size of each packets, again to allow amortization. > > This has been a topic of several debates. In fact, the original design had K > as a parameter, but various people suggested a simpler interface. In fact, the > original proposal was of the form: > cm_request(max_want, min_usable) > > There are a few tradeoffs here that need to be weighed. Perhaps these will > convince you about the choice we made, or at least see where we're coming from! > > 1) If the concern is purely overhead, then the addition of K to the core > CM API may not be the best way to go. (This document describes the API that > the CM MUST implement to do the job of congestion control, and at the same time > enabling application adaptation.) If a particular OS has high user/kernel > crossing cost, it may provide the facility that enqueues multiple cm_request() > calls in sequence and issues them to the CM (within the kernel) one at a time. That still requires procedure calls per write/read, even if inside the OS. If the application has permission to send at line rate or can pace them out automatically (via control of a DMA), the CM should get out of the way. > an application adaptation decision. In the end, it was felt (based on comments > received from people who've built such apps and our own experience) that this > longer-time-scale adaptation was better handled through the query and > rate-based callback API. Rate-based callbacks don't work well with DMA engines. The DMA setup can be prohibitive if done on a per packet basis, esp. if the packets are small and the data rate is high. > 2) Not having an explicit (potentially large) size to schedule makes the > implementation of CM internals simpler. A lot simpler, in fact. Agreed. But the CM should (MUST?:-) not prohibit, by its design requirements, high-performance networking. > 3) Notice that the cmapp_send() callback still occurs individually for > each packet transmission, to avoid burst transmissions. Because of this, the > number of user/kernel crossings or overhead is not reduced by more than a > factor of 2 (not K). On the other hand, doing a callback with a passed > parameter (of the number of packets to send) results in CM giving up any > possibility of shaping the output traffic (i.e., the app may burst packets into > the network). The app can do this anyway. The CM trusts the application to send only one packet per cmapp_send() callback, it can equivalently trust it to pace, if the application asserts that it both can and will. Perhaps this is a MAY: the CM MUST support cm_request(K,size); sizes of K=1 MUST be supported, sizes of K>1 MAY be supported. cmapp_send(K, size) MUST be supported; the application MUST NOT send more than K packets of size if K>1, the application MUST have a pre-negotiated rate with the CM, and the application MUST pace the K packets via that rate (this allows simple applications to set K=1 and be done with it, but doesn't prohibit high-performance variants). Having the CM spec assert a callback-per-MTU is unduly imposing an implementation detail which has substantial performance implications. > > As suggested at the BOF earlier, all reasonable parameters should > > include variances. > > This includes cmapp_update's rate value. Just as RTT includes a > > deviation, rate should be an > > upper bound or average with some variance as well. > > All reports from the CM are instantaneous values. RTT already has an average and STDDEV. This indicates that the CM is already doing averaging. The reported rate is no different - and it's important because the application may want to know the rate of the aggregate. > > All variance information should have a way to control and read the > > averaging interval, > > (and possibly to get a token indicating the averaging algorithm). Some > > parameters of > > the algorithms (decay rate, averaging window) may need to be in the > > interface in the future. > > Joe, this seems somewhat more complicated than desirable, and it is unclear > that we'll get it all correct. The interface we've chosen allows any > application to do all of this quite trivially. Do you agree? Applications would need to poll the CM interface, increasing load. The same way RTT is averaged and reported back, rate should be as well. It's not complicated - it's orthogonal. The CM should just report the information it already has. I agree that if the application doesn't like the averaging algorithm or interval it may want to change them; the CM MAY implement an interface to provide that facility :-) > > 4.3 - application notification > > > > the use of CM_PERSISTENT and CM_TRANSIENT requires the application to > > infer, > > rather than report. > > > > it would be safer to report explicit issues: > > > > timeout_loss > > > > sequence_loss > > > > reported_loss > > > > these should be a bit-vector, because there could be combinations of > > some kinds > > of loss (e.g., reported sequence loss from the other end, etc.) > > Terms such as reported, sequence and timeout seem to be TCP-specific. We have > used PERSISTENT and TRANSIENT to merely represent the concept of persistent > heavy congestion loss/loss of ack clock/timer driven event/etc. vs. transient > data loss etc. In addition, CM really requires applications to provide > feedback on time periods of a RTT. > > We do agree that it may make sense to report a bit-vector, and I will change > the document to reflect that. The problem is that there is the possibility of different flows (TCP, UDP, etc.) being aggregated. The meaning of these terms must be unambiguous and absolute, not relative to the protocol. The bit-vector should indicate: timeout (without loss) timeout (inferred as loss) loss inferred from the receive stream (e.g., sequence numbers) loss explicitly reported by third parties (ECN) loss explicitly reported by the endsystem (SACK?) Sequence numbers and reported errors (NACKS) are not TCP specific. > > cm_notify should have a MUST to call cm_notify(stream,0) if the app has > > no immediate > > data to send. that would make it consistent with the subsequent claim > > that the controller > > be robust and fix apps that don't report it correctly (i.e., when the > > app has an error). > > I.e., if it's just a should, why is the CM doing cleanup? should the > CM > > do cleanup at all? > > (It doesn't matter which - but the two appear inconsistent) > > Absolutely. There are a couple of other reasons to do a "MUST," having to do > with how long a cmapp_send() grant is valid for. This has been fixed now. > > However, the CM MUST still have support for cleanup to support applications > that crash before cm_notify or are written incorrectly and do not do > cm_notify(0). I was wondering whether this was the CM's responsibility, or the wording of the APP requires it (or the sandbox in which it runs, e.g., the OS) to notice the failure and cleaup after itself. If it is a MUST, then the app MUST find a way to do it and the CM need not deal with cleanup; if it is a SHOULD, then you MUST implement a cleanup in the CM. The difference is which sides have MUSTs and SHOULDs. > > 4.5 get/set macroflow > > these routines lack an interface to set sharing policy. the entire > > mechanism underspecifies > > this property, and currently assumes that streams are shared 1/N. that > > is only one choice > > The objective of this interface was only to set up sharing of groups not > sharing policy. The scheduler must provide the interface to set sharing policy > within a macroflow. Because we want to support many different schedulers (each > of which may need different information to set policy), we chose not to specify > an API to the scheduler. In a later guideline document, we plan to describe a > few simple schedulers (at least a hierarchical scheduler and a weigted round > robin) and the API they export to provide relative prioritization. AOK - there should be a description of that, i.e., 'what's NOT in this document, but still needs to be deterimined'. Joe From owner-ecm@wyvern.aciri.org Thu Jul 13 13:19:57 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00348 for ; Thu, 13 Jul 2000 13:19:56 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id KAA07636 for ecm-outgoing; Thu, 13 Jul 2000 10:19:51 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id KAA07631 for ; Thu, 13 Jul 2000 10:19:50 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (sci.isi.edu [128.9.160.93]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id KAA29350; Thu, 13 Jul 2000 10:19:49 -0700 (PDT) Message-ID: <396DFA25.AA8B8AD@isi.edu> Date: Thu, 13 Jul 2000 10:19:33 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en] (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, touch@isi.edu Subject: Re: ECM last call for draft-balakrishnan-cm-02.txt to Proposed Standard References: <200007130027.UAA15428@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit On a separate note, I've been suggesting that RFCs include a section, akin to security considerations, called something like "Impact on Internet Standards". Sort of like the EPA's 'Environmental Impact", this would list any way in which: - the RFC violates any existing protocol (proposed or standard, not experimental or informational) - the RFC relies in a new way or very strongly on an existing protocol - the RFC makes additional requirements on an existing protocol (these are not precise; just placeholders for the idea 'how do you affect standards or would their change affect you'). In this document, e.g., there is an assumption in the definition of flows that a pair of IP addresses share flow properties on all connections between them. I.e., that IP addresses NOT be shared on different hosts with different path properties. (that's a looser version of an existing requirement, but still might be useful to flag). Joe From owner-ecm@wyvern.aciri.org Fri Jul 14 09:05:41 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA18997 for ; Fri, 14 Jul 2000 09:05:39 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id GAA13853 for ecm-outgoing; Fri, 14 Jul 2000 06:02:42 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id GAA13848 for ; Fri, 14 Jul 2000 06:02:40 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id JAA01699; Fri, 14 Jul 2000 09:02:33 -0400 Message-Id: <200007141302.JAA01699@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Hari Balakrishnan X-url: http://nms.lcs.mit.edu/~hari/ To: Joe Touch cc: ecm@aciri.org Subject: Re: cmapp_update averaging [was: something else...] In-reply-to: Your message of "Thu, 13 Jul 2000 10:14:36 PDT" <396DF8FC.F26F04BC@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 14 Jul 2000 09:02:33 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk Joe, Many of the other points you made make sense to me. But I have a problem with this one: On Thu, 13 Jul 2000 10:14:36 PDT, you wrote: > > > As suggested at the BOF earlier, all reasonable parameters should > > > include variances. > > > This includes cmapp_update's rate value. Just as RTT includes a > > > deviation, rate should be an > > > upper bound or average with some variance as well. > > > > All reports from the CM are instantaneous values. > > RTT already has an average and STDDEV. This indicates that the CM > is already doing averaging. The reported rate is no different - > and it's important because the application may want to know the > rate of the aggregate. Mea culpa. srtt and rttdev are obviously not instantaneous values. The rate reported is instantaneous, though, in the sense that it is an instantaneous rate calculated, for example, by taking the ratio of cwnd and srtt, and dividing by the fraction of that ratio allocated to the stream. > > > All variance information should have a way to control and read the > > > averaging interval, > > > (and possibly to get a token indicating the averaging algorithm). Some > > > parameters of > > > the algorithms (decay rate, averaging window) may need to be in the > > > interface in the future. > > > > Joe, this seems somewhat more complicated than desirable, and it is unclear > > that we'll get it all correct. The interface we've chosen allows any > > application to do all of this quite trivially. Do you agree? > > Applications would need to poll the CM interface, increasing load. The > same > way RTT is averaged and reported back, rate should be as well. > > It's not complicated - it's orthogonal. The CM should just report the > information it already has. > I agree that if the application doesn't like the averaging algorithm or > interval it may want to change them; > the CM MAY implement an interface to provide that facility :-) The issue is that the CM is in fact reporting all the information it has. It needs the srtt and rttdev because of the way it mimics TCP-style window-based congestion avoidance and control. It doesn't implement any rate averaging and therefore can't report it in any reasonable way. Like I said before, an application can average these instantaneous things in any manner it chooses. Hari From owner-ecm@wyvern.aciri.org Fri Jul 14 09:12:05 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA20372 for ; Fri, 14 Jul 2000 09:12:03 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id GAA13910 for ecm-outgoing; Fri, 14 Jul 2000 06:11:57 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id GAA13905 for ; Fri, 14 Jul 2000 06:11:55 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id JAA01726; Fri, 14 Jul 2000 09:11:54 -0400 Message-Id: <200007141311.JAA01726@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Hari Balakrishnan X-url: http://nms.lcs.mit.edu/~hari/ To: Joe Touch cc: ecm@aciri.org Subject: Re: application notification In-reply-to: Your message of "Thu, 13 Jul 2000 10:14:36 PDT" <396DF8FC.F26F04BC@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 14 Jul 2000 09:11:54 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk Joe, On Thu, 13 Jul 2000 10:14:36 PDT, you wrote: > > Terms such as reported, sequence and timeout seem to be TCP-specific. We have > > used PERSISTENT and TRANSIENT to merely represent the concept of persistent > > heavy congestion loss/loss of ack clock/timer driven event/etc. vs. transient > > data loss etc. In addition, CM really requires applications to provide > > feedback on time periods of a RTT. > > > > We do agree that it may make sense to report a bit-vector, and I will change > > the document to reflect that. > > The problem is that there is the possibility of different flows (TCP, > UDP, etc.) being aggregated. The meaning of these terms must be > unambiguous and absolute, not relative to the protocol. > > The bit-vector should indicate: > > timeout (without loss) > timeout (inferred as loss) > loss inferred from the receive stream (e.g., sequence numbers) > loss explicitly reported by third parties (ECN) > loss explicitly reported by the endsystem (SACK?) > > Sequence numbers and reported errors (NACKS) are not TCP specific. No, they aren't TCP-specific. But many UDP-based streams don't have a notion of a timeout, which is usually related to reliability. I believe that there are three fundamental types of congestion events that a feedback- and window-based protocol can pick up. - The loss of the feedback self-clock, e.g., when many packets are lost in quick succession, or when many feedback packets are lost, or if a retransmission is lost (e.g., in TCP). This signifies "persistent" congestion, which essentially means that the feedback clock is lost, so go back into slow start. This is generally not dependent on whether SACK is used or not. - The loss of some packets, but not the feedback clock. This signifies "transient" congestion, one that requires some decrease in window (rate) but doesn't necessitate slow start. - ECN. That was the intention of the three orthogonal modes of notification. The bit-vector is a good idea, especially if ECN and one of the other two modes occur at the same time. Do you agree? Hari From owner-ecm@wyvern.aciri.org Fri Jul 14 09:18:38 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA22140 for ; Fri, 14 Jul 2000 09:18:38 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id GAA13943 for ecm-outgoing; Fri, 14 Jul 2000 06:18:30 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from breeze.lcs.mit.edu (breeze.lcs.mit.edu [18.31.0.83]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id GAA13938 for ; Fri, 14 Jul 2000 06:18:28 -0700 (PDT) (envelope-from hari@breeze.lcs.mit.edu) Received: from breeze.lcs.mit.edu (IDENT:hari@localhost.localdomain [127.0.0.1]) by breeze.lcs.mit.edu (8.9.3/8.9.3) with ESMTP id JAA01783; Fri, 14 Jul 2000 09:18:27 -0400 Message-Id: <200007141318.JAA01783@breeze.lcs.mit.edu> X-Mailer: exmh version 2.1.1 10/15/1999 Reply-To: Hari Balakrishnan X-url: http://nms.lcs.mit.edu/~hari/ To: Joe Touch cc: ecm@aciri.org Subject: cm_notify and cleanups In-reply-to: Your message of "Thu, 13 Jul 2000 10:14:36 PDT" <396DF8FC.F26F04BC@isi.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 14 Jul 2000 09:18:27 -0400 From: Hari Balakrishnan Sender: owner-ecm@aciri.org Precedence: bulk On Thu, 13 Jul 2000 10:14:36 PDT, you wrote: > > However, the CM MUST still have support for cleanup to support applications > > that crash before cm_notify or are written incorrectly and do not do > > cm_notify(0). > > I was wondering whether this was the CM's responsibility, or the wording > of the APP requires it (or the sandbox in which it runs, e.g., the OS) > to notice the failure and cleaup after itself. If it is a MUST, then the > app MUST find a way to do it and the CM need not deal with cleanup; if > it is a SHOULD, then you MUST implement a cleanup in the CM. The > difference is which sides have MUSTs and SHOULDs. Joe, I think it should be the CM's responsibility, as it is the best defensive behavior and is pretty straightforward to implement. Furthermore, note that there may not be an OS sandbox conveniently available, e.g., if a stream runs over vanilla UDP. Unless you're willing to make modifications to vanilla UDP to make it CM-aware and also maintain more stream state, which I don't think is a good idea. Hari From owner-ecm@wyvern.aciri.org Fri Jul 14 15:51:17 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28235 for ; Fri, 14 Jul 2000 15:51:16 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id MAA16733 for ecm-outgoing; Fri, 14 Jul 2000 12:50:33 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id MAA16728 for ; Fri, 14 Jul 2000 12:50:33 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (ras02.isi.edu [128.9.176.102]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id MAA01183; Fri, 14 Jul 2000 12:50:30 -0700 (PDT) Message-ID: <396F6C25.33632E1B@isi.edu> Date: Fri, 14 Jul 2000 12:38:13 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, touch@isi.edu Subject: Re: cmapp_update averaging [was: something else...] References: <200007141302.JAA01699@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > Joe, > > Many of the other points you made make sense to me. But I have a problem with > this one: > > On Thu, 13 Jul 2000 10:14:36 PDT, you wrote: > > > > > As suggested at the BOF earlier, all reasonable parameters should > > > > include variances. > > > > This includes cmapp_update's rate value. Just as RTT includes a > > > > deviation, rate should be an > > > > upper bound or average with some variance as well. > > > > > > All reports from the CM are instantaneous values. > > > > RTT already has an average and STDDEV. This indicates that the CM > > is already doing averaging. The reported rate is no different - > > and it's important because the application may want to know the > > rate of the aggregate. > > Mea culpa. srtt and rttdev are obviously not instantaneous values. The rate > reported is instantaneous, though, in the sense that it is an instantaneous > rate calculated, for example, by taking the ratio of cwnd and srtt, and > dividing by the fraction of that ratio allocated to the stream. Putting a windowed value (RTT) into a calculation makes the result of that calculation implicitly windowed. The rate is thus "over the window of the RTT". It is probably sufficient to point that out, in that case. (rather than requiring some of the extra interface I mentioned earlier). Joe From owner-ecm@wyvern.aciri.org Fri Jul 14 15:51:18 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28234 for ; Fri, 14 Jul 2000 15:51:16 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id MAA16740 for ecm-outgoing; Fri, 14 Jul 2000 12:50:36 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id MAA16735 for ; Fri, 14 Jul 2000 12:50:35 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (ras02.isi.edu [128.9.176.102]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id MAA01191; Fri, 14 Jul 2000 12:50:32 -0700 (PDT) Message-ID: <396F6DB9.64D8B45F@isi.edu> Date: Fri, 14 Jul 2000 12:44:57 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, touch@isi.edu Subject: Re: application notification References: <200007141311.JAA01726@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > > The bit-vector should indicate: > > > > timeout (without loss) > > timeout (inferred as loss) > > loss inferred from the receive stream (e.g., sequence numbers) > > loss explicitly reported by third parties (ECN) > > loss explicitly reported by the endsystem (SACK?) > > > > Sequence numbers and reported errors (NACKS) are not TCP specific. > > No, they aren't TCP-specific. But many UDP-based streams don't have a notion > of a timeout, which is usually related to reliability. I believe that there > are three fundamental types of congestion events that a feedback- and > window-based protocol can pick up. > > - The loss of the feedback self-clock, Then say this. > e.g., when many packets are lost in > quick succession, or when many feedback packets are lost, or if a > retransmission is lost (e.g., in TCP). This signifies "persistent" congestion, > which essentially means that the feedback clock is lost, so go back into slow > start. This is generally not dependent on whether SACK is used or not. Persistent means different things; it could mean 'large losses over a RTT', 'many losses in a row', etc. Losing a retransmission is not 'persistent' in any case, unless it occurs many times. > - The loss of some packets, but not the feedback clock. This signifies > "transient" congestion, one that requires some decrease in window (rate) but > doesn't necessitate slow start. > > - ECN. > > That was the intention of the three orthogonal modes of notification. The > bit-vector is a good idea, especially if ECN and one of the other two modes > occur at the same time. > > Do you agree? Basically, except that I would like to see a) the names be well-defined, e.g., persistent = more than X% packets lost over 1 RTT, or more than Y% lost over 2 RTTs.. transient = all other loss patterns b) these are orthogonal to whether the loss is generated by a timeout, noticing gaps in a sequence, or explicit feedback, so it would be useful to say: TYPE_OF_LOSS as one of these three I agree that some protocols will never generate some patterns, but the point is to get away from TCP-specifics and express the underlying reasons, IMO...:-) Joe From owner-ecm@wyvern.aciri.org Fri Jul 14 16:19:37 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28233 for ; Fri, 14 Jul 2000 15:51:16 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id MAA16747 for ecm-outgoing; Fri, 14 Jul 2000 12:50:39 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id MAA16742 for ; Fri, 14 Jul 2000 12:50:38 -0700 (PDT) (envelope-from touch@ISI.EDU) Received: from isi.edu (ras02.isi.edu [128.9.176.102]) by boreas.isi.edu (8.9.3/8.9.3) with ESMTP id MAA01226; Fri, 14 Jul 2000 12:50:35 -0700 (PDT) Message-ID: <396F6DFE.DA9C2863@isi.edu> Date: Fri, 14 Jul 2000 12:46:06 -0700 From: Joe Touch X-Mailer: Mozilla 4.73 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en,pdf MIME-Version: 1.0 To: Hari Balakrishnan CC: ecm@aciri.org, touch@isi.edu Subject: Re: cm_notify and cleanups References: <200007141318.JAA01783@breeze.lcs.mit.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Hari Balakrishnan wrote: > > On Thu, 13 Jul 2000 10:14:36 PDT, you wrote: > > > > However, the CM MUST still have support for cleanup to support applications > > > that crash before cm_notify or are written incorrectly and do not do > > > cm_notify(0). > > > > I was wondering whether this was the CM's responsibility, or the wording > > of the APP requires it (or the sandbox in which it runs, e.g., the OS) > > to notice the failure and cleaup after itself. If it is a MUST, then the > > app MUST find a way to do it and the CM need not deal with cleanup; if > > it is a SHOULD, then you MUST implement a cleanup in the CM. The > > difference is which sides have MUSTs and SHOULDs. > > Joe, > > I think it should be the CM's responsibility, as it is the best > defensive behavior and is pretty straightforward to implement. > Furthermore, note that there may not be an OS sandbox conveniently > available, e.g., if a stream runs over vanilla UDP. Unless you're > willing to make modifications to vanilla UDP to make it CM-aware > and also maintain more stream state, which I don't think is a good idea. Agreed - the above wasn't proposing a different solution, it was requesting a clarification of the wording of MUST/SHOULD/MAY to indicate what you just said :-) Joe From owner-ecm@wyvern.aciri.org Tue Jul 18 15:57:40 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA17197 for ; Tue, 18 Jul 2000 15:57:39 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id MAA47408 for ecm-outgoing; Tue, 18 Jul 2000 12:55:42 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id MAA47403 for ; Tue, 18 Jul 2000 12:55:41 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e6IJtfR01411; Tue, 18 Jul 2000 12:55:41 -0700 (PDT) Message-Id: <200007181955.e6IJtfR01411@daffy.ee.lbl.gov> To: ecm@aciri.org Subject: plan for Pittsburgh meeting Cc: mankin@east.isi.edu Date: Tue, 18 Jul 2000 12:55:41 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk draft-ietf-ecm-cm-00.txt has completed last call (with comments from Joe Touch, to be addressed in -01). However, it's not clear to me that there's WG rough consensus for the document. When issuing the last call, I requested: > If you've already reviewed it and are satisfied with it, please send me a > note to that effect (or post to the list), so I can gauge the degree of > support for the document. and I received only one note. After asking around, it appears that there are a number of people who approve of the general notion, but haven't reviewed the document. Therefore, and in consultation with the ADs, I've asked Hari to give an extended overview of the document at the Pittsburgh meeting, part tutorial and part open issues. We'll then use the meeting to judge WG consensus as well as additional work that's needed, if any. If there's remaining time, I'll lead a brief discussion of the "ECM Scheduler" document that's also one of our deliverables. The meeting is currently scheduled for Monday afternoon, 1PM-3PM. Please take a bit of time between now and the meeting to review the document. Also, if you have other agenda items, or comments on the above, please let me know. Thanks, Vern From owner-ecm@wyvern.aciri.org Fri Jul 21 02:58:44 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA27350 for ; Fri, 21 Jul 2000 02:58:44 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id XAA65871 for ecm-outgoing; Thu, 20 Jul 2000 23:56:55 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id XAA65866 for ; Thu, 20 Jul 2000 23:56:54 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e6L6uoQ13087; Thu, 20 Jul 2000 23:56:50 -0700 (PDT) Message-Id: <200007210656.e6L6uoQ13087@daffy.ee.lbl.gov> To: agenda@ietf.org Subject: agenda for ECM working group Cc: ecm@aciri.org, mankin@east.isi.edu Date: Thu, 20 Jul 2000 23:56:50 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk 1. Agenda bashing (Vern Paxson, 5 min) 2. Status update (Vern Paxson, 5 min) 3. Overview/discussion of draft-ietf-ecm-cm-00.txt (Hari Balakrishnan, 90 min) 4. Getting the ECM scheduler document rolling (Vern Paxson, 20 min) From owner-ecm@wyvern.aciri.org Fri Jul 21 09:09:10 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA08942 for ; Fri, 21 Jul 2000 09:09:10 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id GAA67738 for ecm-outgoing; Fri, 21 Jul 2000 06:06:47 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from ietf.org (odin.ietf.org [132.151.1.176]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id GAA67733 for ; Fri, 21 Jul 2000 06:06:46 -0700 (PDT) (envelope-from nsyracus@cnri.reston.va.us) Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA07384; Fri, 21 Jul 2000 09:06:44 -0400 (EDT) Message-Id: <200007211306.JAA07384@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: ecm@aciri.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Subject: I-D ACTION:draft-ietf-ecm-cm-01.txt Date: Fri, 21 Jul 2000 09:06:43 -0400 Sender: owner-ecm@aciri.org Precedence: bulk --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Endpoint Congestion Management Working Group of the IETF. Title : The Congestion Manager Author(s) : H. Balakrishnan, S. Seshan Filename : draft-ietf-ecm-cm-01.txt Pages : Date : 20-Jul-00 This document describes the Congestion Manager (CM), an end-system module that (i) enables an ensemble of multiple concurrent streams from a sender destined to the same receiver and sharing the same congestion properties to perform proper congestion avoidance and control, and (ii) allows applications to easily adapt to network congestion. This CM framework integrates congestion management across all applications and transport protocols. The CM maintains congestion parameters (available aggregate and per-stream bandwidth, per-receiver round-trip times, etc.) and exports an API that enables applications to learn about network characteristics, pass information to the CM, share congestion information with each other, and schedule data transmissions. This document focuses on applications and transport protocols with their own independent per-byte or per-packet sequence number information, and does not require modifications to the receiver protocol stack. The receiving application must provide feedback to the sending application about received packets and losses, and the latter uses the CM API to update CM state. This document does not address networks with reservations or service discrimination. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ecm-cm-01.txt Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ecm-cm-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ecm-cm-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <20000720141321.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ecm-cm-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ecm-cm-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <20000720141321.I-D@ietf.org> --OtherAccess-- --NextPart-- From owner-ecm@wyvern.aciri.org Fri Jul 21 11:55:02 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09854 for ; Fri, 21 Jul 2000 11:55:01 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id IAA69193 for ecm-outgoing; Fri, 21 Jul 2000 08:53:56 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from daffy.ee.lbl.gov (daffy.ee.lbl.gov [131.243.1.31]) by wyvern.aciri.org (8.9.3/8.9.3) with ESMTP id IAA69188 for ; Fri, 21 Jul 2000 08:53:56 -0700 (PDT) (envelope-from vern@daffy.ee.lbl.gov) Received: (from vern@localhost) by daffy.ee.lbl.gov (8.10.0/8.10.0) id e6LFrp114341; Fri, 21 Jul 2000 08:53:51 -0700 (PDT) Message-Id: <200007211553.e6LFrp114341@daffy.ee.lbl.gov> To: agenda@ietf.org Cc: ecm@aciri.org, mankin@east.isi.edu Subject: slightly revised agenda for ECM working group Date: Fri, 21 Jul 2000 08:53:51 PDT From: Vern Paxson Sender: owner-ecm@aciri.org Precedence: bulk 1. Agenda bashing (Vern Paxson, 5 min) 2. Status update (Vern Paxson, 5 min) 3. Overview/discussion of draft-ietf-ecm-cm-01.txt (Hari Balakrishnan, 90 min) 4. Getting the ECM scheduler document rolling (Vern Paxson, 20 min) (the only change is draft-ietf-ecm-cm-00.txt is now -01.txt; it just popped out) From owner-ecm@wyvern.aciri.org Thu Jul 27 09:29:28 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA24033 for ; Thu, 27 Jul 2000 09:29:27 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id GAA10224 for ecm-outgoing; Thu, 27 Jul 2000 06:26:35 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from sina.com ([202.106.187.156]) by wyvern.aciri.org (8.9.3/8.9.3) with SMTP id GAA10219 for ; Thu, 27 Jul 2000 06:26:34 -0700 (PDT) (envelope-from bosunservice@sina.com) Message-Id: <200007271326.GAA10219@wyvern.aciri.org> Received: (qmail 84552 invoked from network); 27 Jul 2000 13:15:25 -0000 Received: from unknown (HELO localhost) (202.105.19.97) by 202.106.187.156 with SMTP; 27 Jul 2000 13:15:25 -0000 X-Sender: bosunservice@sina.com From: Jason Wong To: ecm@aciri.org Date: Thu, 27 Jul 2000 21:19:26 +0800 Subject: Hope to establish business relations -China Towels Supply Reply-To: bosunservice@sina.com MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-ecm@aciri.org Precedence: bulk Content-Transfer-Encoding: 7bit Dear Sir, We are pleased to have found your online offer and writing to you with the hope of establishing a business relations with you. We Nanhai Towel Co., Ltd.specializes in designing and manufacturing various kinds of towels. Our product capacity is about 1,800 tons towel and 1.2 million yards yarn-dyed fabric per year. We are one of the largest towel manufactured and exporter and covers a big part of the export of either kitchen towels or printing beach towels in South China. For near 50 years of towels making, the company has accumulated a wealth of experience and has a whole set of machine that can be used to weave, bleach, dye and print, most of these advanced equipment were introduced from abroad. Recently the company introduced Italian Vamatex rapier towel looms, electronic dobby looms and magnetic rod printing machines etc. So it have capability to make all kinds of towel goods from cotton yarn . The products are mainly divided into two series. One is towel series, which include tea towel, bath towel, jacquard towel, high-low loop cutting towel, full background reactive printing towel, etc. and they were warmly welcome by the customers. The other series is yarn-dyed fabric, which include all kinds of cotton yarn-dyed cloth, linen, tencel goods, etc. Over 70% of the products were exported to USA, Europe, Japan and other countries and regions. The company has gained the Import & Export authorization and can do business with foreign countries directly. If you are prepared to purchase such kinds of products, please feel free to contact us. We will quote our competitive price for you with promot delivery. We are looking forward to your soonest reply. Best Regards, Jason Wong Gerneral Manager Nanhai Towel Co., Ltd. Add: Room 722,7/F, Furong Digital Center, No.1, Zumiao Rd, Foshan, Guangdong, China Tel: ++86-757-3982 666 Fax: ++86-757-2283667 e-mail: bosunservice@sina.com If the e-mail has been received in error, please let us know and your e-mail address will be removed. Thanks. From owner-ecm@wyvern.aciri.org Sat Jul 29 11:26:27 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA19593 for ; Sat, 29 Jul 2000 11:26:26 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id IAA28241 for ecm-outgoing; Sat, 29 Jul 2000 08:22:40 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from altavistausa.com (max1-3.newyork.corecomm.net [209.81.238.131]) by wyvern.aciri.org (8.9.3/8.9.3) with SMTP id IAA28236; Sat, 29 Jul 2000 08:22:36 -0700 (PDT) (envelope-from callback123@altavistausa.com) From: Subject: re :Lorrrain,I believe you will like this, Japan $.05, malaysia$24 Date: Sat, 29 Jul 2000 12:18:59 Message-Id: <552.505636.476936@altavistausa.com> Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" Sender: owner-ecm@aciri.org Precedence: bulk Untitled Document


Today, everyone knows the impact of the Internet.
But not everyone nows how to cut their phone bill in 1/2. Just a Few examples!!!!

Uk $.04     France $.06    UAE  $.31    Saudi Arabia $.59     Denmark $.04      Sweden $.05

Click Here Now.

If you do not have flash please go to
www.macromedia.com
to download it.

From owner-ecm@wyvern.aciri.org Sat Jul 29 13:08:22 2000 Received: from wyvern.aciri.org (wyvern.aciri.org [192.150.187.14]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16018 for ; Sat, 29 Jul 2000 13:08:22 -0400 (EDT) Received: (from majordomo@localhost) by wyvern.aciri.org (8.9.3/8.9.3) id KAA28683 for ecm-outgoing; Sat, 29 Jul 2000 10:06:21 -0700 (PDT) (envelope-from owner-ecm@smtp.aciri.org) X-Authentication-Warning: wyvern.aciri.org: majordomo set sender to owner-ecm@smtp.aciri.org using -f Received: from altavistausa.com (max1-3.newyork.corecomm.net [209.81.238.131]) by wyvern.aciri.org (8.9.3/8.9.3) with SMTP id KAA28672; Sat, 29 Jul 2000 10:06:17 -0700 (PDT) (envelope-from callback123@altavistausa.com) From: Subject: From Lorraine,as promised,I can lower your bill Date: Sat, 29 Jul 2000 14:02:40 Message-Id: <847.552950.879512@altavistausa.com> Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" Sender: owner-ecm@aciri.org Precedence: bulk Untitled Document


Today, everyone knows the impact of the Internet.
But not everyone nows how to cut their phone bill in 1/2. Just a Few examples!!!!

Uk $.04     France $.06    UAE  $.31    Saudi Arabia $.59     Denmark $.04      Sweden $.05

Click Here Now.

If you do not have flash please go to
www.macromedia.com
to download it.