From nobody Tue Nov 1 15:36:29 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9665612961A; Tue, 1 Nov 2016 15:36:24 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.397 X-Spam-Level: X-Spam-Status: No, score=-3.397 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-1.497] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lSSqOVhzmOQR; Tue, 1 Nov 2016 15:36:23 -0700 (PDT) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by ietfa.amsl.com (Postfix) with ESMTP id 62A0A129589; Tue, 1 Nov 2016 15:36:23 -0700 (PDT) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 0F18817B64; Tue, 1 Nov 2016 15:36:23 -0700 (PDT) Date: Tue, 1 Nov 2016 15:36:22 -0700 From: hiren panchasara To: Bob Briscoe Message-ID: <20161101223622.GW71456@strugglingcoder.info> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="PpfQBwiAIZcuDX26" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Archived-At: Cc: tcpm IETF list , AQM IETF list , tsvwg IETF list , TCP Prague List Subject: Re: [tcpPrague] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2016 22:36:24 -0000 --PpfQBwiAIZcuDX26 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi Bob, A tiny correction in-line: On 11/01/16 at 12:02P, Bob Briscoe wrote: [snip] > * Data Centre TCP (DCTCP) for > o Linux (in the mainline kernel ) > o FreeBSD patch > https://svnweb.freebsd.org/base?view=revision&revision=r277054 It's been integrated in mainline FreeBSD for quite some time now. :-) Thanks for the whole writeup, btw. Cheers, Hiren --PpfQBwiAIZcuDX26 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF7BAABCgBmBQJYGRjjXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lLKwH+N/rsPQuE9jqdWNHS++j39q0 3kXUQd9y/5i4vLas95wmYDeywfR5l/v07uB5T+ybT3TPBaEwIyzYAp4iF9SHoMIz +BB0K6sDPqRvgfABiJ/mz6pet6aVq0OPfIVxTY0SU0ehA98cGVdmyT2ld2ORI1Q5 mEl0qM/YmrXPpEdcfjowm8JIrdkJZrlr9RyGROmN0YXnB0vKI0fml7zhe7I4/chN wntaZ0p36YKdk20/2TG9GtznCeaAzuoK5YV+xkogB0HN+ALkfGobH8RRSNf3JodP IyCGqunWI80HP6QFYjtHspChIkvM5ETj3dWrFLeCYY8BGfHmXEs/mfYZkwLvyw== =fPO9 -----END PGP SIGNATURE----- --PpfQBwiAIZcuDX26-- From nobody Sat Nov 19 07:40:32 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 76E2D128B44; Sat, 19 Nov 2016 07:40:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hRxUdeL5UvMH; Sat, 19 Nov 2016 07:40:20 -0800 (PST) Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C6E2C12950F; Sat, 19 Nov 2016 07:40:19 -0800 (PST) Received: from [85.133.27.70] (port=46054 helo=[192.168.212.35]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.87) (envelope-from ) id 1c87kV-00042g-Pt; Sat, 19 Nov 2016 15:40:17 +0000 From: Bob Briscoe To: Mirja Kuehlewind References: <147791289909.32484.2914298988666416427.idtracker@ietfa.amsl.com> <82E99853-7EE1-46E5-B82F-4A09814F6715@tik.ee.ethz.ch> <0e03a023-fde6-1a8d-e534-62784ad4701d@bobbriscoe.net> <73935f41-fa95-d136-1b3d-85778be9c7df@bobbriscoe.net> Message-ID: Date: Sat, 19 Nov 2016 13:57:42 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <73935f41-fa95-d136-1b3d-85778be9c7df@bobbriscoe.net> Content-Type: multipart/alternative; boundary="------------C0522C4F03029748F1697AE4" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.dnsblock1.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net Archived-At: Cc: tcpm IETF list , tsvwg IETF list , TCP Prague List Subject: Re: [tcpPrague] New Version Notification for draft-ietf-tcpm-accurate-ecn-02.txt X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2016 15:40:23 -0000 This is a multi-part message in MIME format. --------------C0522C4F03029748F1697AE4 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Mirja, During the discussion on draft-briscoe-tsvwg-ecn-l4s-id, you raised the concern about the CE marking being ambiguous (it could have been ECT(0) or ECT(1) before marking). The draft says why there is a low risk CE marks will arrive at a second bottleneck, and why the risk of causing spurious retransmissions is lower still (there have to be >3 contiguous CE markings from the upstream queue). It's quoted below for your convenience. I think we should also the following to this text (as pointed out recently by Koen): "On the plus side, misclassifying a CE-marked Classic packet delivers the congestion notification to the end-to-end transport without any Classic queuing delay. " Risk of reordering classic CE packets: Having to classify all CE packets as L4S risks some classic CE packets arriving early, which is a form of reordering. Reordering can cause the TCP sender to retransmit spuriously. However, one or two packets delivered early does not cause any spurious retransmissions because the subsequent packets continue to move the cumulative acknowledgement boundary forwards. Anyway, the risk of reordering would be low, because: i) it is quite unusual to experience more than one bottleneck queue on a path; ii) even then, reordering would only occur if there was simultaneous mixing of classic and L4S traffic, which would be more unlikely in an access link, which is where most bottlenecks are located; iii) even then, spurious retransmissions would only occur if a contiguous sequence of three or more classic CE packets from one bottleneck arrived at the next, which should in itself happen very rarely with a good AQM. The risk would be completely eliminated in AQMs that were transport-aware (but they should not need to be); Bob On 12/11/16 16:04, Bob Briscoe wrote: > Mirja, > > Sorry, it's become rather hard to follow my replies, given sent in > pieces. I've tagged the questions and answers to make it clearer: > > On 12/11/16 12:53, Bob Briscoe wrote: >> Mirja, >> >> The previous response only addressed your first question, 'cos I sent >> in haste ( I had to go offline to board the flight). Below I address >> the other questions. >> >> On 12/11/16 12:21, Bob Briscoe wrote: >>> Mirja, >>> >>> It's great you've managed to find time to finish the implementation >>> - well done! >>> Pity I didn't find time to reply sooner :O >>> inline... >>> >>> >>> On 31/10/16 12:00, Mirja Kühlewind wrote: >>>> Hi Bob, hi Richard, >>>> >>>> I’ve submitted a new version of this draft with only minor changes. >>>> I finished my implementation but still have to do some testing. >>>> Current code is here: >>>> >>>> https://github.com/mirjak/linux-accecn >>>> >>>> This is really not tested carefully I but have right now some >>>> problems with my testing environment… anyway wanted to submit a new >>>> draft version before the deadline. However, given my current >>>> implementation is, as far as I can tell now, complete, I believe >>>> the spec is fins. >>>> >>>> I only have a few comments/questions regarding the fall-back rules >>>> we describe in the drafts (sec. 3.2.4): >>>> *Q1 *- we don’t say anything about retransmitting the SYN. Should a >>>> retransmitted SYN try to negotiate AccECN or not, or are we relying >>>> on the classic ECN fall-back rules in this case (which would be >>>> clearing the ECN flags)? >>> *A1 *- Good point - I was (we were?) thinking about the option being >>> blocked by a middlebox, but not the AccECN flags in the main header. >>> >>> We need to allow plenty of implementer choice here, but give a good >>> default. The default depends on how likely an AccECN SYN will be >>> dropped because of the AccECN flags. I think your tests [PAM paper] >>> found no drop due to these flags. >>> >>> So, here's some suggesting text: >>> >>> Measurement tests [PAM paper] so far have found that it is extremely >>> rate for middleboxes to drop a SYN just because all three ECN flags >>> are set (i.e. it is negotiating to use AccECN). Therefore, if the >>> sender of an initial AccECN SYN times out waiting for the SYN/ACK it >>> will more likely be due to congestion loss than middlebox >>> interference. This suggests the following strategy will maximize the >>> chances of using AccECN while minimizing delay. >>> >>> If the sender of an AccECN SYN times out before receiving the >>> SYN/ACK, the sender SHOULD attempt to negotiate the use of AccECN at >>> least one more time by continuing to set all three TCP ECN flags on >>> the first retransmitted SYN. If this first retransmission also fails >>> to be acknowledged, the sender SHOULD send subsequent >>> retransmissions of the SYN without any ECN flags set. >>> >>> This adds delay, but no more delay than normal, if the cause was >>> congestion. In the case that a middlebox drops an AccECN (or ECN) >>> SYN deliberately, this strategy will add delay, but current >>> measurements show this case is likely to be rare. >>> >>> Implementers MAY use other fall-back strategies if they >>> are found to be more effective (e.g. attempting >>> to retransmit an AccECN SYN more than once, (most appropriate during >>> high levels of congestion); or falling back to classic ECN feedback >>> rather than non-ECN). > , or cached knowledge associated with the remote host. >>> >>> Nit in 3.2.4: s/tha/that/ >>> >>> I believe there was a bug where the Linux connection-manager got >>> into a black-holing state if it didn't support the TFO option. Even >>> tho the bug was fixed, there are likely to be bugged Linux routers >>> out there. So we need to check the black hole doesn't apply to >>> AccECN SYNs as well. If it does, we will need to suggest restarting >>> a new connection as a fall-back. I'll check the details. >>> >>> >>> Bob >>> >>> >>>> *Q2* - What if we receive a second SYN with AccECN negotiation? We >>>> would probably send a SYNACK that negotiates AccECN but no Option, >>>> assuming that the SYNACK was lost, right? And then probably also go >>>> into a mode where we don’t send the option anymore..? We don’t say >>>> anything about this case in the draft. >> *A2* - The trouble is that SYN/ACKs are just as likely to be lost due >> to congestion (e.g. found this in our L4S testing). But I agree that, >> in this case, I think send the SYN/ACK without the option, but try to >> include the option pretty soon afterwards. >> >> Rest of email will follow when I land... >> >> >> Bob >> >>>> *Q3* - Also should we add a sentence that one could monitor the >>>> segments that carry an option and if it is much more likely that >>>> such a segment would be lost, then disable sending the option? Of >>>> current that only works for segments that carry data as well. > *A3* Yes. 3.2.4 seems like the best place for that as well. >>>> *Q4* - And finally we probably should to add an experimentation >>>> section, where we say that the question which fallbacks are >>>> actually needed must be investigated by experimentation. (Currently >>>> I only implemented clearing of the ECN flags when retransmitting a >>>> segment with the SYN flag set). Then we could also move any text we >>>> would like to keep from Appendix C into this section. > > *A4* - We have 1.3 for this: "Experiment Goals". We could add a > sentence saying most of the issues that need experimental verification > concern path traversal. >>>> *Q5* - Also another side mark: I currently implemented AccECN and >>>> increase the respective counters but don’t use them of course. >>>> Maybe we should add one section in the draft that is directed to >>>> people that would like to use these counters of an existing >>>> implementation, e.g. adding a warning that counters might not >>>> increase even if AccECN is negotiation because options could be >>>> stripped, and that the CE packet counter could increase differently >>>> as it includes control packets without data and therefore usually >>>> both values should be checked… we say this all in the draft >>>> somewhere but maybe it be good to has this summarized in some kind >>>> of ‚usage‘ section…? > > *A5* Yes. Perhaps "AccECN Usage Examples" > > And this would be a good place to talk about behaviour that depends on > other parallel updates to ECN that are in progress (whether > experimental or PS), e.g.: > > ecn-fallback: > Perhaps this (informative?) usage section could also repeat relevant > stuff from ecn-fallback that isn't specific to AccECN, but applies > generally to ECN TCP. > > generalized-ecn: > This usage section would also be a good place to address open issue #3 > (Appx C). But I'm not clear where to define the normative behaviour > concerning the interationc between ECT control packets and AccECN. It > would be really odd to talk about the details of AccECN counters in > generalized-ecn. But it is hard to talk about generalized ECN in the > AccECN draft, because it is not yet a WG draft, and we really don't > want to hold back AccECN. > > *More Nits** > *In 3.3: > s/intervene/intervenes/ >>>> >>>> I guess we could prepare these changes (if any) and submit IETF >>>> Monday (if not today again) and ask for wglc in tcpm…? >>>> >>>> Mirja >>>> >>>> >>>> >>>>> Am 31.10.2016 um 12:21 schrieb internet-drafts@ietf.org: >>>>> >>>>> >>>>> A new version of I-D, draft-ietf-tcpm-accurate-ecn-02.txt >>>>> has been successfully submitted by Mirja Kühlewind and posted to the >>>>> IETF repository. >>>>> >>>>> Name: draft-ietf-tcpm-accurate-ecn >>>>> Revision: 02 >>>>> Title: More Accurate ECN Feedback in TCP >>>>> Document date: 2016-10-31 >>>>> Group: tcpm >>>>> Pages: 37 >>>>> URL: >>>>> https://www.ietf.org/internet-drafts/draft-ietf-tcpm-accurate-ecn-02.txt >>>>> >>>>> Status: >>>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ >>>>> Htmlized: https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-02 >>>>> Diff: >>>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-02 >>>>> >>>>> Abstract: >>>>> Explicit Congestion Notification (ECN) is a mechanism where >>>>> network >>>>> nodes can mark IP packets instead of dropping them to indicate >>>>> incipient congestion to the end-points. Receivers with an ECN- >>>>> capable transport protocol feed back this information to the >>>>> sender. >>>>> ECN is specified for TCP in such a way that only one feedback >>>>> signal >>>>> can be transmitted per Round-Trip Time (RTT). Recently, new TCP >>>>> mechanisms like Congestion Exposure (ConEx) or Data Center TCP >>>>> (DCTCP) need more accurate ECN feedback information whenever more >>>>> than one marking is received in one RTT. This document >>>>> specifies an >>>>> experimental scheme to provide more than one feedback signal >>>>> per RTT >>>>> in the TCP header. Given TCP header space is scarce, it overloads >>>>> the three existing ECN-related flags in the TCP header and >>>>> provides >>>>> additional information in a new TCP option. >>>>> >>>>> >>>>> >>>>> >>>>> Please note that it may take a couple of minutes from the time of >>>>> submission >>>>> until the htmlized version and diff are available at tools.ietf.org. >>>>> >>>>> The IETF Secretariat >>> >> > > -- > ________________________________________________________________ > Bob Briscoehttp://bobbriscoe.net/ -- ________________________________________________________________ Bob Briscoehttp://bobbriscoe.net/ --------------C0522C4F03029748F1697AE4 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Mirja,

During the discussion on draft-briscoe-tsvwg-ecn-l4s-id, you raised the concern about the CE marking being ambiguous (it could have been ECT(0) or ECT(1) before marking). The draft says why there is a low risk CE marks will arrive at a second bottleneck, and why the risk of causing spurious retransmissions is lower still (there have to be >3 contiguous CE markings from the upstream queue). It's quoted below for your convenience.

I think we should also the following to this text (as pointed out recently by Koen):
    "On the plus side, misclassifying a CE-marked Classic packet delivers the congestion notification to the end-to-end transport without any Classic queuing delay. "


   Risk of reordering classic CE packets:  Having to classify all CE
      packets as L4S risks some classic CE packets arriving early, which
      is a form of reordering.  Reordering can cause the TCP sender to
      retransmit spuriously.  However, one or two packets delivered
      early does not cause any spurious retransmissions because the
      subsequent packets continue to move the cumulative acknowledgement
      boundary forwards.  Anyway, the risk of reordering would be low,
      because: i) it is quite unusual to experience more than one
      bottleneck queue on a path; ii) even then, reordering would only
      occur if there was simultaneous mixing of classic and L4S traffic,
      which would be more unlikely in an access link, which is where
      most bottlenecks are located; iii) even then, spurious
      retransmissions would only occur if a contiguous sequence of three
      or more classic CE packets from one bottleneck arrived at the
      next, which should in itself happen very rarely with a good AQM.
      The risk would be completely eliminated in AQMs that were
      transport-aware (but they should not need to be);


Bob

On 12/11/16 16:04, Bob Briscoe wrote:
Mirja,

Sorry, it's become rather hard to follow my replies, given sent in pieces. I've tagged the questions and answers to make it clearer:

On 12/11/16 12:53, Bob Briscoe wrote:
Mirja,

The previous response only addressed your first question, 'cos I sent in haste ( I had to go offline to board the flight). Below I address the other questions.

On 12/11/16 12:21, Bob Briscoe wrote:
Mirja,

It's great you've managed to find time to finish the implementation - well done!
Pity I didn't find time to reply sooner :O
inline...


On 31/10/16 12:00, Mirja Kühlewind wrote:
Hi Bob, hi Richard,

I’ve submitted a new version of this draft with only minor changes. I finished my implementation but still have to do some testing. Current code is here:

https://github.com/mirjak/linux-accecn

This is really not tested carefully I but have right now some problems with my testing environment… anyway wanted to submit a new draft version before the deadline. However, given my current implementation is, as far as I can tell now, complete, I believe the spec is fins.

I only have a few comments/questions regarding the fall-back rules we describe in the drafts (sec. 3.2.4):
Q1 - we don’t say anything about retransmitting the SYN. Should a retransmitted SYN try to negotiate AccECN or not, or are we relying on the classic ECN fall-back rules in this case (which would be clearing the ECN flags)?
A1 - Good point - I was (we were?) thinking about the option being blocked by a middlebox, but not the AccECN flags in the main header.

We need to allow plenty of implementer choice here, but give a good default. The default depends on how likely an AccECN SYN will be dropped because of the AccECN flags. I think your tests [PAM paper] found no drop due to these flags.

So, here's some suggesting text:

Measurement tests [PAM paper] so far have found that it is extremely rate for middleboxes to drop a SYN just because all three ECN flags are set (i.e. it is negotiating to use AccECN). Therefore, if the sender of an initial AccECN SYN times out waiting for the SYN/ACK it will more likely be due to congestion loss than middlebox interference. This suggests the following strategy will maximize the chances of using AccECN while minimizing delay.

If the sender of an AccECN SYN times out before receiving the SYN/ACK, the sender SHOULD attempt to negotiate the use of AccECN at least one more time by continuing to set all three TCP ECN flags on the first retransmitted SYN. If this first retransmission also fails to be acknowledged, the sender SHOULD send subsequent retransmissions of the SYN without any ECN flags set.

This adds delay, but no more delay than normal, if the cause was congestion. In the case that a middlebox drops an AccECN (or ECN) SYN deliberately, this strategy will add delay, but current measurements show this case is likely to be rare.

Implementers MAY use other fall-back strategies if they
are found to be more effective (e.g. attempting
to retransmit an AccECN SYN more than once, (most appropriate during
high levels of congestion); or falling back to classic ECN feedback
rather than non-ECN).
, or cached knowledge associated with the remote host.

Nit in 3.2.4: s/tha/that/

I believe there was a bug where the Linux connection-manager got into a black-holing state if it didn't support the TFO option. Even tho the bug was fixed, there are likely to be bugged Linux routers out there. So we need to check the black hole doesn't apply to AccECN SYNs as well. If it does, we will need to suggest restarting a new connection as a fall-back. I'll check the details.


Bob


Q2 - What if we receive a second SYN with AccECN negotiation? We would probably send a SYNACK that negotiates AccECN but no Option, assuming that the SYNACK was lost, right? And then probably also go into a mode where we don’t send the option anymore..? We don’t say anything about this case in the draft.
A2 - The trouble is that SYN/ACKs are just as likely to be lost due to congestion (e.g. found this in our L4S testing). But I agree that, in this case, I think send the SYN/ACK without the option, but try to include the option pretty soon afterwards.

Rest of email will follow when I land...


Bob

Q3 - Also should we add a sentence that one could monitor the segments that carry an option and if it is much more likely that such a segment would be lost, then disable sending the option? Of current that only works for segments that carry data as well.
A3 Yes. 3.2.4 seems like the best place for that as well.
Q4 - And finally we probably should to add an experimentation section, where we say that the question which fallbacks are actually needed must be investigated by experimentation. (Currently I only implemented clearing of the ECN flags when retransmitting a segment with the SYN flag set). Then we could also move any text we would like to keep from Appendix C into this section.

A4 - We have 1.3 for this: "Experiment Goals". We could add a sentence saying most of the issues that need experimental verification concern path traversal.
Q5 - Also another side mark: I currently implemented AccECN and increase the respective counters but don’t use them of course. Maybe we should add one section in the draft that is directed to people that would like to use these counters of an existing implementation, e.g. adding a warning that counters might not increase even if AccECN is negotiation because options could be stripped, and that the CE packet counter could increase differently as it includes control packets without data and therefore usually both values should be checked… we say this all in the draft somewhere but maybe it be good to has this summarized in some kind of ‚usage‘ section…?

A5 Yes. Perhaps "AccECN Usage Examples"

And this would be a good place to talk about behaviour that depends on other parallel updates to ECN that are in progress (whether experimental or PS), e.g.:

ecn-fallback:
Perhaps this (informative?) usage section could also repeat relevant stuff from ecn-fallback that isn't specific to AccECN, but applies generally to ECN TCP.

generalized-ecn:
This usage section would also be a good place to address open issue #3 (Appx C). But I'm not clear where to define the normative behaviour concerning the interationc between ECT control packets and AccECN. It would be really odd to talk about the details of AccECN counters in generalized-ecn. But it is hard to talk about generalized ECN in the AccECN draft, because it is not yet a WG draft, and we really don't want to hold back AccECN.

More Nits
In 3.3:
s/intervene/intervenes/

I guess we could prepare these changes (if any) and submit IETF Monday (if not today again) and ask for wglc in tcpm…?

Mirja



Am 31.10.2016 um 12:21 schrieb internet-drafts@ietf.org:


A new version of I-D, draft-ietf-tcpm-accurate-ecn-02.txt
has been successfully submitted by Mirja Kühlewind and posted to the
IETF repository.

Name:        draft-ietf-tcpm-accurate-ecn
Revision:    02
Title:        More Accurate ECN Feedback in TCP
Document date:    2016-10-31
Group:        tcpm
Pages:        37
URL: https://www.ietf.org/internet-drafts/draft-ietf-tcpm-accurate-ecn-02.txt
Status: https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/
Htmlized: https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-02
Diff: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-02

Abstract:
   Explicit Congestion Notification (ECN) is a mechanism where network
   nodes can mark IP packets instead of dropping them to indicate
   incipient congestion to the end-points.  Receivers with an ECN-
   capable transport protocol feed back this information to the sender.
   ECN is specified for TCP in such a way that only one feedback signal
   can be transmitted per Round-Trip Time (RTT).  Recently, new TCP
   mechanisms like Congestion Exposure (ConEx) or Data Center TCP
   (DCTCP) need more accurate ECN feedback information whenever more
   than one marking is received in one RTT.  This document specifies an
   experimental scheme to provide more than one feedback signal per RTT
   in the TCP header.  Given TCP header space is scarce, it overloads
   the three existing ECN-related flags in the TCP header and provides
   additional information in a new TCP option.




Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat 



-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------C0522C4F03029748F1697AE4-- From nobody Mon Nov 21 06:27:46 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 82547129A75; Mon, 21 Nov 2016 06:27:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TfPvuGB5g2zo; Mon, 21 Nov 2016 06:27:42 -0800 (PST) Received: from iramx2.ira.uni-karlsruhe.de (iramx2.ira.uni-karlsruhe.de [141.3.10.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5781112967D; Mon, 21 Nov 2016 06:27:42 -0800 (PST) Received: from i72vorta.tm.uni-karlsruhe.de ([141.3.71.26] helo=i72vorta.tm.kit.edu) by iramx2.ira.uni-karlsruhe.de with esmtp port 25 iface 141.3.10.81 id 1c8pZM-0002aP-4u; Mon, 21 Nov 2016 15:27:40 +0100 Received: from [IPv6:::1] (ip6-localhost [IPv6:::1]) by i72vorta.tm.kit.edu (Postfix) with ESMTPS id 39DBBB0025A; Mon, 21 Nov 2016 15:27:57 +0100 (CET) To: Bob Briscoe , tsvwg IETF list , TCP Prague List , AQM IETF list , tcpm IETF list References: From: "Bless, Roland (TM)" Organization: Institute of Telematics, Karlsruhe Institute of Technology Message-ID: Date: Mon, 21 Nov 2016 15:27:57 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-ATIS-AV: ClamAV (iramx2.ira.uni-karlsruhe.de) X-ATIS-Timestamp: iramx2.ira.uni-karlsruhe.de 1479738460. Archived-At: Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 14:27:44 -0000 Hi Bob and all, see below. Am 01.11.2016 um 01:02 schrieb Bob Briscoe: > A few people have been working away to specify and document all the > aspects of the new Low Latency, Low Loss, Scalable throughput (L4S) > service, which held a successful BoF in Berlin. As the decision was to > try to work across multiple WGs, I thought it would be useful to give ... Thanks for putting this together. > * Dual Queue Coupled AQM > o With Curvy RED for Linux (access available shortly) > o With PI2 for Linux [*UPDATED*] I'll repeat my concerns that I already expressed at the L4S BOF in Berlin: While I agree that we probably need to separate low-delay congestion control schemes from traditional "queue-filling" congestion schemes, I strongly suggest to avoid putting a congestion control-specific coupling scheme into the network (a classic case for applying the "end-to-end arguments in system design"). The current Dual queue coupled AQM proposal has got a coupling based on a congestion control specific dropping law p_C=(p_L/2)². So if congestion control schemes change then this coupling needs to be adapted. For example, the currently proposed scheme may fail if that vast majority of TCP traffic is using BBR other some other forthcoming CC scheme instead of Cubic, Reno, Compound etc. The same applies to draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping likelihood is defined. Regards, Roland From nobody Tue Nov 22 06:35:23 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE543129A31; Tue, 22 Nov 2016 06:35:12 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.221 X-Spam-Level: X-Spam-Status: No, score=-4.221 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.onmicrosoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0hCvR9R4Aa6X; Tue, 22 Nov 2016 06:35:10 -0800 (PST) Received: from sessmg23.ericsson.net (sessmg23.ericsson.net [193.180.251.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D3666129A32; Tue, 22 Nov 2016 06:35:08 -0800 (PST) X-AuditID: c1b4fb2d-0bbff70000000994-3d-5834579a78fc Received: from ESESSHC023.ericsson.se (Unknown_Domain [153.88.183.87]) by (Symantec Mail Security) with SMTP id 2B.25.02452.A9754385; Tue, 22 Nov 2016 15:35:07 +0100 (CET) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (153.88.183.145) by oa.msg.ericsson.com (153.88.183.87) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 22 Nov 2016 15:35:06 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.onmicrosoft.com; s=selector1-ericsson-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=1JSHM/0OUFhHIn5RGOnp/Kg57Lkxpn9QEhr1rxGpBHc=; b=glHPGBWAUAIijOAoorT9dOIIFXuUwyyNuSpIjsfYeYuCsAc8zughLWCroYACjZfvH+sNtl8I03Tv0JXRJNHw72MERxtq2GolkN7eQZ50bDDTVEqHKgQPX0msj508XyYuPIIWl8jpncUbymTFDXUwSF9RfqACvwYhkuQEDqn6tMg= Received: from DB4PR07MB348.eurprd07.prod.outlook.com (10.141.234.148) by DB4PR07MB348.eurprd07.prod.outlook.com (10.141.234.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.747.5; Tue, 22 Nov 2016 14:35:05 +0000 Received: from DB4PR07MB348.eurprd07.prod.outlook.com ([10.141.234.148]) by DB4PR07MB348.eurprd07.prod.outlook.com ([10.141.234.148]) with mapi id 15.01.0747.006; Tue, 22 Nov 2016 14:35:05 +0000 From: Ingemar Johansson S To: "Bless, Roland (TM)" , Bob Briscoe , tsvwg IETF list , TCP Prague List , AQM IETF list , tcpm IETF list Thread-Topic: [tcpPrague] [aqm] L4S status update Thread-Index: AQHSRDIMrBiIIyXAnkeCaadJdlnwcqDlDZUg Date: Tue, 22 Nov 2016 14:35:05 +0000 Message-ID: References: In-Reply-To: Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=ingemar.s.johansson@ericsson.com; x-originating-ip: [192.176.1.89] x-ms-office365-filtering-correlation-id: 62566d1f-6b6e-4083-9074-08d412e4c093 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:DB4PR07MB348; x-microsoft-exchange-diagnostics: 1; DB4PR07MB348; 7:6yJcwqJnmZQ9ndDPm4o1Q4nSqtQmLg+a7XDcw7Ixf7GtXwjLgYor9ipjwZAe/FzcgcGKyGS6bbiXnD+/KE9XF+g3BomcG2plWUnuP92oT3Kc9Avzf0+woQk0/GIRt5NGUigCw2KRwISUNYcsXTBJmz/jHNF0eb1mV/VEjn1/Eunoj8cYam0v4/32S2IhHfPO4WIwGJYVtg1JTN5Yt3FSixOhQ5u4TyropBQa63VnUcfa40YbbcKD/aMjn0xAwcjzbbf60bzaQ3DWPapF5zhJycUgNVzf78ifC9hKwFIlK1N33+BHhkcDJscxsxgCzFvYxNHZKXhg5+zQ1r22HAShKadQfK5sC2nvesLeffirGmqw+NiyCQjKmAui5ZeOY+C/CZXiL0ttDAaCn2chqauhs6hf/3UJT3q7iNMk0dqAMNFvNzvcnHTbh23KsjcGTu1NHr92hmj8Bk+kWyYlGncOLg== x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(166708455590820); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6060326)(6040307)(6045199)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6061324)(6041248); SRVR:DB4PR07MB348; BCL:0; PCL:0; RULEID:; SRVR:DB4PR07MB348; x-forefront-prvs: 0134AD334F x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(199003)(13464003)(189002)(5001770100001)(97736004)(3846002)(68736007)(15650500001)(8676002)(3660700001)(229853002)(106356001)(6116002)(106116001)(92566002)(105586002)(561944003)(101416001)(107886002)(102836003)(189998001)(76176999)(50986999)(7736002)(33656002)(74316002)(2950100002)(7846002)(8936002)(305945005)(2906002)(54356999)(2900100001)(76576001)(7696004)(2171001)(81156014)(6506003)(3280700002)(38730400001)(122556002)(66066001)(5660300001)(86362001)(77096005)(81166006)(9686002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB4PR07MB348; H:DB4PR07MB348.eurprd07.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Nov 2016 14:35:05.7240 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB4PR07MB348 X-OriginatorOrg: ericsson.com X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa0iTURjHOXsvex1OTkvxcZnUyKBS81atFFEQEqkwi7YkqKFvupwX9pqk 9EFKA29llJWGOctMNzMcZlZe560huiQlMy9kZl4KQYJllub2GvTtd87v/5yH5+EwhKSMkjLq 5DRWm6zSyGgRWaJ8ofC+rwhU+g5+wfLaVjd59ycdJV8qukvJG83lAnlxVgcp7/k2TofSEfOT b6mIysplQYSuoomIImJEwXGsRp3OaveGnBMlGGazyVS99NKPgd9kFrrukoccGMCB8Ef/ks5D IkaC6xD0DjZT/OENgvlsi92QuJCA/Kl2oa1EgosF0HYlmk/1IKhYsVA2QeNgqDFZkU044wkE Y/nDhE1sxr4wUTBL29gZ+4F+7bWQZ3/In+izZ0jsCUN3bpI2FuMYqO0uWH+IWe+ghWrDeRs6 4CCY78+0JRDeCpPWCXuawK4wOl0u4MfBUNlsIXh2gbnPqxSfj4WlD4UUf78Nequ/0zwfhZaq 3I38EVjMrrYPDDifhKuNA0JeqGG1v2+jIAh0OUaCD40i6DG2Il64Q4mpc0PUU2C1dAr4dbHw 5GkO4hchhfGh3A12h9mxFqoI7Sr9bwqefWCk+DbN8x6oqlggSu172QTmkmlSh0g9cuFYjkuK 9w/wYbXqWI5LSfZJZtOMaP3TdDSseDchw0KYCWEGyRzFvuEBSgmlSucykkwIGELmLB6KDFRK xHGqjExWm3JWe1HDcia0hSFlruL9NZMKCY5XpbGJLJvKav9ZAeMgzUK39GvPux6cDk+ifkmj ew1Rha11D6eE1uLl2huvjksPqcvi2ttiHT0F7EzcA2TNTAw7DEFOHl4/RW1zc8eEl0NO3Htc YfqaoukMLZPNDAd3Pao/czJgu+vigUjjKT9zptrsRD/7GDPTOezutmPExePavoaIg15tRo8L O9/1livC3stILkHlt5vQcqq//orvVDADAAA= Archived-At: Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 14:35:13 -0000 Hi Roland + others. As regards to comments around other new congestion control algorithms and t= hat they may need adapted dropping likelihood relation between a classic qu= eue and L4S queue. I have not tried out but I suspect that BBR may get an unfair treatment, at= the same time it is possible that other delay based CCs may suffer.=20 They question however if this is a major problem?, one may as well see this= as an incentive to switch over to scalable congestion controls and L4S ? T= here is after all no requirement to stick to a particular congestion contro= l no matter what. ? The question whether or not endpoint dependency should be built into the ne= tworks is of course a valid question but given that the state of the art c= ongestion controls like Reno and Cubic rely on a 1/sqrt(p) function then th= at is perhaps OK ?.=20 There will for a foreseeable time come updated endpoint based congestion co= ntrol algorithms that are optimized for one thing or the other (I am guilt= y too :-). However if one can distinguish between two classes (classic and = L4S) where classic belong to the 1/sqrt(p) family then I believe that it is= possible to solve the problem. If we try find a solution where classic =3D= "all sorts of non-scalable non-L4S dependent congestion controls" then I b= elieve that we easily end up in big problems. /Ingemar > -----Original Message----- > From: Bless, Roland (TM) [mailto:roland.bless@kit.edu] > Sent: den 21 november 2016 15:28 > To: Bob Briscoe ; tsvwg IETF list ; > TCP Prague List ; AQM IETF list ; tcpm > IETF list > Subject: Re: [tcpPrague] [aqm] L4S status update >=20 > Hi Bob and all, >=20 > see below. >=20 > Am 01.11.2016 um 01:02 schrieb Bob Briscoe: > > A few people have been working away to specify and document all the > > aspects of the new Low Latency, Low Loss, Scalable throughput (L4S) > > service, which held a successful BoF in Berlin. As the decision was to > > try to work across multiple WGs, I thought it would be useful to give .= .. >=20 > Thanks for putting this together. >=20 > > * Dual Queue Coupled AQM > > o With Curvy RED for Linux (access available shortly) > > o With PI2 for Linux > > [*UPDATED*] >=20 > I'll repeat my concerns that I already expressed at the L4S BOF in Berlin= : >=20 > While I agree that we probably need to separate low-delay congestion > control schemes from traditional "queue-filling" congestion schemes, I > strongly suggest to avoid putting a congestion control-specific coupling > scheme into the network (a classic case for applying the "end-to-end > arguments in system design"). > The current Dual queue coupled AQM proposal has got a coupling based on a > congestion control specific dropping law p_C=3D(p_L/2)=B2. So if congesti= on > control schemes change then this coupling needs to be adapted. For > example, the currently proposed scheme may fail if that vast majority of = TCP > traffic is using BBR other some other forthcoming CC scheme instead of > Cubic, Reno, Compound etc. The same applies to draft-briscoe-tsvwg-ecn- > l4s-id, section 2.5, where the dropping likelihood is defined. >=20 > Regards, > Roland >=20 From nobody Tue Nov 22 08:45:52 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 384CE1297A7; Tue, 22 Nov 2016 08:45:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Oo6pQmvr9HVs; Tue, 22 Nov 2016 08:45:42 -0800 (PST) Received: from iramx2.ira.uni-karlsruhe.de (iramx2.ira.uni-karlsruhe.de [141.3.10.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 298321296CA; Tue, 22 Nov 2016 08:45:42 -0800 (PST) Received: from i72vorta.tm.uni-karlsruhe.de ([141.3.71.26] helo=i72vorta.tm.kit.edu) by iramx2.ira.uni-karlsruhe.de with esmtp port 25 iface 141.3.10.81 id 1c9ECR-0004rn-Ei; Tue, 22 Nov 2016 17:45:39 +0100 Received: from [IPv6:::1] (ip6-localhost [IPv6:::1]) by i72vorta.tm.kit.edu (Postfix) with ESMTPS id 2D3FBB00254; Tue, 22 Nov 2016 17:45:58 +0100 (CET) To: Ingemar Johansson S , Bob Briscoe , tsvwg IETF list , TCP Prague List , AQM IETF list , tcpm IETF list References: From: "Bless, Roland (TM)" Organization: Institute of Telematics, Karlsruhe Institute of Technology Message-ID: <70104ef0-3747-9b47-dcb7-be54fe454443@kit.edu> Date: Tue, 22 Nov 2016 17:45:57 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-ATIS-AV: ClamAV (iramx2.ira.uni-karlsruhe.de) X-ATIS-Timestamp: iramx2.ira.uni-karlsruhe.de 1479833139. Archived-At: Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 16:45:44 -0000 Hi Ingemar, my point was that it's probably better to refrain from building CC-specific behavior into network elements as the CC algorithms may evolve faster and in more flexible ways than we can foresee. Thus, it would be good to have a separation (or coupling) scheme that actually doesn't depend on the 1/sqrt(p) dropping law. Am 22.11.2016 um 15:35 schrieb Ingemar Johansson S: > As regards to comments around other new congestion control algorithms > and that they may need adapted dropping likelihood relation between a > classic queue and L4S queue. I have not tried out but I suspect that > BBR may get an unfair treatment, at the same time it is possible that > other delay based CCs may suffer. It would be interesting to see what happens if BBR is sorted into the L4S queue in comparison to what happens if BBR is sorted into the classic queue (BBR isn't reacting to loss according to 1/sqrt(p)). > They question however if this is a major problem?, one may as well > see this as an incentive to switch over to scalable congestion > controls and L4S ? There is after all no requirement to stick to a > particular congestion control no matter what. ? Yes, and that's why I find that built-in coupling law too limiting. > The question whether or not endpoint dependency should be built into > the networks is of course a valid question but given that the state > of the art congestion controls like Reno and Cubic rely on a > 1/sqrt(p) function then that is perhaps OK ?. There will for a > foreseeable time come updated endpoint based congestion control > algorithms that are optimized for one thing or the other (I am > guilty too :-). However if one can distinguish between two classes > (classic and L4S) where classic belong to the 1/sqrt(p) family then I > believe that it is possible to solve the problem. If we try find a > solution where classic = "all sorts of non-scalable non-L4S dependent > congestion controls" then I believe that we easily end up in big > problems. I'm not sure. Maybe we have a class of "queue-filling" CCs and a class of low-delay targeted CCs. Regards, Roland From nobody Tue Nov 22 11:09:59 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 20EA7129E84; Tue, 22 Nov 2016 11:09:46 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1vpxT9PSdtGh; Tue, 22 Nov 2016 11:09:42 -0800 (PST) Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0B91B129B4C; Tue, 22 Nov 2016 11:09:42 -0800 (PST) Received: from [31.185.252.113] (port=55476 helo=[192.168.0.3]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.87) (envelope-from ) id 1c9GRn-0001V6-Qt; Tue, 22 Nov 2016 19:09:40 +0000 To: "Bless, Roland (TM)" References: From: Bob Briscoe Message-ID: Date: Tue, 22 Nov 2016 19:09:39 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------AD7E6DDACF9AC164C6863BEB" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.dnsblock1.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net Archived-At: Cc: tcpm IETF list , AQM IETF list , tsvwg IETF list , TCP Prague List Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 19:09:46 -0000 This is a multi-part message in MIME format. --------------AD7E6DDACF9AC164C6863BEB Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Roland, I share your concern about cc-specific AQMs. But that is not a good characterization of what we're doing. On the current Internet, everything is meant to be somewhat "friendly" to the original TCP cc (now spec'd in RFC5681). All sorts of cc's work alongside that, with slightly different "fairness" properties, and only one AQM is needed to cover them all. Nonetheless, Reno is the "lamest", so everyone has to try to "Do (not much) Harm" to the lamest. *Is any AQM CC-neutral?** *Note rule 5 in the AQM Guidelines [RFC7567] "AQM algorithms SHOULD NOT interpret specific transport protocol behaviors." In general, the advice in that section is sound, but I don't think we realized at that time just how subtle this issue is. Since then, I discovered that the autotuning parameter table in the PIE algorithm is designed very precisely around the 1/sqrt(p) rule of Reno (see Fig 5 in the PI^2 paper ). Similarly, the sqrt control law in Codel claims to be dependent on Reno {Note 1}. The point is that these AQMs still work fine with Cubic, Compound, Westwood, etc, because all these ccs were designed to interwork with Reno. {Note 2} The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID spec) is to enable a shift to a completely different "norm", but still coexist with all the 'Classic' cc's that coexisted around the old "norm". The new norm is intended to be just as fuzzy as the old norm {Note 3}. The idea is two fuzzy clouds of congestion controls, around an old and a new norm that are related together. *BBR** *I believe BBR attempts to be 'friendly' to loss-based flows when competing in the same queue. But it's still research, and we don't yet know how good it is at that in all scenarios, although we do have code to test now. Given BBR currently sets Not-ECT, it would classify itself into the Classic queue of a DualQ AQM, and if it coexists with Reno it /should/ coexist with L4S traffic in the other queue. See Koen's recent posting about this. There would be nothing to stop someone designing a variant of BBR that coexisted in the L4S queue with Scalable CCs like DCTCP (the point being that if the bottleneck was not DualQ it would keep delay low and if if the bottleneck was DualQ it would benefit even more from the lower queuing delay there). However, it would have to be a bit more careful about its whole round trip of queue probing, to avoid increasing the delay in the L4S queue. You'll see that I suggested to Neil Cardwell that they consider probing with a few packets rather than a whole window, e.g. the chirping technique that Mirja and I looked into back in 2010 was designed to find the same knee between rate increase and delay increase, with far fewer packets. I thought of a better way of using chirping a few weeks ago, so I will be returning to that too. *Specs** *There is no statement that all L4S cc's MUST adhere to a 1/p rule. The L4S ID draft says: "The inverse proportionality requirement above is worded as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility when defining these specifications." I hope that 'SHOULD' is fuzzy enough - I suspect adding more words would make it less fuzzy. But I would welcome wording to make it even more fuzzy if you would like to engage in wordsmithing. Nonetheless, we are trying to steer a path between a rock and a hard place. Because, to shift to a much calmer waters beyond the rocks, we have to define some number to relate L4S to Classic. I am wary because when ECN was specified, there were attempts to define ECN as different to drop. However, ECN originally ended up "the same as drop" because no-one could muster enough backing behind any particular number to relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ). Bob {Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time. {Note 2} Actually, I don't believe PIE would work that well with Cubic at v high BDP, once it was far from Reno-friendly, but that's only intuition from the stability analysis, not from actual testing. {Note 3} Indeed, at the moment, when DCTCP is on its own in the L4S queue of the DualQ AQM as coded now, it hits up against a step threshold, which makes it behave as 1/p^2, not 1/p. For now, that's just because we didn't want to change too much about DCTCP at one time. But it's also got some nice properties. This will all need to be discussed as the DualQ AQM is specified more deeply. On 21/11/16 14:27, Bless, Roland (TM) wrote: > Hi Bob and all, > > see below. > > Am 01.11.2016 um 01:02 schrieb Bob Briscoe: >> A few people have been working away to specify and document all the >> aspects of the new Low Latency, Low Loss, Scalable throughput (L4S) >> service, which held a successful BoF in Berlin. As the decision was to >> try to work across multiple WGs, I thought it would be useful to give ... > Thanks for putting this together. > >> * Dual Queue Coupled AQM >> o With Curvy RED for Linux (access available shortly) >> o With PI2 for Linux [*UPDATED*] > I'll repeat my concerns that I already expressed at the L4S BOF in Berlin: > > While I agree that we probably need to separate low-delay congestion > control schemes from traditional "queue-filling" congestion schemes, > I strongly suggest to avoid putting a congestion control-specific > coupling scheme into the network (a classic case for applying the > "end-to-end arguments in system design"). > The current Dual queue coupled AQM proposal has got a coupling based on > a congestion control specific dropping law p_C=(p_L/2)². So if > congestion control schemes change then this coupling needs to be > adapted. For example, the currently proposed scheme may fail if that > vast majority of TCP traffic is using BBR other some other forthcoming > CC scheme instead of Cubic, Reno, Compound etc. The same applies to > draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping > likelihood is defined. > > Regards, > Roland > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------AD7E6DDACF9AC164C6863BEB Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit Roland,

I share your concern about cc-specific AQMs. But that is not a good characterization of what we're doing.

On the current Internet, everything is meant to be somewhat "friendly" to the original TCP cc (now spec'd in RFC5681). All sorts of cc's work alongside that, with slightly different "fairness" properties, and only one AQM is needed to cover them all. Nonetheless, Reno is the "lamest", so everyone has to try to "Do (not much) Harm" to the lamest.

Is any AQM CC-neutral?
Note rule 5 in the AQM Guidelines [RFC7567]
      "AQM algorithms SHOULD NOT interpret specific transport protocol behaviors."
In general, the advice in that section is sound, but I don't think we realized at that time just how subtle this issue is.

Since then, I discovered that the autotuning parameter table in the PIE algorithm is designed very precisely around the 1/sqrt(p) rule of Reno (see Fig 5 in the PI^2 paper). Similarly, the sqrt control law in Codel claims to be dependent on Reno {Note 1}.

The point is that these AQMs still work fine with Cubic, Compound, Westwood, etc, because all these ccs were designed to interwork with Reno. {Note 2}

The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID spec) is to enable a shift to a completely different "norm", but still coexist with all the 'Classic' cc's that coexisted around the old "norm". The new norm is intended to be just as fuzzy as the old norm {Note 3}. The idea is two fuzzy clouds of congestion controls, around an old and a new norm that are related together.

BBR
I believe BBR attempts to be 'friendly' to loss-based flows when competing in the same queue. But it's still research, and we don't yet know how good it is at that in all scenarios, although we do have code to test now. Given BBR currently sets Not-ECT, it would classify itself into the Classic queue of a DualQ AQM, and if it coexists with Reno it /should/ coexist with L4S traffic in the other queue. See Koen's recent posting about this.

There would be nothing to stop someone designing a variant of BBR that coexisted in the L4S queue with Scalable CCs like DCTCP (the point being that if the bottleneck was not DualQ it would keep delay low and if if the bottleneck was DualQ it would benefit even more from the lower queuing delay there). However, it would have to be a bit more careful about its whole round trip of queue probing, to avoid increasing the delay in the L4S queue. You'll see that I suggested to Neil Cardwell that they consider probing with a few packets rather than a whole window, e.g. the chirping technique that Mirja and I looked into back in 2010 was designed to find the same knee between rate increase and delay increase, with far fewer packets. I thought of a better way of using chirping a few weeks ago, so I will be returning to that too.

Specs
There is no statement that all L4S cc's MUST adhere to a 1/p rule. The L4S ID draft says:
  "The inverse proportionality requirement above is worded
   as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility
   when defining these specifications."

I hope that 'SHOULD' is fuzzy enough - I suspect adding more words would make it less fuzzy. But I would welcome wording to make it even more fuzzy if you would like to engage in wordsmithing.

Nonetheless, we are trying to steer a path between a rock and a hard place. Because, to shift to a much calmer waters beyond the rocks, we have to define some number to relate L4S to Classic. I am wary because when ECN was specified, there were attempts to define ECN as different to drop. However, ECN originally ended up "the same as drop" because no-one could muster enough backing behind any particular number to relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ).


Bob

{Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time.

{Note 2} Actually, I don't believe PIE would work that well with Cubic at v high BDP, once it was far from Reno-friendly, but that's only intuition from the stability analysis, not from actual testing.

{Note 3} Indeed, at the moment, when DCTCP is on its own in the L4S queue of the DualQ AQM as coded now, it hits up against a step threshold, which makes it behave as 1/p^2, not 1/p. For now, that's just because we didn't want to change too much about DCTCP at one time. But it's also got some nice properties. This will all need to be discussed as the DualQ AQM is specified more deeply.


On 21/11/16 14:27, Bless, Roland (TM) wrote:
Hi Bob and all,

see below.

Am 01.11.2016 um 01:02 schrieb Bob Briscoe:

A few people have been working away to specify and document all the
aspects of the new Low Latency, Low Loss, Scalable throughput (L4S)
service, which held a successful BoF in Berlin. As the decision was to
try to work across multiple WGs, I thought it would be useful to give ...

Thanks for putting this together.


  * Dual Queue Coupled AQM
      o With Curvy RED for Linux (access available shortly)
      o With PI2 for Linux <https://github.com/olgabo/dualpi2> [*UPDATED*]

I'll repeat my concerns that I already expressed at the L4S BOF in Berlin:

While I agree that we probably need to separate low-delay congestion
control schemes from traditional "queue-filling" congestion schemes,
I strongly suggest to avoid putting a congestion control-specific
coupling scheme into the network (a classic case for applying the
"end-to-end arguments in system design").
The current Dual queue coupled AQM proposal has got a coupling based on
a congestion control specific dropping law p_C=(p_L/2)². So if
congestion control schemes change then this coupling needs to be
adapted. For example, the currently proposed scheme may fail if that
vast majority of TCP traffic is using BBR other some other forthcoming
CC scheme instead of Cubic, Reno, Compound etc. The same applies to
draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping
likelihood is defined.

Regards,
 Roland



-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------AD7E6DDACF9AC164C6863BEB-- From nobody Tue Nov 22 12:37:46 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8492A129B72; Tue, 22 Nov 2016 12:37:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.45 X-Spam-Level: X-Spam-Status: No, score=-2.45 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3eBdAbwW3yhe; Tue, 22 Nov 2016 12:37:28 -0800 (PST) Received: from mail-qt0-x243.google.com (mail-qt0-x243.google.com [IPv6:2607:f8b0:400d:c0d::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7ED0F129795; Tue, 22 Nov 2016 12:37:28 -0800 (PST) Received: by mail-qt0-x243.google.com with SMTP id n6so4243626qtd.0; Tue, 22 Nov 2016 12:37:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=3SsoJplVkMmbgtJhxpx8FPieAh4oV/A4yIIM2kcOOYw=; b=tj1OVwjr6aCyPIlwJgCpfZykO2ZYnQpsiIt6yAtq5Y6Bd186XXIfAVNgxqY90+vWqY bv8mU5w2xXkYpm3dEd2t0h2iP9Q6r1z2/y+Y3YSb+6G4tC47tdz++Qx2Nv+uhEvKwSvD 7xAQWdUCQUnj5BKZLza0n6Gw4bklSnMhocxVdT55b+VaV27wKhBP++0Iq5XPFcztE2oZ 0RuXq3iAxGxpHUpI+VyCLLhjU6o6gnBrl4WBobRd5wHFS5dxAqu0kt6Xob2TcoSKXTWs BPkeHkDkrJDLFyrP8lcZ+YtHU1kgMWPwWV/T7Z+pm2xCFw8mIRtvi8wzBg7ztDOsDPJE fpYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=3SsoJplVkMmbgtJhxpx8FPieAh4oV/A4yIIM2kcOOYw=; b=KcmK+NEDK1vb18CYBhcJe30jYJQcn+HG+0xH5Ue3BfZ7A6nHZBPupGE8V8FcDEmqqI SqIsgIaf2/noQqVVvNXHrVDfq9hx8htcppgwfMLNTD2Hgm6hEKZNJP9STvPH9q9RCC+4 ZwUssGxSHbxyFXI138GhbH2TMvZBxiGm9Is+BkFZXA4UiEHme7+FoHa22hJ32/Bjs7EA wRIDFAVppS8d3RG5nTDne5mz8EFYmyt/5UIkI1d0grTHVFhv4MbnvjfXsVVgSOx2SrDz GuEHvbL5ibYtfmD1KpiR4vxo9n1fiaeliG55EwSYQz3O+qO5tiwQl3O2mnxg9xMHUZ4R Vx9A== X-Gm-Message-State: AKaTC01BxFTDj0BdAQu9pBU0p8cjPgOXhHfcjBmZ7c8y37/77jf3iOPHwwgXH12yi7siBg== X-Received: by 10.25.32.19 with SMTP id g19mr4977942lfg.162.1479847047464; Tue, 22 Nov 2016 12:37:27 -0800 (PST) Received: from [192.168.100.13] (37-33-82-144.bb.dnainternet.fi. [37.33.82.144]) by smtp.gmail.com with ESMTPSA id e78sm6616283lfg.23.2016.11.22.12.37.25 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 22 Nov 2016 12:37:26 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Jonathan Morton In-Reply-To: Date: Tue, 22 Nov 2016 22:37:20 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <053CE5B0-1879-4A2B-B4E9-8BCDDE5BA482@gmail.com> References: To: Bob Briscoe X-Mailer: Apple Mail (2.3124) Archived-At: Cc: TCP Prague List , tcpm IETF list , "Bless, Roland \(TM\)" , AQM IETF list , tsvwg IETF list Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 20:37:35 -0000 > On 22 Nov, 2016, at 21:09, Bob Briscoe wrote: >=20 > {Note 1} I have never got a good answer to my questions on aqm@ietf as = to why a sqrt that controls the shrinkage of the spacing between dropped = packets has something to do with the steady state law of Reno, = particularly because the law leads to linear growth in p over time. If you have intervals between events which follow a 1/sqrt(N) sequence, = where N is the number of preceding events, you get an event frequency = which increases linearly with time. This applies directly to Codel=92s = signalling strategy, which is to start at one mark per (assumed) RTT, = and to increase the marking frequency if that was insufficient to = control the queue. When you have multiple Reno flows sharing a single queue, there is a = sqrt(N) factor in several of the characteristics, where N is the number = of flows. When such a shared link becomes saturated, all of the flows = must be signalled to slow down, but for stochastic reasons it=92ll = probably take more than N signalling events to do so. Increasing the = signalling frequency while the queue remains insufficiently controlled = has a good chance of quickly finding all the flows, while dropping = relatively few packets (of non-ECN flows). I=92m reasonably convinced that Codel is a near-optimal solution to = congestion signalling on TCP-friendly flows. Regrettably, the latter = are not the only type of traffic actually found on the Internet. Really, the central assumption of Codel is that each flow requires only = one congestion signal event per RTT to cause it to back off. Codel = stops working well on traffic which doesn=92t obey that assumption (a = linear increase in drop frequency is inadequate to mitigate a flood - = you need to work with drop probabilities for that), but it *does* work = acceptably well with multiple flows sharing a queue, due to this = operating-point search. - Jonathan Morton From nobody Thu Nov 24 09:06:13 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 83BC3129489; Thu, 24 Nov 2016 09:06:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3] autolearn=unavailable autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Ve_rbaBVwfZ; Thu, 24 Nov 2016 09:06:10 -0800 (PST) Received: from iramx2.ira.uni-karlsruhe.de (iramx2.ira.uni-karlsruhe.de [141.3.10.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DC2F81295F2; Thu, 24 Nov 2016 08:57:17 -0800 (PST) Received: from i72t450mh.tm.uni-karlsruhe.de ([141.3.71.47]) by iramx2.ira.uni-karlsruhe.de with esmtpsa port 587 iface 141.3.10.81 id 1c9xKl-0000bW-Tj; Thu, 24 Nov 2016 17:57:15 +0100 To: Bob Briscoe , "Bless, Roland (TM)" References: From: Mario Hock Message-ID: <67730f0d-7ee4-1366-1aaf-847b73e6b146@kit.edu> Date: Thu, 24 Nov 2016 17:57:14 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-ATIS-AV: ClamAV (iramx2.ira.uni-karlsruhe.de) X-ATIS-Timestamp: iramx2.ira.uni-karlsruhe.de esmtpsa 1480006635. Archived-At: Cc: TCP Prague List , tcpm IETF list , AQM IETF list , tsvwg IETF list Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 17:06:11 -0000 Hello Bob and Roland, I followed your discussion and want to share my opinion, here. (Comments inline). Am 22.11.2016 um 20:09 schrieb Bob Briscoe: > *Is any AQM CC-neutral?** > *Note rule 5 in the > AQM Guidelines [RFC7567] > "AQM algorithms SHOULD NOT interpret specific transport protocol > behaviors." > In general, the advice in that section is sound, but I don't think we > realized at that time just how subtle this issue is. > > Since then, I discovered that the autotuning parameter table in the PIE > algorithm is designed very precisely around the 1/sqrt(p) rule of Reno > (see Fig 5 in the PI^2 paper ). > Similarly, the sqrt control law in Codel claims to be dependent on Reno > {Note 1}. > > The point is that these AQMs still work fine with Cubic, Compound, > Westwood, etc, because all these ccs were designed to interwork with > Reno. {Note 2} The reason that the AQMs also work with these CCs is because the CCs still have a lot in common with TCP Reno, like that they use a congestion window, that they increase the CWnd up until packet loss etc. Also the 1/sqrt(p)-law is still, more or less, applicable to them. For newer CCs like BBR or PCC, the 1/sqrt(p) does not apply. They are based i.a. on throughput vs. loss/delay probing. BRR, for example, does not even use a congestion window (at least not in the same way as the other CCs do). > The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID > spec) is to enable a shift to a completely different "norm", but still > coexist with all the 'Classic' cc's that coexisted around the old > "norm". The new norm is intended to be just as fuzzy as the old norm > {Note 3}. The idea is two fuzzy clouds of congestion controls, around an > old and a new norm that are related together. I agree with the general idea. But from that reasoning newer CCs, like BBR, belong into the "new" queue, where L4S lives in! If they don't belong in the same queue as L4S, then the queues should not be coupled in the way you propose. This coupling prevents any innovation in the non-L4S queue. > > *BBR** > *I believe BBR attempts to be 'friendly' to loss-based flows when > competing in the same queue. But it's still research, and we don't yet > know how good it is at that in all scenarios, although we do have code > to test now. Given BBR currently sets Not-ECT, it would classify itself > into the Classic queue of a DualQ AQM, and if it coexists with Reno it > /should/ coexist with L4S traffic in the other queue. See Koen's recent > posting > > about this. > > There would be nothing to stop someone designing a variant of BBR that > coexisted in the L4S queue with Scalable CCs like DCTCP (the point being > that if the bottleneck was not DualQ it would keep delay low and if if > the bottleneck was DualQ it would benefit even more from the lower > queuing delay there). However, it would have to be a bit more careful > about its whole round trip of queue probing, to avoid increasing the > delay in the L4S queue. You'll see that I suggested to Neil Cardwell > that they consider probing with a few packets rather than a whole > window, e.g. the chirping > technique that Mirja > and I looked into back in 2010 was designed to find the same knee > between rate increase and delay increase, with far fewer packets. I > thought of a better way of using chirping a few weeks ago, so I will be > returning to that too. I think L4S is not studied well enough, yet to be standardized in a way that other low delay approaches must adjust to the L4S concepts. There is at least BBR, PCC, nv (New Vegas) currently under development that all bring in fresh ideas how congestion control could be made in the future. Standardizing L4S as "the" new concept without intensively study other approaches is just too soon. If we want to act now, we should focus on a solution that enables new approaches in congestion control but is not tailored to either Reno/Cubic or L4S/DCTCP. Otherwise more research is required how we want the future of congestion control look like. Best regards, Mario Hock From nobody Thu Nov 24 12:48:15 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C23E9129CD5; Thu, 24 Nov 2016 12:48:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Iao9TOjApXZS; Thu, 24 Nov 2016 12:48:05 -0800 (PST) Received: from iramx2.ira.uni-karlsruhe.de (iramx2.ira.uni-karlsruhe.de [141.3.10.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D7D5C129CBD; Thu, 24 Nov 2016 12:48:04 -0800 (PST) Received: from i72vorta.tm.uni-karlsruhe.de ([141.3.71.26] helo=i72vorta.tm.kit.edu) by iramx2.ira.uni-karlsruhe.de with esmtp port 25 iface 141.3.10.81 id 1cA0w7-0006CM-EE; Thu, 24 Nov 2016 21:48:03 +0100 Received: from [IPv6:::1] (localhost [127.0.0.1]) by i72vorta.tm.kit.edu (Postfix) with ESMTPS id 26ABDB0033C; Thu, 24 Nov 2016 21:48:25 +0100 (CET) From: "Bless, Roland (TM)" To: Bob Briscoe References: Organization: Institute of Telematics, Karlsruhe Institute of Technology (KIT) Message-ID: <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> Date: Thu, 24 Nov 2016 21:48:02 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.1) Gecko/20060111 Thunderbird/1.5 Mnenhy/0.7.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-ATIS-AV: ClamAV (iramx2.ira.uni-karlsruhe.de) X-ATIS-Timestamp: iramx2.ira.uni-karlsruhe.de 1480020483. Archived-At: Cc: tcpm IETF list , AQM IETF list , tsvwg IETF list , TCP Prague List Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 20:48:08 -0000 Hi Bob, see comments inline, please. Am 22.11.2016 um 20:09 schrieb Bob Briscoe: > I share your concern about cc-specific AQMs. But that is not a good > characterization of what we're doing. Yep, but it's part of what you're proposing. > On the current Internet, everything is meant to be somewhat "friendly" > to the original TCP cc (now spec'd in RFC5681). All sorts of cc's work > alongside that, with slightly different "fairness" properties, and only > one AQM is needed to cover them all. Nonetheless, Reno is the "lamest", > so everyone has to try to "Do (not much) Harm" to the lamest. Yes, and that's actually a deployment problem we agree on. Congestion control (CC) innovation is obstructed by the old loss-based CC so some separation mechanism would be good to let room for low-delay CCs. > *Is any AQM CC-neutral?** > *Note rule 5 in the > AQM Guidelines [RFC7567] > "AQM algorithms SHOULD NOT interpret specific transport protocol > behaviors." > In general, the advice in that section is sound, but I don't think we > realized at that time just how subtle this issue is. I think that AQMs could be in fact designed CC neutral to a large extent, but every CC reacts probably different to the drop/ECN signals. So the dropping strategy implemented by the AQM will always cause a CC-specific reaction. Some AQMs therefore may achieve better results if they are tuned to specific CC behavior (but my e2e argument would also apply here), however, right now it's not that easy to change AQM implementations, esp. those implemented in hardware. > Since then, I discovered that the autotuning parameter table in the PIE > algorithm is designed very precisely around the 1/sqrt(p) rule of Reno > (see Fig 5 in the PI^2 paper ). > Similarly, the sqrt control law in Codel claims to be dependent on Reno > {Note 1}. > The point is that these AQMs still work fine with Cubic, Compound, > Westwood, etc, because all these ccs were designed to interwork with > Reno. {Note 2} > > The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID > spec) is to enable a shift to a completely different "norm", but still > coexist with all the 'Classic' cc's that coexisted around the old > "norm". The new norm is intended to be just as fuzzy as the old norm > {Note 3}. The idea is two fuzzy clouds of congestion controls, around an > old and a new norm that are related together. Yes, and I support that basic idea to introduce network support for separating flows with different CC schemes. However, I'd like to have a solution that doesn't build in a specific coupling law, otherwise we will end up with having either TCP friendly or DCTCP friendly CC schemes. > *BBR** > *I believe BBR attempts to be 'friendly' to loss-based flows when > competing in the same queue. But it's still research, and we don't yet > know how good it is at that in all scenarios, although we do have code I'm not sure how friendly/fair BBR is actually to other CCs, but L4S is also still research... > to test now. Given BBR currently sets Not-ECT, it would classify itself > into the Classic queue of a DualQ AQM, and if it coexists with Reno it > /should/ coexist with L4S traffic in the other queue. See Koen's recent > posting > > about this. Hmm, from what I understood so far BBR reacts differently to packet loss than Reno/Cubic etc. So that coupling by dropping probability may not work correctly in this case, because BBR will not react according to 1/sqrt(p) (cf. the FAQ from the BBR slide set presented at IETF97, https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf) > There would be nothing to stop someone designing a variant of BBR that > coexisted in the L4S queue with Scalable CCs like DCTCP (the point being > that if the bottleneck was not DualQ it would keep delay low and if if > the bottleneck was DualQ it would benefit even more from the lower > queuing delay there). However, it would have to be a bit more careful > about its whole round trip of queue probing, to avoid increasing the > delay in the L4S queue. You'll see that I suggested to Neil Cardwell > that they consider probing with a few packets rather than a whole > window, e.g. the chirping > technique that Mirja > and I looked into back in 2010 was designed to find the same knee > between rate increase and delay increase, with far fewer packets. I > thought of a better way of using chirping a few weeks ago, so I will be > returning to that too. That would be interesting... > *Specs** > *There is no statement that all L4S cc's MUST adhere to a 1/p rule. The > L4S ID draft says: > "The inverse proportionality requirement above is worded > as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility > when defining these specifications." I found two MUSTs in these contexts: draft-briscoe-tsvwg-aqm-dualq-coupled-00: In order to prevent starvation of Classic traffic by scalable L4S traffic (e.g. DCTCP) the drop probability of Classic traffic MUST be proportional to the square of the marking probability of L4S traffic, In other words, the power to which p_L is raised in Eqn. (1) MUST be 2. https://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-l4s-id-02 2.5: The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L). That is p_C ~= (p_L / k)^2 > I hope that 'SHOULD' is fuzzy enough - I suspect adding more words would > make it less fuzzy. But I would welcome wording to make it even more > fuzzy if you would like to engage in wordsmithing. > > Nonetheless, we are trying to steer a path between a rock and a hard > place. Because, to shift to a much calmer waters beyond the rocks, we > have to define some number to relate L4S to Classic. I am wary because > when ECN was specified, there were attempts to define ECN as different > to drop. However, ECN originally ended up "the same as drop" because > no-one could muster enough backing behind any particular number to > relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ). That's a different discussion. > Bob > > {Note 1} I have never got a good answer to my questions on aqm@ietf as > to why a sqrt that controls the shrinkage of the spacing between dropped > packets has something to do with the steady state law of Reno, > particularly because the law leads to linear growth in p over time. Yep. However, according to our experiments Codel's dropping law seems to achieve a quite reasonable loss desynchronization. > {Note 2} Actually, I don't believe PIE would work that well with Cubic > at v high BDP, once it was far from Reno-friendly, but that's only > intuition from the stability analysis, not from actual testing. We tested PIE at 10GB/s and it worked with Cubic similarly as at low speeds. > {Note 3} Indeed, at the moment, when DCTCP is on its own in the L4S > queue of the DualQ AQM as coded now, it hits up against a step > threshold, which makes it behave as 1/p^2, not 1/p. For now, that's just > because we didn't want to change too much about DCTCP at one time. But > it's also got some nice properties. This will all need to be discussed > as the DualQ AQM is specified more deeply. So probably it would be good to find out what is the common denominator of _new_ L4S-like CCs (not only considering DCTCP) and what is a common denominator for CCs in the other class. Regards, Roland > On 21/11/16 14:27, Bless, Roland (TM) wrote: >> Hi Bob and all, >> >> see below. >> >> Am 01.11.2016 um 01:02 schrieb Bob Briscoe: >>> A few people have been working away to specify and document all the >>> aspects of the new Low Latency, Low Loss, Scalable throughput (L4S) >>> service, which held a successful BoF in Berlin. As the decision was to >>> try to work across multiple WGs, I thought it would be useful to give ... >> Thanks for putting this together. >> >>> * Dual Queue Coupled AQM >>> o With Curvy RED for Linux (access available shortly) >>> o With PI2 for Linux [*UPDATED*] >> I'll repeat my concerns that I already expressed at the L4S BOF in Berlin: >> >> While I agree that we probably need to separate low-delay congestion >> control schemes from traditional "queue-filling" congestion schemes, >> I strongly suggest to avoid putting a congestion control-specific >> coupling scheme into the network (a classic case for applying the >> "end-to-end arguments in system design"). >> The current Dual queue coupled AQM proposal has got a coupling based on >> a congestion control specific dropping law p_C=(p_L/2)². So if >> congestion control schemes change then this coupling needs to be >> adapted. For example, the currently proposed scheme may fail if that >> vast majority of TCP traffic is using BBR other some other forthcoming >> CC scheme instead of Cubic, Reno, Compound etc. The same applies to >> draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping >> likelihood is defined. >> >> Regards, >> Roland From nobody Fri Nov 25 06:49:28 2016 Return-Path: X-Original-To: tcpprague@ietfa.amsl.com Delivered-To: tcpprague@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3BDAE12997C; Fri, 25 Nov 2016 06:49:15 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ME2Xv-rzNXSS; Fri, 25 Nov 2016 06:49:10 -0800 (PST) Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3B1BF129ED3; Fri, 25 Nov 2016 06:23:28 -0800 (PST) Received: from [31.185.252.113] (port=34505 helo=[192.168.0.3]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.87) (envelope-from ) id 1cAHPS-00044q-M1; Fri, 25 Nov 2016 14:23:26 +0000 To: Jonathan Morton References: <053CE5B0-1879-4A2B-B4E9-8BCDDE5BA482@gmail.com> From: Bob Briscoe Message-ID: Date: Fri, 25 Nov 2016 14:23:26 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <053CE5B0-1879-4A2B-B4E9-8BCDDE5BA482@gmail.com> Content-Type: multipart/alternative; boundary="------------CD4B2AF5D5C493A6434C123F" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.dnsblock1.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net Archived-At: Cc: tsvwg IETF list , tcpm IETF list , TCP Prague List , "Bless, Roland \(TM\)" , AQM IETF list Subject: Re: [tcpPrague] [aqm] L4S status update X-BeenThere: tcpprague@ietf.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Nov 2016 14:49:15 -0000 This is a multi-part message in MIME format. --------------CD4B2AF5D5C493A6434C123F Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Jonathan, On 22/11/16 20:37, Jonathan Morton wrote: >> On 22 Nov, 2016, at 21:09, Bob Briscoe wrote: >> >> {Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time. > If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time. Yes, I pointed that out originally . And Toke confirmed in response that it did indeed happen in practice. > This applies directly to Codel’s signalling strategy, which is to start at one mark per (assumed) RTT, and to increase the marking frequency if that was insufficient to control the queue. > > When you have multiple Reno flows sharing a single queue, there is a sqrt(N) factor in several of the characteristics, where N is the number of flows. When such a shared link becomes saturated, all of the flows must be signalled to slow down, but for stochastic reasons it’ll probably take more than N signalling events to do so. Increasing the signalling frequency while the queue remains insufficiently controlled has a good chance of quickly finding all the flows, while dropping relatively few packets (of non-ECN flows). Just throwing a square root in "somewhere", doesn't mean it is the correct "somewhere". A stated goal of the sqrt in the CoDel control law is to match the 1/sqrt(p) in TCP Reno's window formula. Quite aside from whether that is a correct goal, it isn't even doing that: * ACK-clocked load (number of flows, N) is proportional to 1/cwnd, ie. proportional to sqrt(p) [see 2nd para of section 4 of PI2 paper ] * because CoDel applies a sqrt to the interval between drops, it results in a linear increase in p with time (because the sqrt effectively gets squared - see the simple maths in my original posting about this ). So, the question is: "Why is a linear increase in p (starting from 0) good for controlling load from N flows, where N is proportional to sqrt(p)?" > > I’m reasonably convinced that Codel is a near-optimal solution to congestion signalling on TCP-friendly flows. The word "optimal" has a precise meaning. I think you mean simply "I am predisposed to CoDel." Whether the control law increases p linearly with time or by the sqrt, or by any function of time, is not the point anyway. Time is not the correct unit for this control law. The CoDel control law has no variables in it (other than the point at which it starts) that depend on any feature of the traffic. Once the CoDel control law starts, it just blindly increases until it reaches a high enough drop level to control the traffic. So the higher p needs to be, the longer it takes. And the lower p needs to be, the more it will overshoot within a round trip. A constant increase in p with time, and no dependency on the traffic is just plain wrong. No way will that result in anything that anyone could prove was "optimal" even if you put caveats around it like "near-optimal" or "reasonably convincingly near-optimal". Perhaps this helps you to see why claims of near-optimality say more about the political or religious zeal of the person making the claim, than they do about CoDel itself. > Regrettably, the latter are not the only type of traffic actually found on the Internet. > > Really, the central assumption of Codel is that each flow requires only one congestion signal event per RTT to cause it to back off. Codel stops working well on traffic which doesn’t obey that assumption (a linear increase in drop frequency is inadequate to mitigate a flood - you need to work with drop probabilities for that), but it *does* work acceptably well with multiple flows sharing a queue, due to this operating-point search. Similarly, isn't the phrase "acceptably well" another way of saying "We don't need to consider any other AQM that might be better at handling a wider range of load scenarios"? Even though there is already an alternative available that increases the drop level dependent on how fast the queue is growing. This religious belief in a particular technology is not healthy. Please can we have some objectivity here. Regards Bob > > - Jonathan Morton > > _______________________________________________ > tcpPrague mailing list > tcpPrague@ietf.org > https://www.ietf.org/mailman/listinfo/tcpprague -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------CD4B2AF5D5C493A6434C123F Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit Jonathan,

On 22/11/16 20:37, Jonathan Morton wrote:

      

        
On 22 Nov, 2016, at 21:09, Bob Briscoe <ietf@bobbriscoe.net> wrote:

{Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time.


      

      
If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time.  

    

    Yes, I pointed that out originally.
    And Toke confirmed in response that it did indeed happen in
    practice.

    

      
This applies directly to Codel’s signalling strategy, which is to start at one mark per (assumed) RTT, and to increase the marking frequency if that was insufficient to control the queue.

When you have multiple Reno flows sharing a single queue, there is a sqrt(N) factor in several of the characteristics, where N is the number of flows.  When such a shared link becomes saturated, all of the flows must be signalled to slow down, but for stochastic reasons it’ll probably take more than N signalling events to do so.  Increasing the signalling frequency while the queue remains insufficiently controlled has a good chance of quickly finding all the flows, while dropping relatively few packets (of non-ECN flows).

    

    Just throwing a square root in "somewhere", doesn't mean it is the
    correct "somewhere". 

    

    A stated goal of the sqrt in the CoDel control law is to match the
    1/sqrt(p) in TCP Reno's window formula. Quite aside from whether
    that is a correct goal, it isn't even doing that:

    * ACK-clocked load (number of flows, N) is proportional to 1/cwnd,
    ie. proportional to sqrt(p) [see 2nd para of section 4 of PI2 paper]

    * because CoDel applies a sqrt to the interval between drops, it
    results in a linear increase in p with time (because the sqrt
    effectively gets squared - see the simple maths in my
      original posting about this).

    

    So, the question is: "Why is a linear increase in p (starting from
    0) good for controlling load from N flows, where N is proportional
    to sqrt(p)?"

    

    

      

I’m reasonably convinced that Codel is a near-optimal solution to congestion signalling on TCP-friendly flows.  

    

    The word "optimal" has a precise meaning. I think you mean simply "I
    am predisposed to CoDel."

    

    Whether the control law increases p linearly with time or by the
    sqrt, or by any function of time, is not the point anyway. Time is
    not the correct unit for this control law. The CoDel control law has
    no variables in it (other than the point at which it starts) that
    depend on any feature of the traffic. Once the CoDel control law
    starts, it just blindly increases until it reaches a high enough
    drop level to control the traffic. So the higher p needs to be, the
    longer it takes. And the lower p needs to be, the more it will
    overshoot within a round trip.

    

    A constant increase in p with time, and no dependency on the traffic
    is just plain wrong.

    

    No way will that result in anything that anyone could prove was
    "optimal" even if you put caveats around it like "near-optimal" or
    "reasonably convincingly near-optimal".

    

    Perhaps this helps you to see why claims of near-optimality say more
    about the political or religious zeal of the person making the
    claim, than they do about CoDel itself.

    

    

      
Regrettably, the latter are not the only type of traffic actually found on the Internet.

Really, the central assumption of Codel is that each flow requires only one congestion signal event per RTT to cause it to back off.  Codel stops working well on traffic which doesn’t obey that assumption (a linear increase in drop frequency is inadequate to mitigate a flood - you need to work with drop probabilities for that), but it *does* work acceptably well with multiple flows sharing a queue, due to this operating-point search.

    

    Similarly, isn't the phrase "acceptably well" another way of saying
    "We don't need to consider any other AQM that might be better at
    handling a wider range of load scenarios"? Even though there is
    already an alternative available that increases the drop level
    dependent on how fast the queue is growing.

    

    This religious belief in a particular technology is not healthy.
    Please can we have some objectivity here.

    

    Regards

    

    

    

    Bob

    

      

 - Jonathan Morton

_______________________________________________
tcpPrague mailing list
tcpPrague@ietf.org
https://www.ietf.org/mailman/listinfo/tcpprague


    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------CD4B2AF5D5C493A6434C123F--


From nobody Fri Nov 25 10:34:31 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7DCAE1296BE; Fri, 25 Nov 2016 10:34:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.902
X-Spam-Level: 
X-Spam-Status: No, score=-6.902 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AWID__NAOo8v; Fri, 25 Nov 2016 10:34:20 -0800 (PST)
Received: from smtp-fr.alcatel-lucent.com (fr-hpida-esg-02.alcatel-lucent.com [135.245.210.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 01BF61296A9; Fri, 25 Nov 2016 10:34:19 -0800 (PST)
Received: from fr712umx4.dmz.alcatel-lucent.com (unknown [135.245.210.45]) by Websense Email Security Gateway with ESMTPS id 42AA2550D022B; Fri, 25 Nov 2016 18:34:14 +0000 (GMT)
Received: from fr712usmtp2.zeu.alcatel-lucent.com (fr712usmtp2.zeu.alcatel-lucent.com [135.239.2.42]) by fr712umx4.dmz.alcatel-lucent.com (GMO-o) with ESMTP id uAPIYE1G016438 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 25 Nov 2016 18:34:17 GMT
Received: from FR711WXCHHUB01.zeu.alcatel-lucent.com (fr711wxchhub01.zeu.alcatel-lucent.com [135.239.2.111]) by fr712usmtp2.zeu.alcatel-lucent.com (GMO) with ESMTP id uAPIYBSA013555 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Fri, 25 Nov 2016 18:34:11 GMT
Received: from FR711WXCHMBA05.zeu.alcatel-lucent.com ([169.254.1.65]) by FR711WXCHHUB01.zeu.alcatel-lucent.com ([135.239.2.111]) with mapi id 14.03.0301.000; Fri, 25 Nov 2016 19:34:11 +0100
From: "De Schepper, Koen (Nokia - BE)" 
To: "Bless, Roland (TM)" , Ingemar Johansson S , Bob Briscoe , =?iso-8859-1?Q?Dave_T=E4ht?= , tsvwg IETF list , TCP Prague List , AQM IETF list , tcpm IETF list 
Thread-Topic: [aqm] [tcpPrague]  L4S status update
Thread-Index: AQHSRN/m0o5ivGe0/ES9p3WGLO/u/KDp+ZvA
Date: Fri, 25 Nov 2016 18:34:10 +0000
Message-ID: 
References:    <70104ef0-3747-9b47-dcb7-be54fe454443@kit.edu>
In-Reply-To: <70104ef0-3747-9b47-dcb7-be54fe454443@kit.edu>
Accept-Language: nl-BE, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [135.239.27.39]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpPrague] [aqm]   L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 25 Nov 2016 18:34:22 -0000

Hi Roland, Ingemar, Bob, Dave,

I see congestion control mainly as an inter-flow-protocol that regulates ho=
w to share the bandwidth. Up to now it was agreed to adapt the rate to 1/sq=
rt(p).=20

Any congestion control deviating from this rule has been proven to fail. Ig=
noring drop will get you either pushed away, or you push away other traffic=
. I think most people agree with this?

Also BBR will face the same problem if it keeps on ignoring drop. I tried a=
 few long BBR flows next to long Cubic flows on our L4S testbed and current=
ly (depending on the RTT, rate and queue size) it either starves itself, st=
arves Cubic, or oscillate between the 2 previous cases over 20 second inter=
vals. I also tried only BBR flows on a PIE AQM. Same story there, either hi=
gh drop (15%), no drop or oscillations, also depending on the same paramete=
rs. Try it yourself and see... I see a lot of value in the other BBR mechan=
isms though, but not the drop ignoring aspects.

I don't see solutions without network support when new congestion controls =
ignore drop and still need to support other tcp flows which are deployed no=
w...... Up to now there are at least 2 network supported solutions:  DualQ =
Coupling that is an inter-protocol-translator (and therefor needs to know b=
oth protocols) and complete flow isolation like FQ which is the inter-flow-=
protocol inhibitor (for all flows) and takes over the end-systems role to s=
hare the BW (so not really according to the end-to-end principle).

For L4S, our main goal should be to define a new inter-flow-protocol betwee=
n L4S flows (L4S drafts assumes rate adapted based on ect(1) marking probab=
ility). Ones this is decided, we can couple it to the Classic drop/mark law=
. The network needs to support the coupling, as it is the only one that kno=
ws the drop and the marking of both classes. It depends on the end-systems,=
 but it is according to the end-to-end principle, because it is the least a=
nd most simple role that the network can have.

If a delay based protocol can be defined (like rate =3D F(queue delay)? ) t=
hen it could also be coupled to a classic drop probability with network sup=
port. But I don't think there is a proposal available. Also I don't think a=
 delay based protocol can achieve the same low latency results as an explic=
it signal.

Koen.


> -----Original Message-----
> From: aqm [mailto:aqm-bounces@ietf.org] On Behalf Of Bless, Roland (TM)
> Sent: dinsdag 22 november 2016 17:46
> To: Ingemar Johansson S ; Bob Briscoe
> ; tsvwg IETF list ; TCP Prague List
> ; AQM IETF list ; tcpm IETF list
> 
> Subject: Re: [aqm] [tcpPrague] L4S status update
>=20
> Hi Ingemar,
>=20
> my point was that it's probably better to refrain from
> building CC-specific behavior into network elements as the
> CC algorithms may evolve faster and in more flexible ways
> than we can foresee. Thus, it would be good to have a separation
> (or coupling) scheme that actually doesn't depend on the
> 1/sqrt(p) dropping law.
>=20
> Am 22.11.2016 um 15:35 schrieb Ingemar Johansson S:
> > As regards to comments around other new congestion control algorithms
> > and that they may need adapted dropping likelihood relation between a
> > classic queue and L4S queue. I have not tried out but I suspect that
> > BBR may get an unfair treatment, at the same time it is possible that
> > other delay based CCs may suffer.
>=20
> It would be interesting to see what happens if BBR is sorted into
> the L4S queue in comparison to what happens if BBR is sorted into
> the classic queue (BBR isn't reacting to loss according to 1/sqrt(p)).
>=20
> > They question however if this is a major problem?, one may as well
> > see this as an incentive to switch over to scalable congestion
> > controls and L4S ? There is after all no requirement to stick to a
> > particular congestion control no matter what. ?
>=20
> Yes, and that's why I find that built-in coupling law too limiting.
>=20
> > The question whether or not endpoint dependency should be built into
> > the networks is of course  a valid question but given that the state
> > of the art congestion controls like Reno and Cubic rely on a
> > 1/sqrt(p) function then that is perhaps OK ?. There will for a
> > foreseeable time come updated endpoint based congestion control
> > algorithms that are optimized  for one thing or the other (I am
> > guilty too :-). However if one can distinguish between two classes
> > (classic and L4S) where classic belong to the 1/sqrt(p) family then I
> > believe that it is possible to solve the problem. If we try find a
> > solution where classic =3D "all sorts of non-scalable non-L4S dependent
> > congestion controls" then I believe that we easily end up in big
> > problems.
>=20
> I'm not sure. Maybe we have a class of "queue-filling" CCs and
> a class of low-delay targeted CCs.
>=20
> Regards,
>  Roland
>=20
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm


From nobody Mon Nov 28 15:07:54 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6CC0812A088; Mon, 28 Nov 2016 15:07:44 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level: 
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KJ3ly8AfwlBt; Mon, 28 Nov 2016 15:07:41 -0800 (PST)
Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 55AD312A0A3; Mon, 28 Nov 2016 15:07:37 -0800 (PST)
Received: from cm-84.210.215.55.getinternet.no ([84.210.215.55]:53898 helo=[192.168.0.129]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.87) (envelope-from ) id 1cBV1L-0005j5-Dw; Mon, 28 Nov 2016 23:07:35 +0000
To: Mario Hock , "Bless, Roland (TM)" 
References:    <67730f0d-7ee4-1366-1aaf-847b73e6b146@kit.edu>
From: Bob Briscoe 
Message-ID: <88f008a0-cd51-9a4b-7d79-c5824ffc5472@bobbriscoe.net>
Date: Mon, 28 Nov 2016 23:07:34 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <67730f0d-7ee4-1366-1aaf-847b73e6b146@kit.edu>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.dnsblock1.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net
Archived-At: 
Cc: tsvwg IETF list , tcpm IETF list , TCP Prague List , AQM IETF list 
Subject: Re: [tcpPrague] [aqm]  L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 28 Nov 2016 23:07:44 -0000

Mario,

On 24/11/16 16:57, Mario Hock wrote:
> Hello Bob and Roland,
>
> I followed your discussion and want to share my opinion, here. 
> (Comments inline).
>
> Am 22.11.2016 um 20:09 schrieb Bob Briscoe:
>> *Is any AQM CC-neutral?**
>> *Note rule 5  in the
>> AQM Guidelines [RFC7567]
>>       "AQM algorithms SHOULD NOT interpret specific transport protocol
>> behaviors."
>> In general, the advice in that section is sound, but I don't think we
>> realized at that time just how subtle this issue is.
>>
>> Since then, I discovered that the autotuning parameter table in the PIE
>> algorithm is designed very precisely around the 1/sqrt(p) rule of Reno
>> (see Fig 5 in the PI^2 paper ).
>> Similarly, the sqrt control law in Codel claims to be dependent on Reno
>> {Note 1}.
>>
>> The point is that these AQMs still work fine with Cubic, Compound,
>> Westwood, etc, because all these ccs were designed to interwork with
>> Reno. {Note 2}
>
> The reason that the AQMs also work with these CCs is because the CCs 
> still have a lot in common with TCP Reno, like that they use a 
> congestion window, that they increase the CWnd up until packet loss 
> etc. Also the 1/sqrt(p)-law is still, more or less, applicable to them.
>
> For newer CCs like BBR or PCC, the 1/sqrt(p) does not apply. They are 
> based i.a. on throughput vs. loss/delay probing. BRR, for example, 
> does not even use a congestion window (at least not in the same way as 
> the other CCs do).
>
>
>> The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID
>> spec) is to enable a shift to a completely different "norm", but still
>> coexist with all the 'Classic' cc's that coexisted around the old
>> "norm". The new norm is intended to be just as fuzzy as the old norm
>> {Note 3}. The idea is two fuzzy clouds of congestion controls, around an
>> old and a new norm that are related together.
>
> I agree with the general idea. But from that reasoning newer CCs, like 
> BBR, belong into the "new" queue, where L4S lives in!
[BB] That's not for you (or I) to say.

The designer of a new CC can choose to innovate within Classic queues or 
L4S. Whichever they choose, they would set the identifier that 
classifies their packets into the appropriate queue, and they would have 
to coexist with the other behaviours in that queue.

We defined L4S as a framework to encourage innovation with new CCs in 
the L4S queue (because it's easier to deploy very nice properties), but 
please don't assume we were trying to mandate that! Also, if considering 
defining a CC with respect to L4S, bear in mind that L4S is only aiming 
for experimental status in the IETF at present.

L4S is not defined as "anything new must go here".


>
> If they don't belong in the same queue as L4S, then the queues should 
> not be coupled in the way you propose. This coupling prevents any 
> innovation in the non-L4S queue.
[BB] Not at all.

>
>
>>
>> *BBR**
>> *I believe BBR attempts to be 'friendly' to loss-based flows when
>> competing in the same queue. But it's still research, and we don't yet
>> know how good it is at that in all scenarios, although we do have code
>> to test now. Given BBR currently sets Not-ECT, it would classify itself
>> into the Classic queue of a DualQ AQM, and if it coexists with Reno it
>> /should/ coexist with L4S traffic in the other queue. See Koen's recent
>> posting
>> 
>> about this.
>>
>> There would be nothing to stop someone designing a variant of BBR that
>> coexisted in the L4S queue with Scalable CCs like DCTCP (the point being
>> that if the bottleneck was not DualQ it would keep delay low and if if
>> the bottleneck was DualQ it would benefit even more from the lower
>> queuing delay there). However, it would have to be a bit more careful
>> about its whole round trip of queue probing, to avoid increasing the
>> delay in the L4S queue. You'll see that I suggested to Neil Cardwell
>> that they consider probing with a few packets rather than a whole
>> window, e.g. the chirping
>>  technique that Mirja
>> and I looked into back in 2010 was designed to find the same knee
>> between rate increase and delay increase, with far fewer packets. I
>> thought of a better way of using chirping a few weeks ago, so I will be
>> returning to that too.
>
> I think L4S is not studied well enough, yet to be standardized in a 
> way that other low delay approaches must adjust to the L4S concepts.
[BB] They don't have to. L4S is aiming for experimental status at 
present. Strictly, other low delay experimental protocols only have to 
prove they coexist with standards track CCs (ie. Reno). In practice, any 
new experimental CC also has to prove it co-exists with the deployed 
base, whether standards track or not (e.g. Cubic, Compound etc). If L4S 
becomes part of the deployed base, it will have got there on its own 
merits, then other experiments that come along later will certainly have 
to interwork.

But anyway, as explained above, this should be no more difficult than 
interworking with Reno in the Classic queue, as if L4S was not there. 
Then interworking with L4S should come for free.

>
> There is at least BBR, PCC, nv (New Vegas) currently under development 
> that all bring in fresh ideas how congestion control could be made in 
> the future. Standardizing L4S as "the" new concept without intensively 
> study other approaches is just too soon.
[BB] I hope I've convinced you that L4S doesn't preclude any of these.

L4S is certainly different in that it is not a CC, rather it is a new 
space for easier CC experimentation. Another difference from all the CCs 
you mention above was the goal of persuading network operators of the 
benefit of creating such a space; because L4S is about improving the 
information flow between network and hosts. But none of that stops other 
CCs using the old space for experimentation.

>
> If we want to act now, we should focus on a solution that enables new 
> approaches in congestion control but is not tailored to either 
> Reno/Cubic or L4S/DCTCP. Otherwise more research is required how we 
> want the future of congestion control look like.
[BB] No-one is stopping you proposing something concrete.

You will, however, need an implementation, and proof that app 
performance has significant benefits, and a group of people interested 
in working on the idea, and .... This all takes work and time. Work on 
what has become L4S first started in 2012, see:
1) Wu, H., Ju, J., Lu, G., Guo, C., Xiong, Y. & Zhang, Y., "Tuning ECN 
for Data Center Networks," In: Proceedings of the 8th International 
Conference on Emerging Networking Experiments and Technologies CoNEXT 
'12 pp.25-36 ACM (2012)
2) http://www.bobbriscoe.net/present.html#1207tsvarea-dctcp

The choice of 1/p as a definition of the new L4S experimentation space 
was not arbitrary. There are numerous benefits to the number of 
congestion signals per RTT being * much higher and
* invariant with flow rate.
You will see further research that exploits these properties soon...

Cheers


Bob

>
> Best regards,
>
> Mario Hock
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


From nobody Mon Nov 28 16:45:37 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EB7A212A1E1; Mon, 28 Nov 2016 16:45:31 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level: 
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pvwLcoNquzcB; Mon, 28 Nov 2016 16:45:29 -0800 (PST)
Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0B69D12A1DF; Mon, 28 Nov 2016 16:45:29 -0800 (PST)
Received: from cm-84.210.215.55.getinternet.no ([84.210.215.55]:54067 helo=[192.168.0.129]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.87) (envelope-from ) id 1cBWY3-0004aH-BF; Tue, 29 Nov 2016 00:45:27 +0000
To: "Bless, Roland (TM)" 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu>
From: Bob Briscoe 
Message-ID: <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>
Date: Tue, 29 Nov 2016 00:45:26 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.dnsblock1.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net
Archived-At: 
Cc: TCP Prague List , tcpm IETF list , AQM IETF list , tsvwg IETF list 
Subject: Re: [tcpPrague] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 29 Nov 2016 00:45:32 -0000

Roland,

On 24/11/16 20:48, Bless, Roland (TM) wrote:
> Hi Bob,
>
> see comments inline, please.
>
> Am 22.11.2016 um 20:09 schrieb Bob Briscoe:
>> I share your concern about cc-specific AQMs. But that is not a good
>> characterization of what we're doing.
> Yep, but it's part of what you're proposing.
>
>> On the current Internet, everything is meant to be somewhat "friendly"
>> to the original TCP cc (now spec'd in RFC5681). All sorts of cc's work
>> alongside that, with slightly different "fairness" properties, and only
>> one AQM is needed to cover them all. Nonetheless, Reno is the "lamest",
>> so everyone has to try to "Do (not much) Harm" to the lamest.
> Yes, and that's actually a deployment problem we agree on. Congestion
> control (CC) innovation is obstructed by the old loss-based CC so some
> separation mechanism would be good to let room for low-delay CCs.
>
>> *Is any AQM CC-neutral?**
>> *Note rule 5  in the
>> AQM Guidelines [RFC7567]
>>        "AQM algorithms SHOULD NOT interpret specific transport protocol
>> behaviors."
>> In general, the advice in that section is sound, but I don't think we
>> realized at that time just how subtle this issue is.
> I think that AQMs could be in fact designed CC neutral to a large
> extent, but every CC reacts probably different to the drop/ECN signals.
> So the dropping strategy implemented by the AQM will always cause a
> CC-specific reaction. Some AQMs therefore may achieve better results if
> they are tuned to specific CC behavior (but my e2e argument would also
> apply here), however, right now it's not that easy to change AQM
> implementations, esp. those implemented in hardware.
I am particularly worried about embedding fq in the Internet. That is 
far worse than embedding a subtly different performance improvement for 
certain congestion controls. With fq, the network determines the precise 
departure time of each packet, completely overriding the host's choice, 
without any understanding of what the applications is trying to achieve.

Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded 
into the Linux WiFi drivers, without even a framework to dynamically 
load any alternative(s). Of course, we can add such a framework, but we 
are seeing Linux become the next major middlebox problem. It might be 
excusable if there were not sound alternatives available,... but there are.
>
>> Since then, I discovered that the autotuning parameter table in the PIE
>> algorithm is designed very precisely around the 1/sqrt(p) rule of Reno
>> (see Fig 5 in the PI^2 paper ).
>> Similarly, the sqrt control law in Codel claims to be dependent on Reno
>> {Note 1}.
>> The point is that these AQMs still work fine with Cubic, Compound,
>> Westwood, etc, because all these ccs were designed to interwork with
>> Reno. {Note 2}
>>
>> The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID
>> spec) is to enable a shift to a completely different "norm", but still
>> coexist with all the 'Classic' cc's that coexisted around the old
>> "norm". The new norm is intended to be just as fuzzy as the old norm
>> {Note 3}. The idea is two fuzzy clouds of congestion controls, around an
>> old and a new norm that are related together.
> Yes, and I support that basic idea to introduce network support for
> separating flows with different CC schemes. However, I'd like to have a
> solution that doesn't build in a specific coupling law, otherwise we
> will end up with having either TCP friendly or DCTCP friendly CC schemes.
And your proposal is...?

Also, what exactly is your rationale for not wanting a coupling law? It 
feels nice not to tie anything down. However, when hosts have fuzziness 
around both norms, additional fuzziness between the norms would just add 
an unnecessary dimension of uncertainty for no benefit.


As I just explained to Mario, the choice of "inversely proportional to 
marking probability" as the principle behind the new L4S space was not 
arbitrary.

     Congestion signals per round trip = probability of congestion 
signal * packets per round trip
        = p * W
where W is the window, and p is marking probability.

L4S is defined for
     W ~= k/p
where k is a constant.

So, L4S Congestion signals per round trip = p * W
                                                                  ~= k

That is, L4S is defined so that congestion signals per round trip is 
invariant as flow rate scales. That's an extremely important property..


>
>> *BBR**
>> *I believe BBR attempts to be 'friendly' to loss-based flows when
>> competing in the same queue. But it's still research, and we don't yet
>> know how good it is at that in all scenarios, although we do have code
> I'm not sure how friendly/fair BBR is actually to other CCs,
See Koen's recent post.
> but L4S is also still research...
Well, L4S is more a space in which research can flourish. And also a 
space where there's an initial CC (DCTCP) already deployed in 3 OSs, 
with a lot of deployment experience in controlled environments, and it's 
showing pretty cool results over the Internet, even though DCTCP wasn't 
designed for the Internet.

I call L4S and incrementally deployable clean-slate.
>
>> to test now. Given BBR currently sets Not-ECT, it would classify itself
>> into the Classic queue of a DualQ AQM, and if it coexists with Reno it
>> /should/ coexist with L4S traffic in the other queue. See Koen's recent
>> posting
>> 
>> about this.
> Hmm, from what I understood so far BBR reacts differently to packet loss
> than Reno/Cubic etc. So that coupling by dropping probability may not
> work correctly in this case, because BBR will not react according to
> 1/sqrt(p) (cf. the FAQ from the BBR slide set presented at IETF97,
> https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf)
Well, it's BBR's problem to show that it can interwork with the existing 
Internet first (and I am not including L4S in "the existing Internet" yet).
>
>> There would be nothing to stop someone designing a variant of BBR that
>> coexisted in the L4S queue with Scalable CCs like DCTCP (the point being
>> that if the bottleneck was not DualQ it would keep delay low and if if
>> the bottleneck was DualQ it would benefit even more from the lower
>> queuing delay there). However, it would have to be a bit more careful
>> about its whole round trip of queue probing, to avoid increasing the
>> delay in the L4S queue. You'll see that I suggested to Neil Cardwell
>> that they consider probing with a few packets rather than a whole
>> window, e.g. the chirping
>>  technique that Mirja
>> and I looked into back in 2010 was designed to find the same knee
>> between rate increase and delay increase, with far fewer packets. I
>> thought of a better way of using chirping a few weeks ago, so I will be
>> returning to that too.
> That would be interesting...
>
>> *Specs**
>> *There is no statement that all L4S cc's MUST adhere to a 1/p rule. The
>> L4S ID draft says:
>>    "The inverse proportionality requirement above is worded
>>     as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility
>>     when defining these specifications."
> I found two MUSTs in these contexts:
>
> draft-briscoe-tsvwg-aqm-dualq-coupled-00:
>     In order to prevent
>     starvation of Classic traffic by scalable L4S traffic (e.g.  DCTCP)
>     the drop probability of Classic traffic MUST be proportional to the
>     square of the marking probability of L4S traffic, In other words, the
>     power to which p_L is raised in Eqn. (1) MUST be 2.
>
> https://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-l4s-id-02
> 2.5:
>     The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
>     be roughly proportional to the square of the likelihood that it would
>     have marked it if it had been an L4S packet (p_L).  That is
>
>        p_C ~= (p_L / k)^2
They are deliberate MUSTs, linking together the meaning of the two 
congestion signals that a network queue applies. Neither constrains 
hosts; the SHOULD in the host behaviour is deliberately more liberal to 
hosts.

The network is given freedom to flex the constant of proportionality (k 
in the above equation), but not the long-term scaling exponent (the 
square). That allows networks and equipment vendors to differentiate 
themselves, without compromising long term scaling.

Given hosts have flex around both "norms", if the network flexed the 
scaling relationship between the "norms" as well, it would add another 
dimension of uncertainty to the system, with no benefit.

>
>> I hope that 'SHOULD' is fuzzy enough - I suspect adding more words would
>> make it less fuzzy. But I would welcome wording to make it even more
>> fuzzy if you would like to engage in wordsmithing.
>>
>> Nonetheless, we are trying to steer a path between a rock and a hard
>> place. Because, to shift to a much calmer waters beyond the rocks, we
>> have to define some number to relate L4S to Classic. I am wary because
>> when ECN was specified, there were attempts to define ECN as different
>> to drop. However, ECN originally ended up "the same as drop" because
>> no-one could muster enough backing behind any particular number to
>> relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ).
> That's a different discussion.
Well, no. Think of it as push-back against your earlier sentence: "I'd 
like to have a solution that doesn't build in a specific coupling law".

We are proposing the square coupling law at the IETF precisely because, 
without a standard, no-one would be able to design a CC and be able to 
say anything about how it shares out capacity.
>
>> Bob
>>
>> {Note 1} I have never got a good answer to my questions on aqm@ietf as
>> to why a sqrt that controls the shrinkage of the spacing between dropped
>> packets has something to do with the steady state law of Reno,
>> particularly because the law leads to linear growth in p over time.
> Yep. However, according to our experiments Codel's dropping law seems to
> achieve a quite reasonable loss desynchronization.
>
>> {Note 2} Actually, I don't believe PIE would work that well with Cubic
>> at v high BDP, once it was far from Reno-friendly, but that's only
>> intuition from the stability analysis, not from actual testing.
> We tested PIE at 10GB/s and it worked with Cubic similarly as at
> low speeds.
OK.
>
>> {Note 3} Indeed, at the moment, when DCTCP is on its own in the L4S
>> queue of the DualQ AQM as coded now, it hits up against a step
>> threshold, which makes it behave as 1/p^2, not 1/p. For now, that's just
>> because we didn't want to change too much about DCTCP at one time. But
>> it's also got some nice properties. This will all need to be discussed
>> as the DualQ AQM is specified more deeply.
> So probably it would be good to find out what is the common denominator
> of _new_ L4S-like CCs (not only considering DCTCP) and what is a common
> denominator for CCs in the other class.
The text proposal to answer this is in the ecn-l4s-id draft.

Pls feel free to suggest specific text improvements.

Cheers




Bob
>
> Regards,
>   Roland
>
>
>> On 21/11/16 14:27, Bless, Roland (TM) wrote:
>>> Hi Bob and all,
>>>
>>> see below.
>>>
>>> Am 01.11.2016 um 01:02 schrieb Bob Briscoe:
>>>> A few people have been working away to specify and document all the
>>>> aspects of the new Low Latency, Low Loss, Scalable throughput (L4S)
>>>> service, which held a successful BoF in Berlin. As the decision was to
>>>> try to work across multiple WGs, I thought it would be useful to give ...
>>> Thanks for putting this together.
>>>
>>>>    * Dual Queue Coupled AQM
>>>>        o With Curvy RED for Linux (access available shortly)
>>>>        o With PI2 for Linux  [*UPDATED*]
>>> I'll repeat my concerns that I already expressed at the L4S BOF in Berlin:
>>>
>>> While I agree that we probably need to separate low-delay congestion
>>> control schemes from traditional "queue-filling" congestion schemes,
>>> I strongly suggest to avoid putting a congestion control-specific
>>> coupling scheme into the network (a classic case for applying the
>>> "end-to-end arguments in system design").
>>> The current Dual queue coupled AQM proposal has got a coupling based on
>>> a congestion control specific dropping law p_C=(p_L/2)². So if
>>> congestion control schemes change then this coupling needs to be
>>> adapted. For example, the currently proposed scheme may fail if that
>>> vast majority of TCP traffic is using BBR other some other forthcoming
>>> CC scheme instead of Cubic, Reno, Compound etc. The same applies to
>>> draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping
>>> likelihood is defined.
>>>
>>> Regards,
>>>   Roland
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


From nobody Mon Nov 28 17:20:39 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 65E2B12A21E; Mon, 28 Nov 2016 17:20:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.45
X-Spam-Level: 
X-Spam-Status: No, score=-2.45 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WAG6i-weNPMo; Mon, 28 Nov 2016 17:20:35 -0800 (PST)
Received: from mail-lf0-x242.google.com (mail-lf0-x242.google.com [IPv6:2a00:1450:4010:c07::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 93F6912A221; Mon, 28 Nov 2016 17:20:34 -0800 (PST)
Received: by mail-lf0-x242.google.com with SMTP id o141so11210007lff.1; Mon, 28 Nov 2016 17:20:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;  h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=OiOmfyqcJqsOd825itUq4zUfXsbVCUWq74Rb8yPGjVk=; b=gU/5NdxvG2Pw4+1eelBOjcIotfXj410rBdKML6eF0JEFvAQmzV8fYXOAcw4dpT/0Or ehc/bEVgRAv62le03ZhIO4grTn3EB/Er3KwWA9vMBgdUwdHbLudYQPikcWMPL0yOq/GL P36wVjCk52k2SPjY6D9tN8uo+c1tEZgT9XOwy2y8luDaMFTu3Hl2r0aXwha8XAfm85Ad xln/tqDyrlXBkOBfewUJ+2A5n6qxKdc3hfOJCT0zkdtG1jR6ydIxiPYfG2w/spWZoQ+R igmvCeaNDrPU045z/2axdIHLyW6uK8x6fS/BF5zAoyEPfbyT4Tvyd1zbvqXMHRnw5LW4 RCKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=OiOmfyqcJqsOd825itUq4zUfXsbVCUWq74Rb8yPGjVk=; b=KwlZmpMe8Xo0x4wBwVsf4T5mMbgffW94uraYkSlYJKxC4wFKR5XldXSAokEDuoVE/8 Ebv+6zT7ZPbBohNcKpQwhrotnReQWPKGDbWspqvkk3yokycPCpxgT8ScyhjFmgi4oVrw GBiquVOun3IxQIHHUnOyVBczlFLogvzootOat9qY4T0aVpfip4vf9dF2bmkwXzvmu6GE G/ycGPdGlwQtWgwy+9k6tvq8xjK+b6sWVQjqfs+Mpt2W5mefvDUavGqSXNCzjz1nPOIZ mlxccJTat2vUjyxKL7zzmTVxRYK4A2VdVwRhXwRjluZIS/ckAtZqHti3VtF7H4EzWTeZ iv/w==
X-Gm-Message-State: AKaTC039hoA9LRHDU33CZCQcxrLZYfeHVY055/DScNc4KcD0ARX1+R14CGROt0s6q/4EVQ==
X-Received: by 10.46.76.2 with SMTP id z2mr12038514lja.32.1480382432404; Mon, 28 Nov 2016 17:20:32 -0800 (PST)
Received: from [192.168.100.13] (37-33-82-144.bb.dnainternet.fi. [37.33.82.144]) by smtp.gmail.com with ESMTPSA id 125sm13206839ljj.26.2016.11.28.17.20.30 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 28 Nov 2016 17:20:31 -0800 (PST)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
From: Jonathan Morton 
In-Reply-To: <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>
Date: Tue, 29 Nov 2016 03:20:28 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>
To: Bob Briscoe 
X-Mailer: Apple Mail (2.3124)
Archived-At: 
Cc: tcpm IETF list , tsvwg IETF list , "Bless, Roland \(TM\)" , TCP Prague List , AQM IETF list 
Subject: Re: [tcpPrague] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 29 Nov 2016 01:20:36 -0000

> On 29 Nov, 2016, at 02:45, Bob Briscoe  wrote:
>=20
> I am particularly worried about embedding fq in the Internet. That is =
far worse than embedding a subtly different performance improvement for =
certain congestion controls. With fq, the network determines the precise =
departure time of each packet, completely overriding the host's choice, =
without any understanding of what the applications is trying to achieve.

You don=E2=80=99t seem to be very fond of *either* component of =
fq_codel, then.  A shame, since it works very well in practice.

What fq_codel does is identify latency-sensitive flows by the fact that =
they are not taking up their fair share of the bandwidth.  Packets =
belonging to these flows are typically delivered *immediately* and =
*without loss*.  This is a far cry from =E2=80=9Ccompletely overriding =
the hosts=E2=80=99s choice=E2=80=9D of timing, as you claim.

In fact, I would actually be grateful if you retracted that particular =
claim.

It must be emphasised, though, that flow-isolating AQMs are not designed =
for the Internet core - there are just too many individual flows to =
manage.  Their place is at the edge, on either end of last-mile links, =
where the bottlenecks are most apparent.  For core, backhaul and peering =
links, plain AQM is sufficient and much easier to implement.

> Even worse, in Jan 2017, I am told that fq_CoDel will become hard =
coded into the Linux WiFi drivers, without even a framework to =
dynamically load any alternative(s). Of course, we can add such a =
framework, but we are seeing Linux become the next major middlebox =
problem. It might be excusable if there were not sound alternatives =
available,... but there are.

This tight integration is because it was necessary to solve some =
serious, long-standing problems with Linux wifi, which couldn=E2=80=99t =
be solved satisfactorily at the qdisc layer because information about =
wifi-specific things was needed - and there were *no* practical =
alternatives which actually solved the problem, otherwise we=E2=80=99d =
have used them.

Wifi is also a last-mile technology, and it is often the bottleneck in =
several types of practical deployment.  Large conferences are a =
particular example.  I=E2=80=99m rather looking forward to seeing the =
first large conference to deploy the new Linux wifi stuff, and seeing =
whether it has made the typical load there easier to cope with.  It =
probably has.

 - Jonathan Morton


From nobody Mon Nov 28 18:56:11 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E7BDF12A26D for ; Mon, 28 Nov 2016 18:55:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.197
X-Spam-Level: 
X-Spam-Status: No, score=-4.197 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RP_MATCHES_RCVD=-1.497, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5o3V6659v-pi for ; Mon, 28 Nov 2016 18:55:53 -0800 (PST)
Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DF76A12A26C for ; Mon, 28 Nov 2016 18:55:49 -0800 (PST)
Received: by mail-io0-x22a.google.com with SMTP id c21so256174310ioj.1 for ; Mon, 28 Nov 2016 18:55:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=jkMOP/KHQzSi2auJF5ZhlJuZm6xyOw++BktHAv0Lkhs=; b=m9ZieNeEpGGS+q83lG7dnx8AD/BUbqQh7RahH63XjuRNxx2F7qoBfkDt6D/CogE5wH kiSgQpS3Yaj0tNew3QsTA7vQoiRZphFApXkPLNZbAxlKm79KWZ4V63wh8TGQPISKP5Dm y/IIVw9vz07lmL+NFXkeSoNZ8AXf6cFHOvGp+Zcclk+c/HfcczTERYK24kCGlmlLabiA mCZtQCLOi1okWITHR9C/KVBkeBDjtJHgVcyD4Gyo//JEfs20YVpaCkE3IwgIi7nucfx3 XPsnvQJ5ayHDmFhZ330H9gU8cWv5tRGe3MXD/smoKrfL6Sio8lC3F7PxE66ycKiG4Fxq oYUA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=jkMOP/KHQzSi2auJF5ZhlJuZm6xyOw++BktHAv0Lkhs=; b=gLwQlcIuqxs7GBcAsyZCXPu0kVpzCY9b22EctARrxzAsLnVc1XP8/e2hykfQgKhbMV SsdRZ3bXoiXoR9/WwdP+kT3ztOZ1ikPrDfIZb2abaW5U8gu++0l3SPnYysqWe/zMCGmr zAb2S4nxl1YeRNd5ldh9n2NB3VXgQSmVKfm7J24QJIU1YTNnwzuG6CS20Kv3XN6fADrd rM2tqpiJeEaRLACyyWkHrHdo86MF4ujA5tU9noXQLi285T7RmK8d3gyte/VwzQJmfp6X 7fqkhayV0HgB08YNbZ9RtrozG+4nM9AP9A+jcjcYsxC9PaLF0RTT2Te5jMgm4E1FkfEQ yi9A==
X-Gm-Message-State: AKaTC01DnFR8a9wVNlxRXEEzjLz+DmWhof5PuWxsINjPBZ6rABoRPbRYYV/dFYJpZ8JPJhDkRTjr7h0tp4wwO+4Y
X-Received: by 10.36.178.9 with SMTP id u9mr21501591ite.80.1480388148902; Mon, 28 Nov 2016 18:55:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.64.252.199 with HTTP; Mon, 28 Nov 2016 18:55:28 -0800 (PST)
In-Reply-To: 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net> 
From: Matt Mathis 
Date: Mon, 28 Nov 2016 18:55:28 -0800
Message-ID: 
To: Jonathan Morton 
Content-Type: multipart/alternative; boundary=f403045d9db6bf22fe054267b9bd
Archived-At: 
Cc: tcpm IETF list , tsvwg IETF list , Bob Briscoe , TCP Prague List , "Bless, Roland \(TM\)" , AQM IETF list 
Subject: Re: [tcpPrague] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 29 Nov 2016 02:55:55 -0000

--f403045d9db6bf22fe054267b9bd
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Bob's point is that fq_anything forfeits any mechanism for an application
or user to imply the value of the traffic by how much congestion they are
willing to inflict on other traffic.

This concept is the foundation of ConEx and related technologies which
could move the capacity allocation problem into the economic domain.  This
actually makes a lot of sense.    However, after investing a lot of time in
it over the years (in collaboration with Bob etal), I have come to believe
that this only makes sense for enforcing financially equitable use of a
deliberately undersized network.    Building capacity is generally cheaper
than building the congestion instrumentation.....

That said, fq_anything does not work at core router scale.

Note that there are two views, each of which is self consistent:

1) You need fq_* to isolate flows; prioritization must be done with
IP/TOS/DSCP bits; aggressive flows can't hurt other flows; low delays
require that flows sharing a Q to be nice to each other and respond to AQM
2) Uniform AQM/drop/mark per packet permits shared economic view of the
value of the traffic (e.g. a price) ; traffic is prioritized by how
aggressive of CC you choose; low delay [is/should be] a design property of
the shared CC and AQM algorithms.

If you have a way to create proper incentives about congestion (e.g. price
and chargeback), #2 is probably a strong system; if that fails #1 is
probably stronger.

Note that half solutions or solutions split between the models don't work.
period.  Arguing about incomplete systems that are missing some of the
parts is pointless because they don't work at some level (often layer 8 or
9).

Drop tail counts as "neither", so even partial deployment of anything can
be an improvement.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Mon, Nov 28, 2016 at 5:20 PM, Jonathan Morton 
wrote:

>
> > On 29 Nov, 2016, at 02:45, Bob Briscoe  wrote:
> >
> > I am particularly worried about embedding fq in the Internet. That is
> far worse than embedding a subtly different performance improvement for
> certain congestion controls. With fq, the network determines the precise
> departure time of each packet, completely overriding the host's choice,
> without any understanding of what the applications is trying to achieve.
>
> You don=E2=80=99t seem to be very fond of *either* component of fq_codel,=
 then.  A
> shame, since it works very well in practice.
>
> What fq_codel does is identify latency-sensitive flows by the fact that
> they are not taking up their fair share of the bandwidth.  Packets
> belonging to these flows are typically delivered *immediately* and *witho=
ut
> loss*.  This is a far cry from =E2=80=9Ccompletely overriding the hosts=
=E2=80=99s choice=E2=80=9D
> of timing, as you claim.
>
> In fact, I would actually be grateful if you retracted that particular
> claim.
>
> It must be emphasised, though, that flow-isolating AQMs are not designed
> for the Internet core - there are just too many individual flows to
> manage.  Their place is at the edge, on either end of last-mile links,
> where the bottlenecks are most apparent.  For core, backhaul and peering
> links, plain AQM is sufficient and much easier to implement.
>
> > Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded
> into the Linux WiFi drivers, without even a framework to dynamically load
> any alternative(s). Of course, we can add such a framework, but we are
> seeing Linux become the next major middlebox problem. It might be excusab=
le
> if there were not sound alternatives available,... but there are.
>
> This tight integration is because it was necessary to solve some serious,
> long-standing problems with Linux wifi, which couldn=E2=80=99t be solved
> satisfactorily at the qdisc layer because information about wifi-specific
> things was needed - and there were *no* practical alternatives which
> actually solved the problem, otherwise we=E2=80=99d have used them.
>
> Wifi is also a last-mile technology, and it is often the bottleneck in
> several types of practical deployment.  Large conferences are a particula=
r
> example.  I=E2=80=99m rather looking forward to seeing the first large co=
nference
> to deploy the new Linux wifi stuff, and seeing whether it has made the
> typical load there easier to cope with.  It probably has.
>
>  - Jonathan Morton
>
> _______________________________________________
> tcpPrague mailing list
> tcpPrague@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpprague
>

--f403045d9db6bf22fe054267b9bd
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


Bob's point is that fq_anything forfeits any mechanism=
 for an application or user to imply the value of the traffic by how much c=
ongestion they are willing to inflict on other traffic.

=
This concept is the foundation of ConEx and related technologies which coul=
d move the capacity allocation problem into the economic domain.=C2=A0 This=
 actually makes a lot of sense. =C2=A0 =C2=A0However, after investing a lot=
 of time in it over the years (in collaboration with Bob etal), I have come=
 to believe that this only makes sense for enforcing financially equitable =
use of a deliberately undersized network. =C2=A0 =C2=A0Building capacity is=
 generally cheaper than building the congestion instrumentation.....
<=
div>
That said, fq_anything does not work at core router scal=
e.

Note that there are two views, each of which is=
 self consistent:

1) You need fq_* to isolate flow=
s; prioritization must be done with IP/TOS/DSCP bits; aggressive flows can&=
#39;t hurt other flows; low delays require that flows sharing a Q to be nic=
e to each other and respond to AQM
2) Uniform AQM/drop/mark per p=
acket permits shared economic view of the value of the traffic (e.g. a pric=
e)=C2=A0; traffic is prioritized by how aggressive of CC you choose; low de=
lay [is/should be] a design property of the shared CC and AQM algorithms.

If you have a way to create proper incentives about=
 congestion (e.g. price and chargeback), #2 is probably a strong system; if=
 that fails #1 is probably stronger.

Note that hal=
f solutions or solutions split between the models don't work. period.=
=C2=A0 Arguing about incomplete systems that are missing some of the parts =
is pointless because they don't work at some level (often layer 8 or 9)=
.

Drop tail counts as "neither", so even=
 partial deployment of anything can be an improvement.

Thanks,
--MM--The best way to predict the future is to create it. =C2=A0- Alan Kay
<=
br>Privacy matters!=C2=A0 We know from recent events that people are using =
our services to speak in defiance of unjust governments. =C2=A0 We treat pr=
ivacy and security as matters of life and death, because for some users, th=
ey are.


On Mon, Nov 28, 2016 at 5:20 PM, Jonathan Mo=
rton <chromatix99@gmail.com> wrote:


> On 29 Nov, 2016, at 02:45, Bob Briscoe <ietf@bobbriscoe.net> wrote:

>

> I am particularly worried about embedding fq in the Internet. That is =
far worse than embedding a subtly different performance improvement for cer=
tain congestion controls. With fq, the network determines the precise depar=
ture time of each packet, completely overriding the host's choice, with=
out any understanding of what the applications is trying to achieve.



You don=E2=80=99t seem to be very fond of *either* component of fq_c=
odel, then.=C2=A0 A shame, since it works very well in practice.



What fq_codel does is identify latency-sensitive flows by the fact that the=
y are not taking up their fair share of the bandwidth.=C2=A0 Packets belong=
ing to these flows are typically delivered *immediately* and *without loss*=
.=C2=A0 This is a far cry from =E2=80=9Ccompletely overriding the hosts=E2=
=80=99s choice=E2=80=9D of timing, as you claim.



In fact, I would actually be grateful if you retracted that particular clai=
m.



It must be emphasised, though, that flow-isolating AQMs are not designed fo=
r the Internet core - there are just too many individual flows to manage.=
=C2=A0 Their place is at the edge, on either end of last-mile links, where =
the bottlenecks are most apparent.=C2=A0 For core, backhaul and peering lin=
ks, plain AQM is sufficient and much easier to implement.



> Even worse, in Jan 2017, I am told that fq_CoDel will become hard code=
d into the Linux WiFi drivers, without even a framework to dynamically load=
 any alternative(s). Of course, we can add such a framework, but we are see=
ing Linux become the next major middlebox problem. It might be excusable if=
 there were not sound alternatives available,... but there are.



This tight integration is because it was necessary to solve some ser=
ious, long-standing problems with Linux wifi, which couldn=E2=80=99t be sol=
ved satisfactorily at the qdisc layer because information about wifi-specif=
ic things was needed - and there were *no* practical alternatives which act=
ually solved the problem, otherwise we=E2=80=99d have used them.



Wifi is also a last-mile technology, and it is often the bottleneck in seve=
ral types of practical deployment.=C2=A0 Large conferences are a particular=
 example.=C2=A0 I=E2=80=99m rather looking forward to seeing the first larg=
e conference to deploy the new Linux wifi stuff, and seeing whether it has =
made the typical load there easier to cope with.=C2=A0 It probably has.



=C2=A0- Jonathan Morton




_______________________________________________

tcpPrague mailing list

tcpPrague@ietf.org

https://www.ietf.org/mailman/listinfo/tcpprague<=
/a>





--f403045d9db6bf22fe054267b9bd--


From nobody Mon Nov 28 19:42:30 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A43E51294F2; Mon, 28 Nov 2016 19:42:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.45
X-Spam-Level: 
X-Spam-Status: No, score=-2.45 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P2uW8ryP8TtH; Mon, 28 Nov 2016 19:42:16 -0800 (PST)
Received: from mail-lf0-x241.google.com (mail-lf0-x241.google.com [IPv6:2a00:1450:4010:c07::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5908012945A; Mon, 28 Nov 2016 19:42:16 -0800 (PST)
Received: by mail-lf0-x241.google.com with SMTP id o141so11408362lff.1; Mon, 28 Nov 2016 19:42:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;  h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H38vsGm3lLVUb1pXAvPmM94o0lBUrJ8kxEo/85EYNM8=; b=zwHDBh39O2AP6Z0iISzJxWsJvxqWqPxdDkzISJlR7czMHScRMo+KRyqFGWYsMGLQ+v 8F31IvJfng0usvisKmcyYMG9Budo7d8c+sZRU7wRdcg0LCRYaUH84i2QzxMnR3ZwSXLK SSihx4P6qtNo02L/advWxgZqJIXGjsiCVqgD9XORUSOm55/2KcaXla7SWH1oq2jXdhwP a3nH+w3Jn9v0eSncGhX02ENTuOZTpync1zlvhaaqzBywOSAfTyKuA9hzpz1hCiGm2ab+ JD4/mDFfXTt+nWz5gFnb83ph0L2H2rb4hgFW2BayTHGHiyIMOQWGceF5rDfgeJOhH6mj hhRA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=H38vsGm3lLVUb1pXAvPmM94o0lBUrJ8kxEo/85EYNM8=; b=CBE6HAV4injevT8ioO1oPlUlsTk7tDFci4dw7UM9iZjGSTiiSiLmoRgSzzdARse2tC jYG2LMHj3fXxtNIL3kcSozfv7ifYhbtsExte5VfPdvWhELGgMrXYmUwK95Xl4e6gyvU0 6lW4S6i90+GyqHgoCo+JPYv6b/m+tBzqdGi2dTmUpZau5Daop3sm/zThKimy74TlSAEv 3zisqwaNGVxDIPGrPCy3DV6QTUNE8tdiicn0tTsxDw5dG1bbxI6lG5HjTSYVe3ni89QC VDSOIEm8iKr3s6qOlE5EscpHXsJ2jSql6lHDNAgRl2GOYXypv5qNR3PVGBTgsZjnR2hG RH8g==
X-Gm-Message-State: AKaTC00zzr2W5vU/bbDppj0yashXmeh6YlNL6GJv60Ev8Ug5FExnE/JBFjC1TqHtO/vcHA==
X-Received: by 10.46.69.137 with SMTP id s131mr12405559lja.26.1480390934462; Mon, 28 Nov 2016 19:42:14 -0800 (PST)
Received: from [192.168.100.13] (37-33-82-144.bb.dnainternet.fi. [37.33.82.144]) by smtp.gmail.com with ESMTPSA id u63sm13156328lja.34.2016.11.28.19.42.12 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 28 Nov 2016 19:42:13 -0800 (PST)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
From: Jonathan Morton 
In-Reply-To: 
Date: Tue, 29 Nov 2016 05:42:11 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>  
To: Matt Mathis 
X-Mailer: Apple Mail (2.3124)
Archived-At: 
Cc: tcpm IETF list , tsvwg IETF list , Bob Briscoe , TCP Prague List , "Bless, Roland \(TM\)" , AQM IETF list 
Subject: Re: [tcpPrague] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 29 Nov 2016 03:42:19 -0000

> On 29 Nov, 2016, at 04:55, Matt Mathis  wrote:
>=20
> Bob's point is that fq_anything forfeits any mechanism for an =
application or user to imply the value of the traffic by how much =
congestion they are willing to inflict on other traffic.

Yes, it does.

I actually consider that a good thing, because most applications will, =
given the choice, choose to inflict more congestion on other traffic in =
order to boost their own performance.  There are honourable exceptions, =
but it=E2=80=99s not a behaviour we can solely rely on.

For example, Steam uses between four and eight parallel TCP streams (I =
can=E2=80=99t figure out what the number depends on) to receive game =
updates, when one or two would already saturate most domestic Internet =
connections.  This magnifies the impact on other things the user might =
be doing with that connection, such as - ironically enough - playing =
multiplayer games.  You=E2=80=99d think Valve, of all companies, would =
keep that in mind.

> This concept is the foundation of ConEx and related technologies which =
could move the capacity allocation problem into the economic domain.

=E2=80=9CEconomic domain=E2=80=9D only works if there is a financial =
cost borne by the causer of the congestion.  Good luck making that work, =
in a world where IoT device manufacturers don=E2=80=99t bear the costs =
of DDoSes launched through them.

> That said, fq_anything does not work at core router scale.

Correct.

> Note that there are two views, each of which is self consistent:
>=20
> 1) You need fq_* to isolate flows; prioritization must be done with =
IP/TOS/DSCP bits; aggressive flows can't hurt other flows; low delays =
require that flows sharing a Q to be nice to each other and respond to =
AQM

=E2=80=A6or that different flows are carefully kept in different queues.

Cake uses set-associative flow hashing to achieve flow isolation much =
more reliably than the current version of fq_codel.

Cake also applies Codel and BLUE in parallel, each covering a different =
AQM regime - BLUE takes over if and when Codel fails to control a =
particular queue.  If both Codel and BLUE fail to control the queue, =
Cake uses head-drops from the longest queue to remain within a total =
memory budget.  All of these avoid penalising well-behaved traffic =
whenever possible.

Cake also has mechanisms which consider =E2=80=9Cper host=E2=80=9D =
fairness simultaneously with =E2=80=9Cper flow=E2=80=9D fairness.  =
Incidentally, the Linux wifi got =E2=80=9Cper station airtime =
fairness=E2=80=9D along with the fq_codel upgrade, achieving a similar =
aim by different means.

SFB uses a Bloom filter to a similar end, though that=E2=80=99s not =
strictly an FQ qdisc.

> 2) Uniform AQM/drop/mark per packet permits shared economic view of =
the value of the traffic (e.g. a price) ; traffic is prioritized by how =
aggressive of CC you choose; low delay [is/should be] a design property =
of the shared CC and AQM algorithms.

It=E2=80=99s fair to say that =E2=80=9Cuniform AQM=E2=80=9D is the only =
mechanism available at the core level, mainly because there are too many =
flows there to treat individually.  But at that level, network =
engineering is supposedly all about providing sufficient link capacity =
so that AQM of any kind is unnecessary, because there is no congestion.  =
Still, plain AQM is a valid and potentially useful mitigation against =
transient overloads.

There are exceptions.  Some ISPs have been known to deliberately =
restrict peering capacity in certain directions to deliberately cause =
congestion to specific types of traffic, supposedly as a financial =
lever.  These ISPs would not be interested in applying AQM to reduce the =
impact of this deliberately-induced congestion.

I do have to ask, though, what protection this =E2=80=9Cshared economic =
view" provides against a single bulk flow which simply ignores all =
congestion signals?

> If you have a way to create proper incentives about congestion (e.g. =
price and chargeback), #2 is probably a strong system; if that fails #1 =
is probably stronger.
>=20
> Note that half solutions or solutions split between the models don't =
work. period.  Arguing about incomplete systems that are missing some of =
the parts is pointless because they don't work at some level (often =
layer 8 or 9).

Indeed.  I=E2=80=99m not aware of any =E2=80=9Ccomplete=E2=80=9D systems =
in category 2 - and I don=E2=80=99t count =E2=80=9Cdata caps=E2=80=9D =
among them, unpopular though they are.

 - Jonathan Morton


From nobody Tue Nov 29 19:46:16 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BD7C01299A1; Tue, 29 Nov 2016 19:46:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.398
X-Spam-Level: 
X-Spam-Status: No, score=-3.398 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-1.497, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A6QjOiZcVW8y; Tue, 29 Nov 2016 19:46:10 -0800 (PST)
Received: from mail.taht.net (mail.taht.net [176.58.107.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 079FE129D2A; Tue, 29 Nov 2016 19:40:38 -0800 (PST)
Received: from dair-2506.local (c-73-202-26-20.hsd1.ca.comcast.net [73.202.26.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.taht.net (Postfix) with ESMTPSA id 07F4621341; Wed, 30 Nov 2016 03:40:34 +0000 (UTC)
To: Jonathan Morton , Matt Mathis 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>   
From: =?UTF-8?Q?Dave_T=c3=a4ht?= 
Message-ID: 
Date: Tue, 29 Nov 2016 19:40:31 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Archived-At: 
Cc: tcpm IETF list , tsvwg IETF list , Bob Briscoe , TCP Prague List , "Bless, Roland \(TM\)" , AQM IETF list 
Subject: Re: [tcpPrague] [aqm]  L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 30 Nov 2016 03:46:11 -0000

The advantages of FQ are probably more widespread than people think.

* Switches tend to multiplex between ports, thus mixing up traffic
naturally.

* Most modern ethernet cards expose 8-64 queues - why? because this
provides clear paths for multiple cpus to forward them - it's not
network related at all! - 1gig cards typically have 8, 10gig, 64.

While 8 induces a severe birthday problem, 64 is generally "just fine"
and in either case, large amounts of mixed traffic respond reasonably to
even only a few queues.

I don't see the trend to ever more hardware queues subsiding, as it's
a function of the number of cores, which keeps going up and up.

* Google'd hosts - and those of a few cloudy providers - long ago
switched to sch_fq which also interleaves (and paces) flows at an
appropriate burst level on a 1ms interval. So source flows are getting
fq'd on an ever more regular basis.

So when the assertion is made that fq is impossible on core routers, I
have to point to the fact that much of the traffic traversing them have
already had a great deal of mixing already applied to it by devices
downstream, AND most core routers have more than one link they are
aggregating via multiplexing in the first place.

That's the FQ in the core and datacenter today. I view the side-effects
of the 1 cpu per hw queue as essentially accomplishing all the fq
needed, AND that applying a single aqm to all those flows is well nigh
impossible as these flows MUST be spread across cores to reach these rates.

...

I see no future in hardware designs that can scale past 25Gbit that do
not do some form of parallelism across flows. This again, is something
that already essentially happens within vlans and with multiple
forwarding lookup tables in those.

100Gbit devices are 4 25GB queues wired together, at minimum. Etc.

(Most of our problems nowadays btw, is on rx, not tx - it's really hard
to do all the work required on the rx side!)

In terms of fq_codel -

For strong reasons (birthday problem) we settled on 1024 queues as a
good general purpose number that works well in practice on real data on
real loads, for speeds between 0 and 10Gbits, for homes and small
business. Initially.

We are now seeing people deploy fq_codel successfully on mid-range
systems with 64 hardware queues at 10Gbit - 64K fq_codel'd queues.

Whether or not the aqm part is that effective with that many queues on
real traffic is beyond me. I've had no complaints - as the alternative -
no aqm, no fq - is far worse.

...

The early experiments with fq_pie (not the bsd version, the sch_fq
derived version) showed something like a 12% (don't quote me!)
improvement in the aqm performance by presenting a more equitable
distribution of test traffic (that is presently over-bursty, created by
bursty things like IWXX, tso/gso/gro offloads, and so on).

I was happy with that result, but cognizant that so much test traffic
looks *nothing like* real traffic. Real traffic does not consist of the
long sustained loads so beloved of experimenters and theorists.

The benchmarks I care about most are voip, videoconferencing, gaming,
dns, tls setup, page load time, and anything else that demands low
latency, of course.

...

Cake has pushed the limits of the possible, with the 8 way set
associative idea making for the perfect fq all the math is based on,
I rather would like more folk testing it - it's stable enough now, and
backported as far back as linux 3.14. I am still unsure of the various
mods to the aqm algo - the new per host fq within fq feature (a common
feature request), the denatting, and the optional shaper is a major
advance, and the classification ideas needing further evaluation.

It seems eminently possible to add l4s ideas to it, as soon as someone
gets around to a usable ect(1) patch for tcp ecn negotiation.

When last I tried it, it could push 17GBit through a 40Gbit card, where
pfifo fast did about 27 on a single core/hw queue benchmark. We were
bottlenecking on rx! to be unable to test harder than that.


From nobody Tue Nov 29 20:09:31 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0C8531294B7; Tue, 29 Nov 2016 20:09:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.398
X-Spam-Level: 
X-Spam-Status: No, score=-3.398 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-1.497, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rnu_8K0EPQOC; Tue, 29 Nov 2016 20:09:21 -0800 (PST)
Received: from mail.taht.net (mail.taht.net [IPv6:2a01:7e00::f03c:91ff:feae:7028]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D01E1129447; Tue, 29 Nov 2016 20:09:20 -0800 (PST)
Received: from dair-2506.local (c-73-202-26-20.hsd1.ca.comcast.net [73.202.26.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.taht.net (Postfix) with ESMTPSA id 1F02021341; Wed, 30 Nov 2016 04:09:17 +0000 (UTC)
To: Bob Briscoe , Jonathan Morton 
References:    <053CE5B0-1879-4A2B-B4E9-8BCDDE5BA482@gmail.com> 
From: =?UTF-8?Q?Dave_T=c3=a4ht?= 
Message-ID: 
Date: Tue, 29 Nov 2016 20:09:16 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
Archived-At: 
Cc: tcpm IETF list , AQM IETF list , tsvwg IETF list , "Bless, Roland \(TM\)" , TCP Prague List 
Subject: Re: [tcpPrague] [aqm]  L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 30 Nov 2016 04:09:23 -0000

On 11/25/16 6:23 AM, Bob Briscoe wrote:
> Jonathan,
> 
> On 22/11/16 20:37, Jonathan Morton wrote:
>>> On 22 Nov, 2016, at 21:09, Bob Briscoe  wrote:
>>>
>>> {Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time.
>> If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time.  

We do have a tendency to talk past each other regarding codel's rate.
the 1/sqrt(N) only applies during codel's initial seeking toward an
ideal drop rate. after that -

for 99.99999% of the time, after that, there is no definable *rate* in
codel - it varies based on the workload and the time spent above and
below the target. I really wish I'd produced an animation of this rather
than the first graph I did at stanford.

...

as for what that initial ramp has to do with reno or tcp's behaviors, it
seems to work, and we've tried other things that didn't work as well.
Give cake's variant a shot. You tell me.

> Yes, I pointed that out originally
> . And
> Toke confirmed in response that it did indeed happen in practice.
>> This applies directly to Codel’s signalling strategy, which is to start at one mark per (assumed) RTT, and to increase the marking frequency if that was insufficient to control the queue.
>>
>> When you have multiple Reno flows sharing a single queue, there is a sqrt(N) factor in several of the characteristics, where N is the number of flows.  When such a shared link becomes saturated, all of the flows must be signalled to slow down, but for stochastic reasons it’ll probably take more than N signalling events to do so.  Increasing the signalling frequency while the queue remains insufficiently controlled has a good chance of quickly finding all the flows, while dropping relatively few packets (of non-ECN flows).
> Just throwing a square root in "somewhere", doesn't mean it is the
> correct "somewhere".
> 
> A stated goal of the sqrt in the CoDel control law is to match the
> 1/sqrt(p) in TCP Reno's window formula. Quite aside from whether that is
> a correct goal, it isn't even doing that:
> * ACK-clocked load (number of flows, N) is proportional to 1/cwnd, ie.
> proportional to sqrt(p) [see 2nd para of section 4 of PI2 paper
> ]
> * because CoDel applies a sqrt to the interval between drops, it results
> in a linear increase in p with time (because the sqrt effectively gets
> squared - see the simple maths in my original posting about this
> ).
> 
> So, the question is: "Why is a linear increase in p (starting from 0)
> good for controlling load from N flows, where N is proportional to sqrt(p)?"

Why do you keep insisting that the rate is this rate? See above.

>>
>> I’m reasonably convinced that Codel is a near-optimal solution to congestion signalling on TCP-friendly flows.  
> The word "optimal" has a precise meaning. I think you mean simply "I am
> predisposed to CoDel."

I have yet to see one convincing benchmark or paper of a single queued
aqm that outperforms fq_codel.

We've done thousands over here. And shared the test data. And the tool.
What you got?

> 
> Whether the control law increases p linearly with time or by the sqrt,
> or by any function of time, is not the point anyway. Time is not the
> correct unit for this control law. The CoDel control law has no
> variables in it (other than the point at which it starts) that depend on
> any feature of the traffic. Once the CoDel control law starts, it just
> blindly increases until it reaches a high enough drop level to control
> the traffic. So the higher p needs to be, the longer it takes. And the
> lower p needs to be, the more it will overshoot within a round trip.

Sigh.

> 
> A constant increase in p with time, and no dependency on the traffic is
> just plain wrong.
> 
> No way will that result in anything that anyone could prove was
> "optimal" even if you put caveats around it like "near-optimal" or
> "reasonably convincingly near-optimal".
> 
> Perhaps this helps you to see why claims of near-optimality say more
> about the political or religious zeal of the person making the claim,
> than they do about CoDel itself.

Benchmarks, here. Not zeal.

> 
>> Regrettably, the latter are not the only type of traffic actually found on the Internet.
>>
>> Really, the central assumption of Codel is that each flow requires only one congestion signal event per RTT to cause it to back off.  Codel stops working well on traffic which doesn’t obey that assumption (a linear increase in drop frequency is inadequate to mitigate a flood - you need to work with drop probabilities for that), but it *does* work acceptably well with multiple flows sharing a queue, due to this operating-point search.
> Similarly, isn't the phrase "acceptably well" another way of saying "We
> don't need to consider any other AQM that might be better at handling a
> wider range of load scenarios"? Even though there is already an
> alternative available that increases the drop level dependent on how
> fast the queue is growing.
> 
> This religious belief in a particular technology is not healthy. Please
> can we have some objectivity here.

I really, really feel like this is really going overboard.

It took me this long to reply to this mail because of this bluster.

We have, on this side, published all the code, published every
benchmark, published the vast majority of our test data, and at every
point, shown fq_codel to be the win it is. Cake is a few percentage
points better in some respects. If some technology arrives worth
benchmarking then by god we'll benchmark it.

I don't know how to fall on our sword more than this.

So far as I know, to date, you - and very few on these lists - have
bothered to even try the production quality code we've made readily
available in now dozens of router distributions, and cheap hardware, in
any scenario. But I suspect millions now have. With no complaints. In
fact, near universal praise.

The religion is on your side, bob, not ours. I will always benchmark any
code you give me with the tools I have. My sole mission in life is to
beat bufferbloat in my lifetime.

I have tired of the dialog on these lists because since the bufferbloat
effort started, several billion machines has shipped without decent
queue management,
and I would much rather solve the problems as best I can, for those that
are willing to try the solutions we have developed so far.

- and especially - as we learn from the deployment, what other real
problems exist, to feed forward into the process.

Even then, even with the make-wifi-fast fq_codel and airtime fairness
patches for the ath9k still landing, we will at best, improve the
performance of only a few million more devices, in the hands of early
adopters, next year, a small fraction of the sold total. There are
hundreds of chipsets left to fix.

It's a big Internet. There's room for everyone's ideas on it. I look
forward to seeing yours reach a deployable state.



> Regards
> 
> 
> 
> Bob
>>
>>  - Jonathan Morton
>>
>> _______________________________________________
>> tcpPrague mailing list
>> tcpPrague@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpprague
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> 
> 
> 
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
> 


From nobody Tue Nov 29 22:16:14 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ECA90126CD8; Tue, 29 Nov 2016 22:16:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.398
X-Spam-Level: 
X-Spam-Status: No, score=-3.398 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-1.497, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TYsmtiTW8-JO; Tue, 29 Nov 2016 22:16:05 -0800 (PST)
Received: from mail.taht.net (mail.taht.net [176.58.107.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 844EA129488; Tue, 29 Nov 2016 22:15:44 -0800 (PST)
Received: from dair-2506.local (c-73-202-26-20.hsd1.ca.comcast.net [73.202.26.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.taht.net (Postfix) with ESMTPSA id D97FE21341; Wed, 30 Nov 2016 06:15:39 +0000 (UTC)
To: Jonathan Morton , Bob Briscoe 
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net> 
From: =?UTF-8?Q?Dave_T=c3=a4ht?= 
Message-ID: 
Date: Tue, 29 Nov 2016 22:15:37 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Archived-At: 
Cc: tcpm IETF list , AQM IETF list , tsvwg IETF list , "Bless, Roland \(TM\)" , TCP Prague List 
Subject: Re: [tcpPrague] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 30 Nov 2016 06:16:11 -0000

On 11/28/16 5:20 PM, Jonathan Morton wrote:

>> Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded into the Linux WiFi drivers, without even a framework to dynamically load any alternative(s). Of course, we can add such a framework, but we are seeing Linux become the next major middlebox problem. It might be excusable if there were not sound alternatives available,... but there are.

A) As soon as the need for a pluggable framework arises, the linux wifi
architecture is now simple enough to do so. As it is the results are so
spectacular compared the previous "std" behavior that it will take some
doing to prove a different point. Reductions in latency under load of
over an order of magnitude, bandwidth improvements of 5x, stuff like
that... ( https://lwn.net/Articles/705884/ )

Given that this effort took 3 years out of multiple people's lives, and
a year of slow steady, patching out the old wifi engine, with thousands
of hours of testing and refinement, and a summer lost to fixing bugs, on
next to no budget...

http://blog.cerowrt.org/post/crypto_fq_bug/

I hope that more people give it a go soon (most of it is already
available in nightly builds of lede-project ar71xx based platforms, and
nearing shipment elsewhere in things like the turris omnia). There is at
least one bug left to fix, always.

The ECN support works beautifully, in particular, and this is the first
time in my life, since working on wifi in 1998, that I'd feel
comfortable deploying a single AP to multiple station solution in a
commercial multi-customer environment - (after a bit more testing in the
yurtlab campground testbed) - before now, only p2p was sanely saleable.

B) Secondly, FQ technology has always existed in wifi - many WISP
vendors altered the MAC to behave via TDMA - or called something TDMA
that was really SFQ. Some like ubnt, have even gone so far as
implementing full duplex over separate channels in their high end
airstream products, on top of that. FQ has long been a part of meraki's
product line. I don't know how cisco does their magic, but...

C) Airtime fairness first appeared as optional features in cisco and at
least one other manufacture's gear at least 6 years back. the problem it
solves was first described in 2003.

So it is ironically the inclusion of an aqm for the first time that is
the most novel feature of this make-wifi-fast work (on top of fq, and
with airtime fairness), integrated tightly. Very little work on aqm
technology prior to now has been applied to highly contended, half
duplex, lossy layer 2 medium systems, and I expect there will be much
more work to be done here.

So... it is far from "just linux" you need to deal with - and far less -
the initial patchset is for the ath9k chipset only, with the ath10k
running a bit behind, and the mt76 still floating in limbo. We are still
in search of other chipsets worth optimizing, with the hope we'll find
something that is in an android as well as at least one other 802.11ac
chipset. From a queue management perspective there is mu-mimo next
(which requires gang scheduling of up to 16 stations), reducing txop
size under contention, and dealing with currently excessive layer 2
retries sanely under contention.

BUT: from an engineering perspective, there are two other things wrong
with wifi that are more severe than these latter problems - the first
one - excessive channel scans - was seriously detrimental to voip and
videoconferencing and the air around the user - and is on its way to
being improved - bug and pending fix described here:

http://blog.cerowrt.org/post/disabling_channel_scans/

The second huge problem is that rate control, now that there is such a
large search space in wireless-n and ac - is no longer as quickly
effective as it needs to be - taking 10-20sec or more to settle in on a
good rate, when 100ms would be ideal. Work on fixing that (which also
leverages the all the work above) is hopefully coming along again
(google for minstrel-blues for some of the more public details)

These last four problems are much more severe than coming up with a
different fq, aqm, or airtime fairness algorithm is, and thus, most
worth focusing on next. Hopefully - someone *else* focusing on next -
I'm tired of working on microsecond timescales and would like to find a
new gig that relied on no timescale shorter than a season.

...

Certainly I hope that by january an ath9k laptop and an ath9k AP will
behave better than any linux system has behaved since 802.11b has,
certainly be superior to current LTE in all respects - but it will
probably be another 3-6 months before the major distros pick it up, and
we have not identified another common laptop wifi chipset that can be
improved in these ways.

(the core work mostly applies to APs, and does help a lot no matter the
client chipset or OS. I also hope to find someone working on an enodeb
to start applying the same algorithms in this timeframe)


> This tight integration is because it was necessary to solve some serious, long-standing problems with Linux wifi, which couldn’t be solved satisfactorily at the qdisc layer because information about wifi-specific things was needed - and there were *no* practical alternatives which actually solved the problem, otherwise we’d have used them.
> 
> Wifi is also a last-mile technology, and it is often the bottleneck in several types of practical deployment.  Large conferences are a particular example.  I’m rather looking forward to seeing the first large conference to deploy the new Linux wifi stuff, and seeing whether it has made the typical load there easier to cope with.  It probably has.

Hopefully the upcoming SCALE conference will be our first major
deployment test. If not, battlemesh, wlan slovinia, or sudoroom.

Or for all I know, open-mesh, eero, portal, meraki, and/or plume have
already picked up the code.


> 
>  - Jonathan Morton
> 
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
> 


From nobody Wed Nov 30 13:56:17 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6D053129B6F; Wed, 30 Nov 2016 13:56:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -7.096
X-Spam-Level: 
X-Spam-Status: No, score=-7.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-2.896] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vkkdsvRPqk6G; Wed, 30 Nov 2016 13:56:10 -0800 (PST)
Received: from mailhost.jlc.net (mailhost.jlc.net [199.201.159.4]) by ietfa.amsl.com (Postfix) with ESMTP id C459F129A20; Wed, 30 Nov 2016 13:56:09 -0800 (PST)
Received: by mailhost.jlc.net (Postfix, from userid 104) id 6836F90A29E; Wed, 30 Nov 2016 16:56:08 -0500 (EST)
Date: Wed, 30 Nov 2016 16:56:08 -0500
From: John Leslie 
To: Jonathan Morton 
Message-ID: <20161130215608.GC98127@verdi>
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net> 
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
User-Agent: Mutt/1.6.1 (2016-04-27)
Archived-At: 
Cc: AQM IETF list , tcpm IETF list , Bob Briscoe , tsvwg IETF list , TCP Prague List 
Subject: Re: [tcpPrague] [tcpm] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 30 Nov 2016 21:56:12 -0000

(could we agree to reduce the Cc list on this thread?)

Jonathan Morton  wrote:
>...
> What fq_codel does is identify latency-sensitive flows by the fact that
> they are not taking up their fair share of the bandwidth.  Packets
> belonging to these flows are typically delivered *immediately* and
> *without loss*.

   This is a nice feature; and I certainly don't mean to badmouth fq_codel.

   But it's guessing... And it depends on those flows being able to know
"their fair share" milliseconds in advance: which can fail on short notce.

   Better would be to identify latency-sensitive flows by a specific signal,
and continue to deliver a "fair share" of packets "immediately" and marked
to show the congestion.

   Consider voice traffic. Voice can be intelligible at rather low data
rates; but sounds much better at higher data rates. For conferencing uses,
low delay is pretty critical.

   Sudden reductions in data rate are annoying, but unexpected silence
while you wait for delayed packets are much worse; and losing track of
how much delay there actually is can be fatal.

   Low-latency is "nice" for all traffic; but I claim it's critical for
voice. (Of course, it's "critical" for some other things, too...)

+++++++++++++++++

   Ideally, each packet would carry an indication of a latency target,
beyond which it's less useful and congestion needs to be indicated. I
have been very disappointed that we designed IPv6 without leaving space
for this.

++++++++++++++++

   (Actually, we _did_ originally design for this: TTL was indended to be
decremented every second, whether or not forwarded. Alas, this was grossly
inadequate resolution; so this usage was abandoned.)

--
John Leslie 


From nobody Wed Nov 30 17:54:17 2016
Return-Path: 
X-Original-To: tcpprague@ietfa.amsl.com
Delivered-To: tcpprague@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 10C32129671; Wed, 30 Nov 2016 17:53:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.45
X-Spam-Level: 
X-Spam-Status: No, score=-2.45 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iEtEo_kGvySs; Wed, 30 Nov 2016 17:53:42 -0800 (PST)
Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F3C62129BBC; Wed, 30 Nov 2016 17:53:39 -0800 (PST)
Received: by mail-lf0-x244.google.com with SMTP id p100so17565181lfg.2; Wed, 30 Nov 2016 17:53:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;  h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=HFsnaGFHHHBFSunCWT5YINw57YiaXb236OS0SNOOA5k=; b=z9PL0wsxgjuXuEvkMz42KaCXaF5EPSsdtt1FlJQN5nzhqMOZ19r91LEhorHGKsykYJ eVZryu8g6YKRUqVL6ioErA+OR38MrSkolPyQdp3+Sk7KvtRNrEdDlGi9amNPzSiY1dqz qEyWBUGqbYYH5JxBHgPKwLBOFdZ2/MuHMBzixw4ibo9U3a0MDwyC0v/7OAe6pKsFpsXr 0S+tRqbH+IrMoaCqxg4KXnOsxjyEBn4FQQy8JNfX7XhLIpHaEwePZbDYY5cLxbxbdnuL 8MZp42Po8sbHRXAc9ZqKuuauWK5N7Q44TatfevqP4p0+2nxuhiRBDyd2NhuskIqRyDqz SqWg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=HFsnaGFHHHBFSunCWT5YINw57YiaXb236OS0SNOOA5k=; b=c/XFWK+82ZMluk+NQ7FQagmlPgp9BY+2zcQcAzxg2zwaJfCIY/ikOV4uzEQhOD+XV3 dsN8yIcYRKqSSbH7EiDVLUqEn6y6u7k1jGQ9+znH6Td5co1JkHf8n1p2IuDMLhOk4VEi k1icXNqmWf1KQCSb/xpAO6yZKSW9b+cej/xeB/pq+Niob2w52t8m5/nMDHs6IGyuME81 kYOz8dkl642TAdIOIgHs7mk5ZI7HjKzaJnaUTXwAgLusSWJzwoEu95G/sbI0qE0PMlRS KFRDVFdWN4727ik/A648TJLTsUDrjsT/FZqVds+AINXw01IjHjvisR8TUfTI/B9z14// NYBw==
X-Gm-Message-State: AKaTC03jj6VE997tVdP9qWhYGuYo4sVXi3hWradVveRTN/MnMnz+DZjxFx6djSunYIdGmg==
X-Received: by 10.25.67.85 with SMTP id m21mr11299886lfj.28.1480557218139; Wed, 30 Nov 2016 17:53:38 -0800 (PST)
Received: from [192.168.100.13] (37-33-82-144.bb.dnainternet.fi. [37.33.82.144]) by smtp.gmail.com with ESMTPSA id s7sm14870498lja.14.2016.11.30.17.53.36 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 30 Nov 2016 17:53:37 -0800 (PST)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
From: Jonathan Morton 
In-Reply-To: <20161130215608.GC98127@verdi>
Date: Thu, 1 Dec 2016 03:53:34 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <83C0C0DF-AEA0-4607-86F6-21FB1712BC07@gmail.com>
References:    <627c9db2-a41f-917d-e639-c27df25bdc51@kit.edu> <39e0cfa7-2650-9d98-a933-7e7c016dc276@bobbriscoe.net>  <20161130215608.GC98127@verdi>
To: John Leslie 
X-Mailer: Apple Mail (2.3124)
Archived-At: 
Cc: AQM IETF list , tcpm IETF list , Bob Briscoe , tsvwg IETF list , TCP Prague List 
Subject: Re: [tcpPrague] [tcpm] [aqm] L4S status update
X-BeenThere: tcpprague@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: "To coordinate implementation and standardisation of TCP Prague across platforms. TCP Prague will be an evolution of DCTCP designed to live alongside other TCP variants and derivatives." 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 01 Dec 2016 01:53:46 -0000

> On 30 Nov, 2016, at 23:56, John Leslie  wrote:
>=20
> Better would be to identify latency-sensitive flows by a specific =
signal,
> and continue to deliver a "fair share" of packets "immediately" and =
marked
> to show the congestion.

Cake does that, too.  It looks at Diffserv markings and sorts them into =
categories, each having a separate set of queues.  It even takes care to =
avoid both obvious incentives for using inappropriate DSCPs, and hard =
bandwidth caps where there is spare bandwidth to use - properties that =
few other Diffserv implementations achieve simultaneously.

All of the presets (except for a legacy =E2=80=9Cprecedence=E2=80=9D =
mode) recognise at least CS1 (for background traffic, which in theory =
BitTorrent should be using), TOS4 (often used by SSH), CS6 (for NTP), =
and VA/EF (for voice traffic) as distinct from best-effort traffic, and =
apply appropriate parameters.

A similar system can be built using multiple fq_codel instances and a =
classifier, and such a construction has been available as part of =
OpenWRT for some time.

In practice, not very much traffic carries an appropriate Diffserv code. =
 We=E2=80=99ve found the =E2=80=9Csparse =3D latency sensitive=E2=80=9D =
heuristic to be a good one in practice, both in Cake and in fq_codel.

 - Jonathan Morton