From acmorton@att.com Sun Dec 2 19:04:41 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2546E21F8971; Sun, 2 Dec 2012 19:04:41 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -104.338 X-Spam-Level: X-Spam-Status: No, score=-104.338 tagged_above=-999 required=5 tests=[AWL=2.261, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pOA2e4hZTErN; Sun, 2 Dec 2012 19:04:40 -0800 (PST) Received: from nbfkord-smmo04.seg.att.com (nbfkord-smmo04.seg.att.com [209.65.160.86]) by ietfa.amsl.com (Postfix) with ESMTP id 977A321F8973; Sun, 2 Dec 2012 19:04:39 -0800 (PST) Received: from unknown [144.160.128.153] (EHLO flpi408.enaf.ffdc.sbc.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 3c61cb05.0.148325.00-490.387146.nbfkord-smmo04.seg.att.com (envelope-from ); Mon, 03 Dec 2012 03:04:39 +0000 (UTC) X-MXL-Hash: 50bc16c740eedd27-cb7ecad2f1d06e640536e1370a679ea29ef3ce09 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB334Y76022114; Sun, 2 Dec 2012 19:04:35 -0800 Received: from fflint04.pst.cso.att.com (fflint04.pst.cso.att.com [150.234.39.64]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB334UO4022020 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 2 Dec 2012 19:04:31 -0800 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by fflint04.pst.cso.att.com (RSA Interceptor); Sun, 2 Dec 2012 19:04:17 -0800 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB334GKB003156; Sun, 2 Dec 2012 22:04:16 -0500 Received: from mailgw1.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB3349Wb003065; Sun, 2 Dec 2012 22:04:10 -0500 Received: from lt-hp1044652.att.com (vpn-135-70-98-242.vpn.swst.att.com[135.70.98.242](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121203030403gw100632pre>; Mon, 3 Dec 2012 03:04:09 +0000 X-Originating-IP: [135.70.98.242] Message-Id: <7.0.1.0.0.20121202213404.0020f330@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Sun, 02 Dec 2012 22:02:31 -0500 To: Henning Schulzrinne , Matt Mathis From: Al Morton In-Reply-To: References: <7.0.1.0.0.20121125083420.085ee458@att.com> <7.0.1.0.0.20121125225022.085ee310@att.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.128.153] X-AnalysisOut: [v=2.0 cv=UcDmvtuN c=1 sm=0 a=xwOvzTHDVLE4u4nGvK72ag==:17 a] X-AnalysisOut: [=uA28CKpmbTUA:10 a=5-yN49bxW6EA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=KV2HdAoy] X-AnalysisOut: [uXUA:10 a=doUQZJtgAAAA:8 a=48vgC7mUAAAA:8 a=hZG83p_yAAAA:8] X-AnalysisOut: [ a=6jU5HIQSCzjAYxuITDEA:9 a=CjuIK1q_8ugA:10 a=lZB815dzVvQA] X-AnalysisOut: [:10 a=Hz7IrDYlS0cA:10 a=6jP4kvS9rEKP4eQ2:21 a=QrV_FUjDGCsq] X-AnalysisOut: [-n9y:21] Cc: "ippm@ietf.org" , Nicholas Weaver , Brian Trammell , "lmap@ietf.org" Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 03:04:41 -0000 Regarding "advertised rates": - I think there's variation among the underlying units of measure for the numbers advertised today - this is the point I tried to make during Henning's Transport Area presentation. It would certainly help with service verification if the parameters describing service were clearly defined for those performing the verification - I appreciate that most consumers won't care much about the units of measure, but verification should be scientific (e.g., conducted at the same layer as the spec). - The model-based metrics have appeal to me, in that they are derived from fundamental metrics that we have a reasonable hope of specifying sufficiently such that independent implementations and different hardware/software platforms will be able to make equivalent measurements. It may even turn-out that similar methods can provide a basis for model-based estimates of multiple (application?) metrics (Matt's memo focuses on TCP, which is certainly enough for now). Changing topics slightly, in the list: http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf (pg. 22) I'd prefer to pare this down to start. Even where we have RFCs to refer to, the details of the test stream (for active measurement without user traffic, or active/hybrid measurement during user traffic) need to be agreed, and even the question of whether tests will be conducted during user activity or not needs discussion, maybe this is the first question to address. Al At 06:53 PM 11/30/2012, Henning Schulzrinne wrote: >One thing that I have learned through the MBA experience that it is >hard to find a single metric that addresses all use cases. For >example, consumers (and national bodies charged with consumer >protection) might want to know whether advertised rates agree with >actual rates. Unless ISPs are advertising the model-based metric, it >may not be as useful. For other applications, such as VoIP, latency >and packet loss for specific traffic types matter. Having a metric >that is RTT-independent and uses a different traffic pattern is not >helpful in that case, as much as it might be otherwise. > >On a more general note, from a consumer perspective, there are >probably three kinds of metrics that are useful: > >(a) Comparison between advertised and real-life metric, to allow >fair comparison of different providers. > >(b) Relative metrics: I don't much care what the number is (it could >be expressed in some number between 0 and 100, rather than Mb/s or >ms), but it should preserve relative ordering. If provider A is >significantly better according to the metric X, I should expect my >application (related to that metric) to perform significantly >better, even if I don't get the exact same performance. A rough >example for that is cellular throughput: It is very unlikely that an >individual consumer will get exactly the average throughput measured >by whatever entity, given geographic and device variations, but they >should still be able to pick a better-performing carrier based on >that metric. That's similar to Al's gas mileage example, "your >mileage may vary", but any Prius is still likely to get better gas >mileage than a Lincoln Navigator. > >(c) Threshold metrics: The metric should tell me whether a >particular service is likely to be suitable for my need. Latency is >the classical example: If my access link has a round-trip latency of >400 ms, I may die prematurely in a first-person-shooter game. >Absolute accuracy isn't that important (say, 400 vs. 420 ms), but >consistent definition of the path component and other attributes is. > >Some metrics might serve all three roles, but the emphasis differs. > >I would be interested in comments on the metrics that the FCC >Measurement Broadband America project uses. > >(a) Which ones need standardization first? (Some are already based >on IPPM RFCs, but most aren't.) > >(b) Which ones might be less valuable and/or should be substantially modified? > >See >http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf >(pg. 22) for the current list. > >Henning > >________________________________ >From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al >Morton [acmorton@att.com] >Sent: Monday, November 26, 2012 12:20 AM >To: Matt Mathis >Cc: lmap@ietf.org; Nicholas Weaver; ippm@ietf.org; Brian Trammell >Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... > >Matt, > >I think we agree that better metrics are needed for LMAP, >and that ideally a metric can be applied at any point or pair of points. > >RFC 2330 and most IPPM metrics have a decidedly Source-to-Destination >host construction. Frankly, we did a better job with arbitrary >measurement points in Rec Y.1540. It's worth a look: >http://www.itu.int/rec/T-REC-Y.1540/en > >There are many goals the new metrics could serve: > - externally observable > - measurable by users > - easily related to user application performance > - service verification > - support consumer choice of ??? > - etc. >We need to agree on the set that are needed. > >While there may be some information in a measurement that covers >a combination of networks, measuring more than one network >is only useful when the combined network path delivers >the target performance - the sunny day outcome. Otherwise, >they are inconclusive, as you said in your memo. Some target >performance levels only apply in the scope where they are offered. > >We don't really have the milk fat scenario here, unless the producer, >the truck driver, the convenience store, and your home refrigerator >can all add some unknown amount of fat to the milk. ;-) > >When considering this problem, I've been thinking of the fuel >efficiency ratings provided for new cars in the US and Europe (at least). >The specs illustrate very well that the quantity measured is variable >depending on the conditions of the measurement (in the US, the miles per >gallon for city and highway driving), and they imply the >range of values that normal users can experience (a fairly wide range). >These are some aspects that would be useful to reproduce in our >metrics, I think. > >regards, and YMMV, >Al From Henning.Schulzrinne@fcc.gov Sun Dec 2 20:10:35 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 970F121F8991; Sun, 2 Dec 2012 20:10:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8MSPoIiqyoI3; Sun, 2 Dec 2012 20:10:30 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id 0FBF321F8992; Sun, 2 Dec 2012 20:10:29 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 2 Dec 2012 23:10:27 -0500 From: Henning Schulzrinne To: Al Morton , Matt Mathis Thread-Topic: Advertised rate tests Thread-Index: AQHN0QumbGgQnnYOlU+Klp+qOrfGXg== Date: Mon, 3 Dec 2012 04:10:24 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Brian Trammell , Nicholas Weaver , "lmap@ietf.org" , "ippm@ietf.org" Subject: [lmap] Advertised rate tests X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 04:10:35 -0000 Changing the subject to reflect one topic.=0A= =0A= Currently, the FCC MBA effort measures long-term TCP throughput on a reason= ably well-defined sub-path, roughly reflecting the part of the path seen as= being under the direct control of the consumer ISP. The measurements indic= ate that this tends to agree with the advertised rates. In the old days, it= made sense to compare this to "line rate", but that's clearly not useful a= ny more, given that even DSL is now a variable-rate (line-noise dependent) = L2 service. The rate is essentially determined, at least for cable and fibe= r, by rate controllers.=0A= =0A= Are you suggesting that the long-term TCP throughput is not a good headline= metric?=0A= =0A= For the model-based metric, do they provide a similar Mb/s metric that is l= ikely to correspond to the advertised throughput? In other words, would a m= odel-based metric for a "20 Mb/s" service yield 20 Mb/s or some other numbe= r?=0A= =0A= Henning=0A= =0A= ________________________________________=0A= From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al Morton = [acmorton@att.com]=0A= Sent: Sunday, December 02, 2012 10:02 PM=0A= To: Henning Schulzrinne; Matt Mathis=0A= Cc: ippm@ietf.org; Nicholas Weaver; Brian Trammell; lmap@ietf.org=0A= Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures...=0A= =0A= Regarding "advertised rates":=0A= =0A= - I think there's variation among the underlying units of measure for=0A= the numbers advertised today - this is the point I tried to make=0A= during Henning's Transport Area presentation. It would certainly help=0A= with service verification if the parameters describing service were=0A= clearly defined for those performing the verification - I appreciate tha= t=0A= most consumers won't care much about the units of measure, but verificat= ion=0A= should be scientific (e.g., conducted at the same layer as the spec).=0A= =0A= - The model-based metrics have appeal to me, in that they are derived=0A= from fundamental metrics that we have a reasonable hope of specifying=0A= sufficiently such that independent implementations and different=0A= hardware/software platforms will be able to make equivalent measurements= .=0A= It may even turn-out that similar methods can provide a basis for=0A= model-based estimates of multiple (application?) metrics (Matt's=0A= memo focuses on TCP, which is certainly enough for now).=0A= From Henning.Schulzrinne@fcc.gov Sun Dec 2 20:14:53 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 59B6321F8992; Sun, 2 Dec 2012 20:14:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BDmWUrSJYCbZ; Sun, 2 Dec 2012 20:14:52 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id A8A3A21F8991; Sun, 2 Dec 2012 20:14:52 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0421.002; Sun, 2 Dec 2012 23:14:51 -0500 From: Henning Schulzrinne To: Al Morton , Matt Mathis Thread-Topic: List of metrics Thread-Index: AQHN0Qy9FKy7sELVoUuyeA/UzUluFA== Date: Mon, 3 Dec 2012 04:14:48 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Brian Trammell , Nicholas Weaver , "lmap@ietf.org" , "ippm@ietf.org" Subject: [lmap] List of metrics X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 04:14:53 -0000 Currently, all the active metrics assume an idle link, as the goal is to me= asure the link performance without background traffic. (We can leave out th= e only passive metric, the total byte count, as that's probably not all tha= t interesting for this discussion.)=0A= =0A= At least from our experience, removing samples due to user activity is not = a major problem, i.e., it removes only a marginal number of busy-hour sampl= es.=0A= =0A= I don't see how any other measurement policy can yield useful results for t= he current metrics.=0A= =0A= Henning=0A= =0A= ________________________________________=0A= From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al Morton = [acmorton@att.com]=0A= Sent: Sunday, December 02, 2012 10:02 PM=0A= To: Henning Schulzrinne; Matt Mathis=0A= Cc: ippm@ietf.org; Nicholas Weaver; Brian Trammell; lmap@ietf.org=0A= Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures...=0A= =0A= =0A= Changing topics slightly, in the list:=0A= http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appen= dix.pdf=0A= (pg. 22)=0A= =0A= I'd prefer to pare this down to start. Even where we have RFCs to refer to,= =0A= the details of the test stream (for active measurement without user traffic= ,=0A= or active/hybrid measurement during user traffic) need to be agreed,=0A= and even the question of whether tests will be conducted during user=0A= activity or not needs discussion, maybe this is the first question to=0A= address.=0A= =0A= Al= From acmorton@att.com Mon Dec 3 05:58:01 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CDBEA21F86B1; Mon, 3 Dec 2012 05:58:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.506 X-Spam-Level: X-Spam-Status: No, score=-105.506 tagged_above=-999 required=5 tests=[AWL=-0.365, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_HTML_ONLY=1.457, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U1-YvI6+As4S; Mon, 3 Dec 2012 05:58:01 -0800 (PST) Received: from nbfkord-smmo04.seg.att.com (nbfkord-smmo04.seg.att.com [209.65.160.86]) by ietfa.amsl.com (Postfix) with ESMTP id C815B21F8668; Mon, 3 Dec 2012 05:58:00 -0800 (PST) Received: from unknown [144.160.128.153] (EHLO flpi408.enaf.ffdc.sbc.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 8efacb05.0.234697.00-427.621754.nbfkord-smmo04.seg.att.com (envelope-from ); Mon, 03 Dec 2012 13:58:00 +0000 (UTC) X-MXL-Hash: 50bcafe848b4597e-16927f636aaa961b70a7430f961e7119f49ba073 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB3Dvx6S013447; Mon, 3 Dec 2012 05:57:59 -0800 Received: from fflint04.pst.cso.att.com (fflint04.pst.cso.att.com [150.234.39.64]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB3DvssC013386 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 3 Dec 2012 05:57:56 -0800 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by fflint04.pst.cso.att.com (RSA Interceptor); Mon, 3 Dec 2012 05:57:26 -0800 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB3DvQEb021747; Mon, 3 Dec 2012 08:57:26 -0500 Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB3DvJ0H021593; Mon, 3 Dec 2012 08:57:19 -0500 Received: from lt-hp1044652.att.com (dn135-16-251-78.dhcpn.ugn.att.com[135.16.251.78](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121203135716gw100632q0e>; Mon, 3 Dec 2012 13:57:17 +0000 X-Originating-IP: [135.16.251.78] Message-Id: <7.0.1.0.0.20121203083106.07e57808@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Mon, 03 Dec 2012 08:55:43 -0500 To: Henning Schulzrinne , Matt Mathis From: Al Morton In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.128.153] X-AnalysisOut: [v=2.0 cv=UcDmvtuN c=1 sm=0 a=xwOvzTHDVLE4u4nGvK72ag==:17 a] X-AnalysisOut: [=DwyFDWaKD4wA:10 a=9lgtAcny86UA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=eSl8r9IW] X-AnalysisOut: [rWUA:10 a=48vgC7mUAAAA:8 a=B6KMzFptAAAA:20 a=gS61Lz15b1P9q] X-AnalysisOut: [OwGdlkA:9 a=CjuIK1q_8ugA:10 a=_W_S_7VecoQA:10 a=lZB815dzVv] X-AnalysisOut: [QA:10 a=Hz7IrDYlS0cA:10 a=9GkpEwqxJgc93UgZ:21 a=iYxQocQxDI] X-AnalysisOut: [oqMkhk:21 a=XBp72y_trfwdYDgt:21] Cc: "ippm@ietf.org" , "lmap@ietf.org" Subject: Re: [lmap] [ippm] Advertised rate tests X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 13:58:01 -0000 At 11:10 PM 12/2/2012, Henning Schulzrinne wrote:
Changing the subject to reflect one topic.
Currently, the FCC MBA effort measures long-term TCP throughput on a reasonably well-defined sub-path, roughly reflecting the part of the path seen as being under the direct control of the consumer ISP. The measurements indicate that this tends to agree with the advertised rates. In the old days, it made sense to compare this to "line rate", but that's clearly not useful any more, given that even DSL is now a variable-rate (line-noise dependent) L2 service. The rate is essentially determined, at least for cable and fiber, by rate controllers.

I think that line rate may be a worthwhile member of the
measurement list, especially when it is variable (capturing
variability within a given spec should be an important aspect
of this exercise, IMO).
Some ISPs are just selling an IP packet transfer service,
and the advertised rate may include IP header and payload.
TCP throughput is measured at a higher layer (just
the tip of the iceberg when it comes to differences).

Are you suggesting that the long-term TCP throughput is not a good headline metric?

I would suggest that it is one of several, because especially with TCP,
YMMV due to many factors along the transport connection's path.
TCP Throughput seems closer to "city" miles per gallon than "highway",
so I think some highway-end metric may be useful, too.

For the model-based metric, do they provide a similar Mb/s metric that is likely to correspond to the advertised throughput? In other words, would a model-based metric for a "20 Mb/s" service yield 20 Mb/s or some other number?

The short answer is yes. I think it's reasonable to assume that what
we standardize will influence the specifications that ISPs advertise.
For more details, see Matt's memo
http://tools.ietf.org/html/draft-mathis-ippm-model-based-metrics-00
and doc at
https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#

regards,
Al


Henning

________________________________________
From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al Morton [acmorton@att.com]
Sent: Sunday, December 02, 2012 10:02 PM
To: Henning Schulzrinne; Matt Mathis
Cc: ippm@ietf.org; Nicholas Weaver; Brian Trammell; lmap@ietf.org
Subject: Re: [lmap] [ippm]  Focus on Tests, Not Architectures...

Regarding "advertised rates":

- I think there's variation among the underlying units of measure for
   the numbers advertised today - this is the point I tried to make
   during Henning's Transport Area presentation. It would certainly help
   with service verification if the parameters describing service were
   clearly defined for those performing the verification - I appreciate that
   most consumers won't care much about the units of measure, but verification
   should be scientific (e.g., conducted at the same layer as the spec).

- The model-based metrics have appeal to me, in that they are derived
   from fundamental metrics that we have a reasonable hope of specifying
   sufficiently such that independent implementations and different
   hardware/software platforms will be able to make equivalent measurements.
   It may even turn-out that similar methods can provide a basis for
   model-based estimates of multiple (application?) metrics (Matt's
   memo focuses on TCP, which is certainly enough for now).
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm
From acmorton@att.com Mon Dec 3 06:24:15 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 69C4C21F86C7; Mon, 3 Dec 2012 06:24:15 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.113 X-Spam-Level: X-Spam-Status: No, score=-106.113 tagged_above=-999 required=5 tests=[AWL=0.486, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1HyM5eM0XMJi; Mon, 3 Dec 2012 06:24:14 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 3ED4321F84E9; Mon, 3 Dec 2012 06:24:14 -0800 (PST) Received: from unknown [144.160.128.153] (EHLO flpi408.enaf.ffdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id d06bcb05.0.245503.00-439.653035.nbfkord-smmo06.seg.att.com (envelope-from ); Mon, 03 Dec 2012 14:24:14 +0000 (UTC) X-MXL-Hash: 50bcb60e0740803a-ce3f709bfb60a05b82952d0cde308d1700df9f4a Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB3EOCkl006521; Mon, 3 Dec 2012 06:24:13 -0800 Received: from fflint04.pst.cso.att.com (fflint04.pst.cso.att.com [150.234.39.64]) by flpi408.enaf.ffdc.sbc.com (8.14.5/8.14.5) with ESMTP id qB3EO0WX006330 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 3 Dec 2012 06:24:06 -0800 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by fflint04.pst.cso.att.com (RSA Interceptor); Mon, 3 Dec 2012 06:23:35 -0800 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB3ENZOx007761; Mon, 3 Dec 2012 09:23:35 -0500 Received: from mailgw1.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qB3ENTZS007622; Mon, 3 Dec 2012 09:23:30 -0500 Received: from lt-hp1044652.att.com (dn135-16-251-78.dhcpn.ugn.att.com[135.16.251.78](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121203142326gw100632q1e>; Mon, 3 Dec 2012 14:23:27 +0000 X-Originating-IP: [135.16.251.78] Message-Id: <7.0.1.0.0.20121203090303.07e576c0@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Mon, 03 Dec 2012 09:21:55 -0500 To: Henning Schulzrinne , Matt Mathis From: Al Morton In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.128.153] X-AnalysisOut: [v=2.0 cv=RfEn/SRv c=1 sm=0 a=xwOvzTHDVLE4u4nGvK72ag==:17 a] X-AnalysisOut: [=DwyFDWaKD4wA:10 a=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=kj9] X-AnalysisOut: [zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=9-0LEm5jLw4A:10 a=48vgC7mU] X-AnalysisOut: [AAAA:8 a=doUQZJtgAAAA:8 a=m7SQDFo_DTHfN1RZ30kA:9 a=CjuIK1q] X-AnalysisOut: [_8ugA:10 a=lZB815dzVvQA:10 a=Hz7IrDYlS0cA:10 a=0fBNF8E5WBn] X-AnalysisOut: [oXR2E:21 a=ZtQD0pYsygndC46T:21] Cc: "ippm@ietf.org" , "lmap@ietf.org" Subject: Re: [lmap] [ippm] List of metrics X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 14:24:15 -0000 Ok, then the UDP latency and UDP packet loss tests were suspended on detecting user activity, and the Latency under Load test used other active tests to provide a known load. I think it would be worthwhile to discuss hybrid active+passive testing, at least. This is what Brian had proposed in http://tools.ietf.org/html/draft-trammell-ippm-hybrid-ps-00 Such tests may tend toward use in diagnostics rather than consumer info, and the value may be tightly linked with the passive characterization of user traffic (appropriately anonymized and privatized, of course), IOW it may not be useful if the load is overly summarized. And some of the Technical Appendix page 22 list also seem diagnostic oriented and somewhat overlapping (ICMP latency, UDP Latency). It's probably good to think of usage categories as we proceed. Al At 11:14 PM 12/2/2012, Henning Schulzrinne wrote: >Currently, all the active metrics assume an idle link, as the goal >is to measure the link performance without background traffic. (We >can leave out the only passive metric, the total byte count, as >that's probably not all that interesting for this discussion.) > >At least from our experience, removing samples due to user activity >is not a major problem, i.e., it removes only a marginal number of >busy-hour samples. > >I don't see how any other measurement policy can yield useful >results for the current metrics. > >Henning > >________________________________________ >From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al >Morton [acmorton@att.com] >Sent: Sunday, December 02, 2012 10:02 PM >To: Henning Schulzrinne; Matt Mathis >Cc: ippm@ietf.org; Nicholas Weaver; Brian Trammell; lmap@ietf.org >Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... > > >Changing topics slightly, in the list: >http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf >(pg. 22) > >I'd prefer to pare this down to start. Even where we have RFCs to refer to, >the details of the test stream (for active measurement without user traffic, >or active/hybrid measurement during user traffic) need to be agreed, >and even the question of whether tests will be conducted during user >activity or not needs discussion, maybe this is the first question to >address. > >Al >_______________________________________________ >ippm mailing list >ippm@ietf.org >https://www.ietf.org/mailman/listinfo/ippm From Henning.Schulzrinne@fcc.gov Mon Dec 3 10:25:24 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D2DD621F8955; Mon, 3 Dec 2012 10:25:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3RD7Rd1DCyNY; Mon, 3 Dec 2012 10:25:24 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id 8A8F921F894F; Mon, 3 Dec 2012 10:25:21 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0421.002; Mon, 3 Dec 2012 13:25:19 -0500 From: Henning Schulzrinne To: Al Morton , Matt Mathis Thread-Topic: [ippm] List of metrics Thread-Index: AQHN0Qy9FKy7sELVoUuyeA/UzUluFJgHIZm7gABBuso= Date: Mon, 3 Dec 2012 18:25:18 +0000 Message-ID: References: , <7.0.1.0.0.20121203090303.07e576c0@att.com> In-Reply-To: <7.0.1.0.0.20121203090303.07e576c0@att.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.63] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "ippm@ietf.org" , "lmap@ietf.org" Subject: Re: [lmap] [ippm] List of metrics X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Dec 2012 18:25:25 -0000 Yes, for all active tests, tests are suspended when user activity is detect= ed. Besides measurement accuracy, the perception was that if users (=3D vol= unteers) got the notion that participating in this project would slow down = their Internet connection, we'd lose them pretty quickly.=0A= =0A= The UDP and ICMP latency tests aren't meant to be diagnostic - they do high= light important technology differences. As you can see from the published r= eports (and the GaTech SIGCOMM paper), DSL tends to impose significantly hi= gher delays than cable, due to interleaving and other factors.=0A= =0A= I agree that hybrid tests could be useful and would likely be seen as more = diagnostic, given their inherent higher variability. I certainly wouldn't w= ant to exclude them from consideration, and don't see much difficulty of in= tegrating them into the LMAP framework, but they are probably of lower prio= rity from a public policy perspective.=0A= =0A= ________________________________________=0A= From: Al Morton [acmorton@att.com]=0A= Sent: Monday, December 03, 2012 9:21 AM=0A= To: Henning Schulzrinne; Matt Mathis=0A= Cc: lmap@ietf.org; ippm@ietf.org=0A= Subject: Re: [ippm] List of metrics=0A= =0A= Ok, then the UDP latency and UDP packet loss tests were suspended=0A= on detecting user activity, and the Latency under Load test=0A= used other active tests to provide a known load.=0A= =0A= I think it would be worthwhile to discuss hybrid active+passive=0A= testing, at least. This is what Brian had proposed in=0A= http://tools.ietf.org/html/draft-trammell-ippm-hybrid-ps-00=0A= Such tests may tend toward use in diagnostics rather than consumer info,=0A= and the value may be tightly linked with the passive characterization=0A= of user traffic (appropriately anonymized and privatized, of course),=0A= IOW it may not be useful if the load is overly summarized.=0A= =0A= And some of the Technical Appendix page 22 list also seem diagnostic=0A= oriented and somewhat overlapping (ICMP latency, UDP Latency).=0A= It's probably good to think of usage categories as we proceed.=0A= =0A= Al=0A= =0A= From mlinsner@cisco.com Fri Dec 21 09:32:29 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AEEDE21F8A7E for ; Fri, 21 Dec 2012 09:32:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.202 X-Spam-Level: X-Spam-Status: No, score=-9.202 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C+TmaIygt+HN for ; Fri, 21 Dec 2012 09:32:28 -0800 (PST) Received: from rcdn-iport-1.cisco.com (rcdn-iport-1.cisco.com [173.37.86.72]) by ietfa.amsl.com (Postfix) with ESMTP id 249D921F872C for ; Fri, 21 Dec 2012 09:32:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=8170; q=dns/txt; s=iport; t=1356111148; x=1357320748; h=date:subject:from:to:message-id:mime-version; bh=EQ6xT7/cb85QzRz68+nM7lPQmWB3xD/kVSwU/7zzSts=; b=M121S3DqZQYlbZmGyZmL56++ujZrumeK+8Re5kor51WR/4s5PoUh3u9M UW3Sp08gs14zGEY3GmiH7e/+LjuHBBtO4iV2LbpdstYn7mX0nXkHDxjRh S438yLr+spQY4qhvXyPOrb2wPHtTPz0tmvUpcJS52ziWk3twMlwXJsQCN o=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Au4HAMmc1FCtJV2Z/2dsb2JhbABFgkl/qBOJCwGILGkWc4IlPU8afRQciAoMlGGhXJEaA4gtNY0pgRyET4pdgxI X-IronPort-AV: E=Sophos;i="4.84,331,1355097600"; d="scan'208,217";a="155356673" Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by rcdn-iport-1.cisco.com with ESMTP; 21 Dec 2012 17:32:27 +0000 Received: from [10.116.195.122] (rtp-mlinsner-8719.cisco.com [10.116.195.122]) by rcdn-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id qBLHWPNs005130 for ; Fri, 21 Dec 2012 17:32:26 GMT User-Agent: Microsoft-MacOutlook/14.2.3.120616 Date: Fri, 21 Dec 2012 12:32:25 -0500 From: Marc Linsner To: Message-ID: Thread-Topic: LMAP Use Case draft Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3438937947_904657" Subject: [lmap] LMAP Use Case draft X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2012 17:32:29 -0000 > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3438937947_904657 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable In an effort to move things forward, I submitted this draft to dissect the use cases away from both the requirements and architecture discussions. Th= e goal is to have a discussion and come to consensus on the use cases to be considered if/when the LMAP work moves forward. Hopefully, breaking topics into smaller more portable documents will allow the topics to move forward at different speeds. Admittedly, this draft is far from complete, but I made an attempt to capture (at least in part) the conversation around use cases from the list. Fire away=8A.. -Marc- A new version of I-D, draft-linsner-lmap-use-cases-00.txt has been successfully submitted by Marc Linsner and posted to the IETF repository. Filename: draft-linsner-lmap-use-cases Revision: 00 Title: Large-Scale Broadband Measurement Use Cases Creation date: 2012-12-21 WG ID: Individual Submission Number of pages: 5 URL: =20 http://www.ietf.org/internet-drafts/draft-linsner-lmap-use-cases-00.txt Status: =20 http://datatracker.ietf.org/doc/draft-linsner-lmap-use-cases Htmlized: http://tools.ietf.org/html/draft-linsner-lmap-use-cases-00 Abstract: Measuring broadband performance on a large scale is important for network diagnostics by providers and users, as well for as public policy. To conduct such measurements, user networks gather data, either on their own initiative or instructed by a measurement controller, and then upload the measurement results to a designated measurement server. Understanding the various scenarios and users of measuring broadband performance is essential to development of the system requirements. The details of the measurement metrics themselves are beyond the scope of this document. =20 The IETF Secretariat --B_3438937947_904657 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
In an effort to move things = forward, I submitted this draft to dissect the use cases away from both the = requirements and architecture discussions.  The goal is to have a discu= ssion and come to consensus on the use cases to be considered if/when the LM= AP work moves forward.  Hopefully, breaking topics into smaller more po= rtable documents will allow the topics to move forward at different speeds.<= /div>

Admittedly, this draft is far from complete, but I = made an attempt to capture (at least in part) the conversation around use ca= ses from the list.

Fire away…..
-Marc-


A new version of I-D, draft-linsner-l= map-use-cases-00.txt
has been successfully submitted by Marc Linsner and posted to the
IETF repository.

Filename:  draft-linsner-lmap-use-c= ases
Revision:<= span class=3D"Apple-tab-span" style=3D"white-space: pre; ">  00
Title:  Large-Scale Broadband Measureme= nt Use Cases
Cr= eation date: =  2012-12-21
 Individual= Submission
Num= ber of pages: 5
=


Abstract:
   Measuring broadband performance = on a large scale is important for
   network diagnostics by providers and users, = as well for as public
   policy.  To conduct such measurements, user ne= tworks gather data,
   either on their own initiative or instructed by a measurem= ent
  = ; controller, and then upload the measurement results to a designated
<= div style=3D"font-family: Consolas; font-size: medium; ">   measurem= ent server.  Understanding the various scenarios and users of
   measur= ing broadband performance is essential to development of the
   system requiremen= ts.  The details of the measurement metrics
   themselves are beyond th= e scope of this document.


 &nb= sp;            &= nbsp;            = ;            &nb= sp;            &= nbsp;            = ;            &nb= sp;    

The IET= F Secretariat
<= br>
--B_3438937947_904657-- From jerome.benoit@grenouille.com Wed Dec 26 15:25:47 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BC77721F8CCA for ; Wed, 26 Dec 2012 15:25:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.995 X-Spam-Level: X-Spam-Status: No, score=-0.995 tagged_above=-999 required=5 tests=[AWL=0.441, BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, HOST_MISMATCH_NET=0.311, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j2vBtaTqPss2 for ; Wed, 26 Dec 2012 15:25:47 -0800 (PST) Received: from laposte.grenouille.com (ns37873.ovh.net [91.121.8.57]) by ietfa.amsl.com (Postfix) with ESMTP id 12D8321F8CC3 for ; Wed, 26 Dec 2012 15:25:47 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by laposte.grenouille.com (Postfix) with ESMTP id 9A07A7F203 for ; Thu, 27 Dec 2012 00:25:45 +0100 (CET) X-Virus-Scanned: spam & virus filtering at laposte.grenouille.com Received: from laposte.grenouille.com ([127.0.0.1]) by localhost (ns37873.ovh.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2BzV8gj4z1R6 for ; Thu, 27 Dec 2012 00:25:38 +0100 (CET) Date: Thu, 27 Dec 2012 00:25:31 +0100 From: =?ISO-8859-1?B?Suly9G1l?= Benoit To: lmap@ietf.org Message-ID: <20121227002531.652c715c@nemesis.grenouille.com> Organization: Grenouille.com X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.13; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/6Y43xeBc1MgBD5JV=P+qP5s"; protocol="application/pgp-signature" Subject: [lmap] Comments on LMAP architecture X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2012 23:25:47 -0000 --Sig_/6Y43xeBc1MgBD5JV=P+qP5s Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello,=20 I've already stated that the logical architecture is overloaded. We currently implementing a measurement system that will fit most of the requirement needed to run large scale measurement from the network edge and I still think that the thin functional separation should not be exposed as a pre-requirement :=20 - Why do you separate measurement result data from measurement definition and metadata ? when you want to dig data, you need the surrounding information that permit to aggregate data without the need to contact anyone.=20 - How will you perform end to end measurement active measurement if the measurement client (I prefer agent) do not embed a measurement server ?=20 - Are you sure that ISP will implement a network parameter server (I prefer network characteristics server) ? We plan to collect the information either via measurements or via the end user. And how will you trust the information given by this server ?=20 I've really a bunch of comments to write (and probably RFCs when the deployment of our measurement system will be on beta stage in France) but I think it's enough to start a fruitful discussion.=20 Cheers. =20 --=20 J=E9r=F4me Benoit aka fraggle La M=E9t=E9o du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D --Sig_/6Y43xeBc1MgBD5JV=P+qP5s Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlDbh2sACgkQ+qDLUJ/pFh3ucQCgyZWdqYcMrOkw2Lr7Zc/arJiA uloAoIKtH25DSp73thqi9Oxte9QH9l2d =v2rq -----END PGP SIGNATURE----- --Sig_/6Y43xeBc1MgBD5JV=P+qP5s-- From Henning.Schulzrinne@fcc.gov Sat Dec 29 20:15:01 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 25E4921F8797 for ; Sat, 29 Dec 2012 20:15:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.449 X-Spam-Level: X-Spam-Status: No, score=-6.449 tagged_above=-999 required=5 tests=[AWL=-0.150, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8x67bdxvnhGI for ; Sat, 29 Dec 2012 20:14:59 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id B626B21F870A for ; Sat, 29 Dec 2012 20:14:59 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0438.000; Sat, 29 Dec 2012 23:14:58 -0500 From: Henning Schulzrinne To: =?iso-8859-1?Q?J=E9r=F4me_Benoit?= , "lmap@ietf.org" Thread-Topic: [lmap] Comments on LMAP architecture Thread-Index: AQHN48BZDRacQwUTNkyKe14Q0rqkeJgwvPE2 Date: Sun, 30 Dec 2012 04:14:56 +0000 Message-ID: References: <20121227002531.652c715c@nemesis.grenouille.com> In-Reply-To: <20121227002531.652c715c@nemesis.grenouille.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.63] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on LMAP architecture X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Dec 2012 04:15:01 -0000 Quick comments:=0A= =0A= 1) I'm not sure where you infer the separation of the two. You certainly do= n't want to include a complete measurement description with each individual= measurement sample, so there will be some notion of a label that refers to= the description. Can you describe your concern in more detail?=0A= =0A= I'm envisioning something similar to Netflow, if that helps to make the dis= cussion a bit more concrete.=0A= =0A= 2) I'm afraid I don't understand. I agree that 'agent' is a more generic te= rm (and I will substitute this in the next version of the draft), but the i= dea is that the infrastructure agent and the in-home agent exchange active = measurement traffic, as specified for each metric. In general, traffic will= flow in both directions. In many practical scenarios, NAT considerations w= ill dictate that the in-home agent initiates the (TCP) session or sends the= ICMP request, but that's not a fundamental part of the architecture.=0A= =0A= 3) The parameter server is an optional component. Not every infrastructure = will have one, but our experience is that ISPs are interested in providing = accurate information and spend a fair amount of manual effort doing so in o= ur current measurement project. They have an interest in providing accurate= information since errors are likely to hurt their performance numbers. The= parameter server helps in automating the process and avoids errors, dealin= g with spreadsheets and the like.=0A= =0A= Unfortunately, it's not always possible to easily measure what the "nominal= " performance is, particularly for more complicated technologies, such as c= able with PowerBoost (temporary speed increases).=0A= =0A= So far, we have found no reason not to trust the information provided by th= e ISP. If you want to get legal about it, false advertising laws in most co= untries, as well as competitors, would discourage any intentional fudging b= y the ISP. Also, the measurements provide a sanity check - if the ISP decla= red 100 Mb/s, but your measurements show 50, this would likely either cause= the measurement sample to be discarded, or a manual investigation.=0A= =0A= Henning=0A= =0A= ________________________________________=0A= From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of J=E9r=F4me= Benoit [jerome.benoit@grenouille.com]=0A= Sent: Wednesday, December 26, 2012 6:25 PM=0A= To: lmap@ietf.org=0A= Subject: [lmap] Comments on LMAP architecture=0A= =0A= Hello,=0A= =0A= I've already stated that the logical architecture is overloaded. We=0A= currently implementing a measurement system that will fit most of the=0A= requirement needed to run large scale measurement from the network=0A= edge and I still think that the thin functional separation should not=0A= be exposed as a pre-requirement :=0A= - Why do you separate measurement result data from measurement=0A= definition and metadata ? when you want to dig data, you need the=0A= surrounding information that permit to aggregate data without the=0A= need to contact anyone.=0A= - How will you perform end to end measurement active measurement if the=0A= measurement client (I prefer agent) do not embed a measurement=0A= server ?=0A= - Are you sure that ISP will implement a network parameter server (I=0A= prefer network characteristics server) ? We plan to collect the=0A= information either via measurements or via the end user. And how=0A= will you trust the information given by this server ?=0A= =0A= I've really a bunch of comments to write (and probably RFCs when the=0A= deployment of our measurement system will be on beta stage in France)=0A= but I think it's enough to start a fruitful discussion.=0A= =0A= Cheers.=0A= =0A= --=0A= J=E9r=F4me Benoit aka fraggle=0A= La M=E9t=E9o du Net - http://grenouille.com=0A= OpenPGP Key ID : 9FE9161D=0A= Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D=0A= From jerome.benoit@grenouille.com Mon Dec 31 06:25:39 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 18C6621F8633 for ; Mon, 31 Dec 2012 06:25:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.141 X-Spam-Level: X-Spam-Status: No, score=-1.141 tagged_above=-999 required=5 tests=[AWL=0.294, BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, HOST_MISMATCH_NET=0.311, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BypKaUtoGQJn for ; Mon, 31 Dec 2012 06:25:38 -0800 (PST) Received: from laposte.grenouille.com (ns37873.ovh.net [91.121.8.57]) by ietfa.amsl.com (Postfix) with ESMTP id AEFA021F865D for ; Mon, 31 Dec 2012 06:25:37 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by laposte.grenouille.com (Postfix) with ESMTP id D3AF17F21E; Mon, 31 Dec 2012 15:25:35 +0100 (CET) X-Virus-Scanned: spam & virus filtering at laposte.grenouille.com Received: from laposte.grenouille.com ([127.0.0.1]) by localhost (ns37873.ovh.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LA6G-HUJE3Mz; Mon, 31 Dec 2012 15:25:19 +0100 (CET) Date: Mon, 31 Dec 2012 15:25:16 +0100 From: =?ISO-8859-1?B?Suly9G1l?= Benoit To: Henning Schulzrinne Message-ID: <20121231152516.14ff63a2@nemesis.grenouille.com> In-Reply-To: References: <20121227002531.652c715c@nemesis.grenouille.com> Organization: Grenouille.com X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.13; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/81WgR/3zAGK_qdE784dd8ht"; protocol="application/pgp-signature" Cc: "lmap@ietf.org" Subject: Re: [lmap] Comments on LMAP architecture X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 14:25:39 -0000 --Sig_/81WgR/3zAGK_qdE784dd8ht Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Sun, 30 Dec 2012 04:14:56 +0000 Henning Schulzrinne wrote: > Quick comments: >=20 > 1) I'm not sure where you infer the separation of the two.=20 I infer because of the logical separation of the data collector and the measurement server. > You certainly don't want to include a complete measurement description wi= th each individual measurement sample,=20 Not in every sample, but before, I think I have to explain a bit how your measurement agent work : grenouille_probe receive a measurement definition expressed in a specific language that say for example "do this and every ten seconds collect measurement results samples". To be concrete : do FTP download of a 600 MB file from ftp://pipo.com/iso_file_path/iso_file_name and every ten seconds, give me the instant bitrate, the instant bitrate is what I call the samples. The complete measurement result contains all timestamped instant bitrate + the measurement definition + the access link + other relevant meta-data . The measurement collector do serialization and data mining is easier. =20 > so there will be some notion of a label that refers to the description. > Can you describe your concern in more detail? You can do it also with keeping an external and unique reference to the measurement definition + access link characteristics + whatever is relevant. But that will not really ease the data mining. The data mining will trigger traffic to measurement controller and network parameter server, which I think is not a good idea. =20 >=20 > I'm envisioning something similar to Netflow, if that helps to make the d= iscussion a bit more concrete. >=20 Keep in mind that IPFIX architecture is relevant to passive measurement. A network element raw flow (as IPFIX define them) contain already almost all the information needed to do troubleshooting, the filtering permit to isolate what you want to troubleshoot.=20 active measurement results do not contain all the information needed, only what you want to measure : timestamped TCP throughput from A to B sample every 5 seconds, RTT from a forged packets train that aim to trigger the delay from several landmarks, etc. all this numbers are meaningless without measurement definition. =20 > 2) I'm afraid I don't understand. I agree that 'agent' is a more generic = term (and I will substitute this in the next version of the draft), but the= idea is that the infrastructure agent and the in-home agent exchange activ= e measurement traffic, as specified for each metric. In general, traffic wi= ll flow in both directions. In many practical scenarios, NAT considerations= will dictate that the in-home agent initiates the (TCP) session or sends t= he ICMP request, but that's not a fundamental part of the architecture. In practice, network usage include end to end traffic (between 1ton NATed home networks). If you want to measure this kind of usage, the measurement agent have to include the measurement server. Typical measurement are P2P protocols metrics. There's other use case but none of them are really ready for production (this mean that a complete overlay thought end to end that permit to do most of the diagnostic with a few overhead is not ready).=20 In the future, the grenouille_landmark will be a parametrisable socket servers shipped is the measurement agent. The agent will send message status to a to be defined protocol that will help to keep track of agent state, capabilities and home network functionalities (UPnP on/off, etc.) .=20 >=20 > 3) The parameter server is an optional component. Not every infrastructur= e will have one, but our experience is that ISPs are interested in providin= g accurate information and spend a fair amount of manual effort doing so in= our current measurement project. They have an interest in providing accura= te information since errors are likely to hurt their performance numbers. T= he parameter server helps in automating the process and avoids errors, deal= ing with spreadsheets and the like. >=20 > Unfortunately, it's not always possible to easily measure what the "nomin= al" performance is, particularly for more complicated technologies, such as= cable with PowerBoost (temporary speed increases). >=20 If ISPs care to collaborate, that fine. But if they do not want, can't and that happen when you're not a backed by a national agency, they tend to trick the measurement system but that's another story. --=20 J=E9r=F4me Benoit aka fraggle La M=E9t=E9o du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D --Sig_/81WgR/3zAGK_qdE784dd8ht Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlDhoE0ACgkQ+qDLUJ/pFh3FIQCg49idShqIuJQ2sM3cQkc6G59p RrEAnjFV4moa15hHfiFiaUhEgTf6GEaL =C8Cm -----END PGP SIGNATURE----- --Sig_/81WgR/3zAGK_qdE784dd8ht-- From Henning.Schulzrinne@fcc.gov Mon Dec 31 06:52:08 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 95A6921F8863 for ; Mon, 31 Dec 2012 06:52:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.419 X-Spam-Level: X-Spam-Status: No, score=-6.419 tagged_above=-999 required=5 tests=[AWL=-0.120, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wxJAIycA4tcJ for ; Mon, 31 Dec 2012 06:51:57 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id 8F86221F851C for ; Mon, 31 Dec 2012 06:51:56 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas03.fccnet.win.fcc.gov ([fe80::8db2:ac93:84fc:c36b%13]) with mapi id 14.01.0438.000; Mon, 31 Dec 2012 09:51:55 -0500 From: Henning Schulzrinne To: =?iso-8859-1?Q?J=E9r=F4me_Benoit?= Thread-Topic: [lmap] Comments on LMAP architecture Thread-Index: AQHN48BZDRacQwUTNkyKe14Q0rqkeJgwvPE2gAKUrgD//7EwDg== Date: Mon, 31 Dec 2012 14:51:54 +0000 Message-ID: References: <20121227002531.652c715c@nemesis.grenouille.com> , <20121231152516.14ff63a2@nemesis.grenouille.com> In-Reply-To: <20121231152516.14ff63a2@nemesis.grenouille.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.63] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "lmap@ietf.org" Subject: Re: [lmap] Comments on LMAP architecture X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Dec 2012 14:52:08 -0000 Let me try to amplify: The data collector is logically separate, but can ob= viously be in the same box. This is necessary for the cases where the agent= on the home/enterprise side generates the measurements, as would be the ca= se for most latency measurements, for example.=0A= =0A= To summarize: Two measurement agents, one in the home and one in the networ= k somewhere, exchange measurement traffic, such as, say, ICMP echo requests= or a TCP download. The agent at home takes the metric (throughput, latency= ) plus identifying information (metric, time, endpoint identifier) and send= s it to the data collector. The transmission format is likely different fro= m the format made available to users of the measurement data. One proposal = was to use IPFIX as a way to encode the data, as IPFIX is just a container = format for records. (Obviously, this is only about the transmission format,= not the other components of IPFIX, which are indeed not applicable.)=0A= =0A= The data collector can then transform, combine and annotate data in any for= mat, or multiple formats, for appropriate usage by authorized entities. It = might make data available in an SQL database, a downloadable file or via an= API, for example.=0A= =0A= I'm not sure I understand why you want them to be the same entity. I admit = I couldn't quite follow your IPFIX objection, but maybe you can explain, ba= sed on the summary above, where you see the problem.=0A= =0A= Henning=0A= ________________________________________=0A= From: J=E9r=F4me Benoit [jerome.benoit@grenouille.com]=0A= Sent: Monday, December 31, 2012 9:25 AM=0A= To: Henning Schulzrinne=0A= Cc: lmap@ietf.org=0A= Subject: Re: [lmap] Comments on LMAP architecture=0A= =0A= On Sun, 30 Dec 2012 04:14:56 +0000=0A= Henning Schulzrinne wrote:=0A= =0A= > Quick comments:=0A= >=0A= > 1) I'm not sure where you infer the separation of the two.=0A= =0A= I infer because of the logical separation of the data collector and the=0A= measurement server.=0A= From jerome.benoit@grenouille.com Mon Dec 31 21:45:50 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3240321F8919 for ; Mon, 31 Dec 2012 21:45:50 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.215 X-Spam-Level: X-Spam-Status: No, score=-1.215 tagged_above=-999 required=5 tests=[AWL=0.220, BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, HOST_MISMATCH_NET=0.311, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RUBN33hseIot for ; Mon, 31 Dec 2012 21:45:49 -0800 (PST) Received: from laposte.grenouille.com (ns37873.ovh.net [91.121.8.57]) by ietfa.amsl.com (Postfix) with ESMTP id C25FD21F8917 for ; Mon, 31 Dec 2012 21:45:45 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by laposte.grenouille.com (Postfix) with ESMTP id 762877F071; Tue, 1 Jan 2013 06:45:43 +0100 (CET) X-Virus-Scanned: spam & virus filtering at laposte.grenouille.com Received: from laposte.grenouille.com ([127.0.0.1]) by localhost (ns37873.ovh.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SZpuLc3XnziG; Tue, 1 Jan 2013 06:45:33 +0100 (CET) Date: Tue, 1 Jan 2013 06:45:24 +0100 From: =?ISO-8859-1?B?Suly9G1l?= Benoit To: Henning Schulzrinne Message-ID: <20130101064524.14e414a9@nemesis.grenouille.com> In-Reply-To: References: <20121227002531.652c715c@nemesis.grenouille.com> <20121231152516.14ff63a2@nemesis.grenouille.com> Organization: Grenouille.com X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.13; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/kNweS7BDKm4ohyqy2/TYppH"; protocol="application/pgp-signature" Cc: "lmap@ietf.org" Subject: Re: [lmap] Comments on LMAP architecture X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jan 2013 05:45:50 -0000 --Sig_/kNweS7BDKm4ohyqy2/TYppH Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Mon, 31 Dec 2012 14:51:54 +0000 Henning Schulzrinne wrote: Ok, don't get me wrong.=20 > Let me try to amplify: The data collector is logically separate, but can = obviously be in the same box. This is necessary for the cases where the age= nt on the home/enterprise side generates the measurements, as would be the = case for most latency measurements, for example. Which is fine, just state it on the RFC (amplify the fact that the logical separation can also be a implemented on the agent).=20 >=20 > To summarize: Two measurement agents, one in the home and one in the netw= ork somewhere, exchange measurement traffic, such as, say, ICMP echo reques= ts or a TCP download. The agent at home takes the metric (throughput, laten= cy) plus identifying information (metric, time, endpoint identifier) and se= nds it to the data collector. The transmission format is likely different f= rom the format made available to users of the measurement data. One proposa= l was to use IPFIX as a way to encode the data, as IPFIX is just a containe= r format for records. (Obviously, this is only about the transmission forma= t, not the other components of IPFIX, which are indeed not applicable.) >=20 > The data collector can then transform, combine and annotate data in any f= ormat, or multiple formats, for appropriate usage by authorized entities. I= t might make data available in an SQL database, a downloadable file or via = an API, for example. >=20 > I'm not sure I understand why you want them to be the same entity. I admi= t I couldn't quite follow your IPFIX objection, but maybe you can explain, = based on the summary above, where you see the problem. > My IPFIX objection is based a an real implementation of a well know active measurement (with extras) and whatever how drunk is I am is, the IPFIX data format do not respect Shannon's theoretical information system standard speed derivation (the IPFIX standard is fine, but do not include octet/time derivative unit) and that's a pity for active measurement. Just state that fact in the RFC and I will never complain=20 about IPFIX standard anymore (maybe, IPFIX is really a bunch of crappy point in time metrics from my point of view, but I will launch that delighted troll on an other mailing list :)) ++ --=20 J=E9r=F4me Benoit aka fraggle La M=E9t=E9o du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D --Sig_/kNweS7BDKm4ohyqy2/TYppH Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlDid/QACgkQ+qDLUJ/pFh2EWQCgiBDimA6/UyxV9nwTRqfOYOyV IecAn2ZA/IfPi2feWpLeyHEe5hg/8MSs =U0yC -----END PGP SIGNATURE----- --Sig_/kNweS7BDKm4ohyqy2/TYppH--