From bertietf@bwijnen.net Thu Nov 1 01:29:22 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 186FA21F8513 for ; Thu, 1 Nov 2012 01:29:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BXeFNE8t5+PD for ; Thu, 1 Nov 2012 01:29:21 -0700 (PDT) Received: from postgirl.ripe.net (postgirl.ipv6.ripe.net [IPv6:2001:67c:2e8:11::c100:1342]) by ietfa.amsl.com (Postfix) with ESMTP id DDD8621F8511 for ; Thu, 1 Nov 2012 01:29:19 -0700 (PDT) Received: from ayeaye.ripe.net ([193.0.23.5]) by postgirl.ripe.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1TTq9I-0001PB-Ob; Thu, 01 Nov 2012 09:29:14 +0100 Received: from dog.ripe.net ([193.0.1.217] helo=guest2.guestnet.ripe.net) by ayeaye.ripe.net with esmtp (Exim 4.72) (envelope-from ) id 1TTq9I-0004Gw-7U; Thu, 01 Nov 2012 09:29:12 +0100 Message-ID: <509232D8.1080309@bwijnen.net> Date: Thu, 01 Nov 2012 09:29:12 +0100 From: "Bert Wijnen (IETF)" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: "Eggert, Lars" , Henning Schulzrinne References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Anti-Virus: Kaspersky Anti-Virus for Linux Mail Server 5.6.48/RELEASE, bases: 20120425 #7816575, check: 20121101 clean X-RIPE-Spam-Level: -- X-RIPE-Spam-Report: Spam Total Points: -2.9 points pts rule name description ---- ---------------------- ------------------------------------ -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-RIPE-Signature: 86ab03e524994f79ca2c75a176445dd4d5d4b4e2d5b78540ca4b6cf3924604ce Cc: "lmap@ietf.org" Subject: Re: [lmap] Bar BOF - when & where? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2012 08:29:22 -0000 So, does the below tell you at which date it is? It lists a "desired" date and a "alternate date" If I look at the doodle poll at http://www.doodle.com/78t3r4m3mfrea5yp Then only 16 people can make it at the "desired" date while 22 can make it at the "alternate" date. So ???? On 10/31/12 11:40 AM, Eggert, Lars wrote: > On Oct 31, 2012, at 11:35, "Eggert, Lars" > wrote: >> when & where is the bar BOF? Has anyone asked for a room? (I have not seen an IRTF-related request, but maybe someone went and asked an AD?) > > ah, found it: > >> Meeting Name: Broadband Performance Measurement >> >> Desired Meeting Date: 11/07/2012 >> Alternate Meeting Date: 11/08/2013 >> Number of Days: 1 >> Meeting Area/Type: TSV >> Expected Attendance: 50 >> Desired Start Time: 19:30:00 **Moved to 20:00:00** >> Desired End Time: 20:30:00 >> Room Configuration: Conference >> Speakerphone: No >> LCD Projector: Yes > > That time is in conflict with the IRSG dinner, so IRTF chairs won't be able to make it. If there are pieces of LMAP that are intended for the IRTF, someone please clue us in after the meeting. > > Thanks, > Lars > > > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > From bclaise@cisco.com Thu Nov 1 23:33:57 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9A3B321F95CC for ; Thu, 1 Nov 2012 23:33:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[AWL=4.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C8TtcXR5vAdp for ; Thu, 1 Nov 2012 23:33:55 -0700 (PDT) Received: from av-tac-sj.cisco.com (av-tac-sj.cisco.com [171.68.227.119]) by ietfa.amsl.com (Postfix) with ESMTP id 551AD21F9431 for ; Thu, 1 Nov 2012 23:33:53 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from fire.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-sj.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id qA26Xn2n003227; Thu, 1 Nov 2012 23:33:49 -0700 (PDT) Received: from [10.21.166.157] ([10.21.166.157]) by fire.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id qA26XkwH015820; Thu, 1 Nov 2012 23:33:47 -0700 (PDT) Message-ID: <5093694A.7000903@cisco.com> Date: Fri, 02 Nov 2012 07:33:46 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: "Bert Wijnen (IETF)" References: <509232D8.1080309@bwijnen.net> In-Reply-To: <509232D8.1080309@bwijnen.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "lmap@ietf.org" , "Eggert, Lars" , Henning Schulzrinne Subject: Re: [lmap] Bar BOF - when & where? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2012 06:33:57 -0000 Can we please have a definitive answer on the date/time/location Regards, Benoit > So, does the below tell you at which date it is? > It lists a "desired" date and a "alternate date" > If I look at the doodle poll at > http://www.doodle.com/78t3r4m3mfrea5yp > Then only 16 people can make it at the "desired" date > while 22 can make it at the "alternate" date. > > So ???? > > On 10/31/12 11:40 AM, Eggert, Lars wrote: >> On Oct 31, 2012, at 11:35, "Eggert, Lars" >> wrote: >>> when & where is the bar BOF? Has anyone asked for a room? (I have >>> not seen an IRTF-related request, but maybe someone went and asked >>> an AD?) >> >> ah, found it: >> >>> Meeting Name: Broadband Performance Measurement >>> >>> Desired Meeting Date: 11/07/2012 >>> Alternate Meeting Date: 11/08/2013 >>> Number of Days: 1 >>> Meeting Area/Type: TSV >>> Expected Attendance: 50 >>> Desired Start Time: 19:30:00 **Moved to 20:00:00** >>> Desired End Time: 20:30:00 >>> Room Configuration: Conference >>> Speakerphone: No >>> LCD Projector: Yes >> >> That time is in conflict with the IRSG dinner, so IRTF chairs won't >> be able to make it. If there are pieces of LMAP that are intended for >> the IRTF, someone please clue us in after the meeting. >> >> Thanks, >> Lars >> >> >> >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >> > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > > From jason.weil@twcable.com Fri Nov 2 07:23:50 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 88E8C21F86A6 for ; Fri, 2 Nov 2012 07:23:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.359 X-Spam-Level: X-Spam-Status: No, score=-0.359 tagged_above=-999 required=5 tests=[AWL=1.104, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NKvAlgMjOKZa for ; Fri, 2 Nov 2012 07:23:48 -0700 (PDT) Received: from cdpipgw01.twcable.com (cdpipgw01.twcable.com [165.237.59.22]) by ietfa.amsl.com (Postfix) with ESMTP id 8EBFE21F859B for ; Fri, 2 Nov 2012 07:23:44 -0700 (PDT) X-SENDER-IP: 10.136.163.14 X-SENDER-REPUTATION: None X-IronPort-AV: E=Sophos;i="4.80,699,1344225600"; d="scan'208";a="463777804" Received: from unknown (HELO PRVPEXHUB05.corp.twcable.com) ([10.136.163.14]) by cdpipgw01.twcable.com with ESMTP/TLS/RC4-MD5; 02 Nov 2012 10:23:05 -0400 Received: from PRVPEXVS12.corp.twcable.com ([10.136.163.45]) by PRVPEXHUB05.corp.twcable.com ([10.136.163.14]) with mapi; Fri, 2 Nov 2012 10:23:37 -0400 From: "Weil, Jason" To: Benoit Claise , "Bert Wijnen (IETF)" Date: Fri, 2 Nov 2012 10:23:34 -0400 Thread-Topic: [lmap] Bar BOF - when & where? Thread-Index: Ac25BaXN63GpqV8GSiu2muu0uARz4g== Message-ID: In-Reply-To: <5093694A.7000903@cisco.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.10.0.110310 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Henning Schulzrinne , "Eggert, Lars" , "lmap@ietf.org" Subject: Re: [lmap] Bar BOF - when & where? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2012 14:23:50 -0000 Martin Stiemerling and I are meeting with the the IETF Secretariat on Sunday afternoon to verify the start time and room information for the meeting scheduled for Wednesday 11/07/2012. I'll post the final information for the meeting on Sunday afternoon to the list and on the announcement board once it has been confirmed. The requested start time of 20:00 was changed to 21:00 in the meeting confirmation email, but this may just be a typo. Sorry for the confusion, Jason On 11/2/12 2:33 AM, "Benoit Claise" wrote: >Can we please have a definitive answer on the date/time/location > >Regards, Benoit >> So, does the below tell you at which date it is? >> It lists a "desired" date and a "alternate date" >> If I look at the doodle poll at >> http://www.doodle.com/78t3r4m3mfrea5yp >> Then only 16 people can make it at the "desired" date >> while 22 can make it at the "alternate" date. >> >> So ???? >> >> On 10/31/12 11:40 AM, Eggert, Lars wrote: >>> On Oct 31, 2012, at 11:35, "Eggert, Lars" >>> wrote: >>>> when & where is the bar BOF? Has anyone asked for a room? (I have >>>> not seen an IRTF-related request, but maybe someone went and asked >>>> an AD?) >>> >>> ah, found it: >>> >>>> Meeting Name: Broadband Performance Measurement >>>> >>>> Desired Meeting Date: 11/07/2012 >>>> Alternate Meeting Date: 11/08/2013 >>>> Number of Days: 1 >>>> Meeting Area/Type: TSV >>>> Expected Attendance: 50 >>>> Desired Start Time: 19:30:00 **Moved to 20:00:00** >>>> Desired End Time: 20:30:00 >>>> Room Configuration: Conference >>>> Speakerphone: No >>>> LCD Projector: Yes >>> >>> That time is in conflict with the IRSG dinner, so IRTF chairs won't >>> be able to make it. If there are pieces of LMAP that are intended for >>> the IRTF, someone please clue us in after the meeting. >>> >>> Thanks, >>> Lars >>> >>> >>> >>> _______________________________________________ >>> lmap mailing list >>> lmap@ietf.org >>> https://www.ietf.org/mailman/listinfo/lmap >>> >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >> >> > >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. From jamesmilleresquire@gmail.com Fri Nov 2 11:46:32 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1932D1F0C7E for ; Fri, 2 Nov 2012 11:46:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kgAwKX9jzn6b for ; Fri, 2 Nov 2012 11:46:20 -0700 (PDT) Received: from mail-vc0-f172.google.com (mail-vc0-f172.google.com [209.85.220.172]) by ietfa.amsl.com (Postfix) with ESMTP id D43CD1F0C6A for ; Fri, 2 Nov 2012 11:46:19 -0700 (PDT) Received: by mail-vc0-f172.google.com with SMTP id fl11so4527252vcb.31 for ; Fri, 02 Nov 2012 11:46:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=cgNKqOG0ICPWa4c/pffLGVLyLaGCcpNg7DMxJtJ+U6g=; b=TSlynNcbqDQut0IVGp5xzgay57EWQqvos5GqdyMF1hJiJXKDr1eK2xkV+8/IbDEB4w H9XatGGomjhltZnZRT8W2JDNO/LQSUjKOfebhcVbwrsnfnAmxK5tONtgmgy+JumU+tcq yqU1tEBne2ykBQcF5Zc2rF2AoPZBYWnKT3H82hrA7syXK5/eYF+3iTFdgrj9DGSuPl62 2JivMDkuSyLO8JrWasXX3KKuMdsuyM7UvgfULyx03aNytinH3VI63Y1AOzCQKdTweVHf TB8XTtzWXUyWUN/M6cDsCIF64XC6tSXHAXOdPWgjtMgOq+oMFcubhxeHTrtIfISHizc9 l+/Q== Received: by 10.52.33.165 with SMTP id s5mr2379914vdi.55.1351881974189; Fri, 02 Nov 2012 11:46:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.200.226 with HTTP; Fri, 2 Nov 2012 11:45:44 -0700 (PDT) In-Reply-To: References: <5093694A.7000903@cisco.com> From: James Miller Date: Fri, 2 Nov 2012 14:45:44 -0400 Message-ID: To: "Weil, Jason" Content-Type: multipart/alternative; boundary=20cf3079bc70d98e2804cd878cee Cc: Benoit Claise , "lmap@ietf.org" , "Eggert, Lars" , "Bert Wijnen \(IETF\)" , Henning Schulzrinne Subject: Re: [lmap] Bar BOF - when & where? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Nov 2012 18:46:32 -0000 --20cf3079bc70d98e2804cd878cee Content-Type: text/plain; charset=ISO-8859-1 Thanks Jason for all your support organizing the room and time. I'm finalizing my travel orders and expect to be there with Henning to help lead the BAR BOF. I would like to post the final info to the ng-bbperformance list as well when things settle. We'll all plan for Wednesday 8pm though for the meeting (and assume the 9pm start is a typo!). On Fri, Nov 2, 2012 at 10:23 AM, Weil, Jason wrote: > Martin Stiemerling and I are meeting with the the IETF Secretariat on > Sunday afternoon to verify the start time and room information for the > meeting scheduled for Wednesday 11/07/2012. I'll post the final > information for the meeting on Sunday afternoon to the list and on the > announcement board once it has been confirmed. The requested start time of > 20:00 was changed to 21:00 in the meeting confirmation email, but this may > just be a typo. > > Sorry for the confusion, > > Jason > > > > On 11/2/12 2:33 AM, "Benoit Claise" wrote: > > >Can we please have a definitive answer on the date/time/location > > > >Regards, Benoit > >> So, does the below tell you at which date it is? > >> It lists a "desired" date and a "alternate date" > >> If I look at the doodle poll at > >> http://www.doodle.com/78t3r4m3mfrea5yp > >> Then only 16 people can make it at the "desired" date > >> while 22 can make it at the "alternate" date. > >> > >> So ???? > >> > >> On 10/31/12 11:40 AM, Eggert, Lars wrote: > >>> On Oct 31, 2012, at 11:35, "Eggert, Lars" > >>> wrote: > >>>> when & where is the bar BOF? Has anyone asked for a room? (I have > >>>> not seen an IRTF-related request, but maybe someone went and asked > >>>> an AD?) > >>> > >>> ah, found it: > >>> > >>>> Meeting Name: Broadband Performance Measurement > >>>> > >>>> Desired Meeting Date: 11/07/2012 > >>>> Alternate Meeting Date: 11/08/2013 > >>>> Number of Days: 1 > >>>> Meeting Area/Type: TSV > >>>> Expected Attendance: 50 > >>>> Desired Start Time: 19:30:00 **Moved to 20:00:00** > >>>> Desired End Time: 20:30:00 > >>>> Room Configuration: Conference > >>>> Speakerphone: No > >>>> LCD Projector: Yes > >>> > >>> That time is in conflict with the IRSG dinner, so IRTF chairs won't > >>> be able to make it. If there are pieces of LMAP that are intended for > >>> the IRTF, someone please clue us in after the meeting. > >>> > >>> Thanks, > >>> Lars > >>> > >>> > >>> > >>> _______________________________________________ > >>> lmap mailing list > >>> lmap@ietf.org > >>> https://www.ietf.org/mailman/listinfo/lmap > >>> > >> _______________________________________________ > >> lmap mailing list > >> lmap@ietf.org > >> https://www.ietf.org/mailman/listinfo/lmap > >> > >> > > > >_______________________________________________ > >lmap mailing list > >lmap@ietf.org > >https://www.ietf.org/mailman/listinfo/lmap > > > This E-mail and any of its attachments may contain Time Warner Cable > proprietary information, which is privileged, confidential, or subject to > copyright belonging to Time Warner Cable. This E-mail is intended solely > for the use of the individual or entity to which it is addressed. If you > are not the intended recipient of this E-mail, you are hereby notified that > any dissemination, distribution, copying, or action taken in relation to > the contents of and attachments to this E-mail is strictly prohibited and > may be unlawful. If you have received this E-mail in error, please notify > the sender immediately and permanently delete the original and any copy of > this E-mail and any printout. > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > --20cf3079bc70d98e2804cd878cee Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Jason for all your support organizing the room and time.

I'm finalizing my travel orders and expect to be there with Hen= ning to help lead the BAR BOF.

I would like to pos= t the final info to the ng-bbperformance list as well when things settle. = =A0We'll all plan for Wednesday 8pm though for the meeting (and assume = the 9pm start is a typo!).

On Fri, Nov 2, 2012 at 10:23 AM, Weil, Jason= <jason.weil@twcable.com> wrote:
Martin Stiemerling and I are meeting with the the IETF Secretariat on
Sunday afternoon to verify the start time and room information for the
meeting scheduled for Wednesday 11/07/2012. I'll post the final
information for the meeting on Sunday afternoon to the list and on the
announcement board once it has been confirmed. The requested start time of<= br> 20:00 was changed to 21:00 in the meeting confirmation email, but this may<= br> just be a typo.

Sorry for the confusion,

Jason



On 11/2/12 2:33 AM, "Benoit Claise" <bclaise@cisco.com> wrote:

>Can we please have a defi= nitive answer on the date/time/location
>
>Regards, Benoit
>> So, does the below tell you at which date it is?
>> It lists a "desired" date and a "alternate date&quo= t;
>> If I look at the doodle poll at
>> =A0 =A0http://www.doodle.com/78t3r4m3mfrea5yp
>> Then only 16 people can make it at the "desired" date >> while 22 can make it at the "alternate" date.
>>
>> So ????
>>
>> On 10/31/12 11:40 AM, Eggert, Lars wrote:
>>> On Oct 31, 2012, at 11:35, "Eggert, Lars" <lars@netapp.com>
>>> =A0 wrote:
>>>> when & where is the bar BOF? Has anyone asked for a ro= om? (I have
>>>> not seen an IRTF-related request, but maybe someone went a= nd asked
>>>> an AD?)
>>>
>>> ah, found it:
>>>
>>>> Meeting Name: Broadband Performance Measurement
>>>>
>>>> Desired Meeting Date: 11/07/2012
>>>> Alternate Meeting Date: 11/08/2013
>>>> Number of Days: 1
>>>> Meeting Area/Type: TSV
>>>> Expected Attendance: 50
>>>> Desired Start Time: 19:30:00 **Moved to 20:00:00**
>>>> Desired End Time: 20:30:00
>>>> Room Configuration: Conference
>>>> Speakerphone: No
>>>> LCD Projector: Yes
>>>
>>> That time is in conflict with the IRSG dinner, so IRTF chairs = won't
>>> be able to make it. If there are pieces of LMAP that are inten= ded for
>>> the IRTF, someone please clue us in after the meeting.
>>>
>>> Thanks,
>>> Lars
>>>
>>>
>>>
>>> _______________________________________________
>>> lmap mailing list
>>> lmap@ietf.org
>>> https://www.ietf.org/mailman/listinfo/lmap
>>>
>> _______________________________________________
>> lmap mailing list
>> lmap@ietf.org
>> https://www.ietf.org/mailman/listinfo/lmap
>>
>>
>
>_______________________________________________
>lmap mailing list
>lmap@ietf.org
>https://www.ietf.org/mailman/listinfo/lmap


This E-mail and any of its attachments= may contain Time Warner Cable proprietary information, which is privileged= , confidential, or subject to copyright belonging to Time Warner Cable. Thi= s E-mail is intended solely for the use of the individual or entity to whic= h it is addressed. If you are not the intended recipient of this E-mail, yo= u are hereby notified that any dissemination, distribution, copying, or act= ion taken in relation to the contents of and attachments to this E-mail is = strictly prohibited and may be unlawful. If you have received this E-mail i= n error, please notify the sender immediately and permanently delete the or= iginal and any copy of this E-mail and any printout.
_____________________________= __________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap

--20cf3079bc70d98e2804cd878cee-- From jamesmilleresquire@gmail.com Sun Nov 4 00:51:45 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 247F221F86C7 for ; Sun, 4 Nov 2012 00:51:45 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JGJDsATDJPPM for ; Sun, 4 Nov 2012 00:51:44 -0700 (PDT) Received: from mail-vb0-f44.google.com (mail-vb0-f44.google.com [209.85.212.44]) by ietfa.amsl.com (Postfix) with ESMTP id 1B25F21F86B7 for ; Sun, 4 Nov 2012 00:51:44 -0700 (PDT) Received: by mail-vb0-f44.google.com with SMTP id fc26so5561622vbb.31 for ; Sun, 04 Nov 2012 00:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=5eQSt+q0SyUCvh42x34obe4nVwqXxPP9Lt+CDS5VwUw=; b=UGicpUaARWX4ahlIS+F9QVOAeyc3+qSCdWYYh5UGPlCzFAgdmiF2DfdS5GEsF79vfB F7WHvEOcLfTsHAkWO0UkUn4gsC5WKzTc0o5U8tlyvI2Gy2YU4qKOo13Hf41AvI6gi7Ap i0QnIbjPFn0qkKYq3po2xmznxQqkchp7CCzLpmd+GJVSi8buF4BfFWJcBNupiaWTL9Ev pPTETuv99d84fzlpoEh3Y+x6F/0sjj6s+syJaHMUp9P8xhGbhFRa/TM+WNMRsvNB/EkW E+qpfZI8KgMWzLb7af22UFyKS0aJo4c20BCjQ8wnR7DviuaaU23ydwifWiTjyAHB6AZn gnbQ== Received: by 10.220.230.133 with SMTP id jm5mr6467243vcb.4.1352015503614; Sun, 04 Nov 2012 00:51:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.200.226 with HTTP; Sun, 4 Nov 2012 00:51:13 -0700 (PDT) In-Reply-To: References: <507EEC05.8000608@it.uc3m.es> From: James Miller Date: Sun, 4 Nov 2012 02:51:13 -0500 Message-ID: To: "Weil, Jason" Content-Type: multipart/alternative; boundary=14dae9cdc6c1d2e45004cda6a3dc Cc: marcelo bagnulo braun , Walter Johnston , "James.Miller@fcc.gov" , Henning Schulzrinne , "lmap@ietf.org" Subject: Re: [lmap] Bar BOF X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 07:51:45 -0000 --14dae9cdc6c1d2e45004cda6a3dc Content-Type: text/plain; charset=ISO-8859-1 Jason, I've received my confirmed travel orders and will be at the meetings from Monday morning (Huzzah!). If you'd like to drop a note tomorrow when the meeting time is sorted I'd like to get a notice to the ng-bbperformance list as well. Feel free to post there or I will. Thanks again. On Wed, Oct 17, 2012 at 1:56 PM, Weil, Jason wrote: > All, > > I requested a meeting room for this discussion under the TSV area late > last week based on a discussion with the authors and a number of potential > participants at a meeting on 10.10.2012. I just received word that it was > approved this morning and the information has been forwarded to the IETF > Secretariat for scheduling. I originally scheduled the room for > 19:30-20:30 on 10.7.2012, but this time was pushed back 30 mins to > 20:00-21:00 as the Plenary technically is scheduled to end at 19:45. > > Here is the information from the meeting request: > > Meeting Name: Broadband Performance Measurement > > Desired Meeting Date: 11/07/2012 > Alternate Meeting Date: 11/08/2013 > Number of Days: 1 > Meeting Area/Type: TSV > Expected Attendance: 50 > Desired Start Time: 19:30:00 **Moved to 20:00:00** > Desired End Time: 20:30:00 > Room Configuration: Conference > Speakerphone: No > LCD Projector: Yes > > > > > Thanks, > > Jason Weil > > > > > On 10/17/12 1:33 PM, "marcelo bagnulo braun" wrote: > > >Hi, > > > >Any news about the Bar BOF? > >Now that the IETF agenda is fixed, do we have a date and time for the > >Bar BOF? > > > >Shouldnt we start discussing the agenda for the meeting? > > > >Regards, marcelo > > > >_______________________________________________ > >lmap mailing list > >lmap@ietf.org > >https://www.ietf.org/mailman/listinfo/lmap > > > This E-mail and any of its attachments may contain Time Warner Cable > proprietary information, which is privileged, confidential, or subject to > copyright belonging to Time Warner Cable. This E-mail is intended solely > for the use of the individual or entity to which it is addressed. If you > are not the intended recipient of this E-mail, you are hereby notified that > any dissemination, distribution, copying, or action taken in relation to > the contents of and attachments to this E-mail is strictly prohibited and > may be unlawful. If you have received this E-mail in error, please notify > the sender immediately and permanently delete the original and any copy of > this E-mail and any printout. > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > --14dae9cdc6c1d2e45004cda6a3dc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Jason,

I've received my confirmed travel orders and = will be at the meetings from Monday morning (Huzzah!). =A0If you'd like= to drop a note tomorrow when the meeting time is sorted I'd like to ge= t a notice to the ng-bbperformance list as well. =A0Feel free to post there= or I will.

Thanks again.

On Wed, Oct 17, 20= 12 at 1:56 PM, Weil, Jason <jason.weil@twcable.com> wro= te:
All,

I requested a meeting room for this discussion under the TSV area late
last week based on a discussion with the authors and a number of potential<= br> participants at a meeting on 10.10.2012. I just received word that it was approved this morning and the information has been forwarded to the IETF Secretariat for scheduling. I originally scheduled the room for
19:30-20:30 on 10.7.2012, but this time was pushed back 30 mins to
20:00-21:00 as the Plenary technically is scheduled to end at 19:45.

Here is the information from the meeting request:

Meeting Name: Broadband Performance Measurement

Desired Meeting Date: 11/07/2012
Alternate Meeting Date: 11/08/2013
Number of Days: 1
Meeting Area/Type: TSV
Expected Attendance: 50
Desired Start Time: 19:30:00 **Moved to 20:00:00**
Desired End Time: 20:30:00
Room Configuration: Conference
Speakerphone: No
LCD Projector: Yes




Thanks,

Jason Weil




On 10/17/12 1:33 PM, "marcelo bagnulo braun" <marcelo@it.uc3m.es> wrote:

>Hi,
>
>Any news about the Bar BOF?
>Now that the IETF agenda is fixed, do we have a date and time for the >Bar BOF?
>
>Shouldnt we start discussing the agenda for the meeting?
>
>Regards, marcelo
>
>_______________________________________________
>lmap mailing list
>lmap@ietf.org
>https://www.ietf.org/mailman/listinfo/lmap


This E-mail and any of its attachments may contain Time Warner = Cable proprietary information, which is privileged, confidential, or subjec= t to copyright belonging to Time Warner Cable. This E-mail is intended sole= ly for the use of the individual or entity to which it is addressed. If you= are not the intended recipient of this E-mail, you are hereby notified tha= t any dissemination, distribution, copying, or action taken in relation to = the contents of and attachments to this E-mail is strictly prohibited and m= ay be unlawful. If you have received this E-mail in error, please notify th= e sender immediately and permanently delete the original and any copy of th= is E-mail and any printout.
___________________________________= ____________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap

--14dae9cdc6c1d2e45004cda6a3dc-- From jason.weil@twcable.com Sun Nov 4 13:19:23 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8257521F88D2; Sun, 4 Nov 2012 13:19:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.203 X-Spam-Level: X-Spam-Status: No, score=0.203 tagged_above=-999 required=5 tests=[AWL=-0.194, BAYES_20=-0.74, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x+IqIhekSCiv; Sun, 4 Nov 2012 13:19:23 -0800 (PST) Received: from cdpipgw01.twcable.com (cdpipgw01.twcable.com [165.237.59.22]) by ietfa.amsl.com (Postfix) with ESMTP id B19B421F8914; Sun, 4 Nov 2012 13:19:22 -0800 (PST) X-SENDER-IP: 10.136.163.12 X-SENDER-REPUTATION: None X-IronPort-AV: E=Sophos;i="4.80,711,1344225600"; d="scan'208,217";a="464737276" Received: from unknown (HELO PRVPEXHUB03.corp.twcable.com) ([10.136.163.12]) by cdpipgw01.twcable.com with ESMTP/TLS/RC4-MD5; 04 Nov 2012 16:18:43 -0500 Received: from PRVPEXVS12.corp.twcable.com ([10.136.163.45]) by PRVPEXHUB03.corp.twcable.com ([10.136.163.12]) with mapi; Sun, 4 Nov 2012 16:19:19 -0500 From: "Weil, Jason" To: "lmap@ietf.org" Date: Sun, 4 Nov 2012 16:19:19 -0500 Thread-Topic: LMAP BarBOF Meeting Information - Wednesday, 11.7.12 20:00-21:30 Thread-Index: Ac260g1OOycYzQyqRQqseUwyI/2ZGQ== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.10.0.110310 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CCBC460C9044jasonweiltwcablecom_" MIME-Version: 1.0 Cc: "ippm@ietf.org" Subject: [lmap] LMAP BarBOF Meeting Information - Wednesday, 11.7.12 20:00-21:30 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 21:19:23 -0000 --_000_CCBC460C9044jasonweiltwcablecom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable WHAT: Large-Scale Measurement of Broadband Performance (LMAP) DATE: Wednesday, 11/07/2012 TIME: 20:00 =96 21:30 LOCATION: SALON B A much larger room has been provided capable of holding up to 170 people. F= eel free to bring anyone interested in providing comments on the discussion= . A projector will be available for anyone interested in presenting slides.= Authors of the draft-shuzrinne-lmap-requirements-00 document will help lea= d the discussion and information on work progressing in the Broadband Forum= in this area will be discussed as well. Thank You, Jason ________________________________ This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. --_000_CCBC460C9044jasonweiltwcablecom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

WHAT: = Large-Scale Measurement of Broadband Performance (LMAP)
DATE: = Wednesday, 11/07/2012
TIME: = 20:00 =96 21:30
LOCATION: SALON B

A much larger room has been provided capable of holding up to 170 peop= le. Feel free to bring anyone interested in providing comments on the discu= ssion. A projector will be available for anyone interested in presenting sl= ides. Authors of the draft-shuzrinne-lmap-requirements-00 document will help lead the discussion and information on work progressing= in the Broadband Forum in this area will be discussed as well.

Thank You,

Jason

This E-mail and any of its a= ttachments may contain Time Warner Cable proprietary information, which is = privileged, confidential, or subject to copyright belonging to Time Warner = Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you a= re not the intended recipient of this E-mail, you are hereby notified that = any dissemination, distribution, copying, or action taken in relation to th= e contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have receiv= ed this E-mail in error, please notify the sender immediately and permanent= ly delete the original and any copy of this E-mail and any printout.
 --_000_CCBC460C9044jasonweiltwcablecom_-- From Henning.Schulzrinne@fcc.gov Sun Nov 4 14:30:13 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 066E221F8787 for ; Sun, 4 Nov 2012 14:30:13 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.348 X-Spam-Level: X-Spam-Status: No, score=-4.348 tagged_above=-999 required=5 tests=[AWL=-2.252, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, MIME_BASE64_TEXT=1.753, MIME_CHARSET_FARAWAY=2.45, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LCEzM73swqwl for ; Sun, 4 Nov 2012 14:30:12 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id 8CB8221F8785 for ; Sun, 4 Nov 2012 14:30:12 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 17:30:10 -0500 From: Henning Schulzrinne To: =?gb2312?B?x+zA2g==?= , "lmap@ietf.org" Thread-Topic: [lmap] what parameters provide the network parameter server? Thread-Index: Ac2x5VyO3ZbY2Em1R2GuFIISYI/bkwI9cht/ Date: Sun, 4 Nov 2012 22:30:08 +0000 Message-ID: References: <000001cdb1e6$24415bf0$6cc413d0$@bupt.edu.cn> In-Reply-To: <000001cdb1e6$24415bf0$6cc413d0$@bupt.edu.cn> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Subject: Re: [lmap] what parameters provide the network parameter server? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2012 22:30:13 -0000 VGhlIG5ldHdvcmsgcGFyYW1ldGVyIHNlcnZlciBpcyBtZWFudCB0byBwcm92aWRlIGluZm9ybWF0 aW9uIHRvIHRoZSBzdWJzY3JpYmVyIGFib3V0IHRoZWlyIG5ldHdvcmsgc3Vic2NyaXB0aW9uLCBz dWNoIGFzIHRoZWlyIG5vbWluYWwgc3BlZWQuIEFzIGFuIGV4YW1wbGUsIHlvdSBjb3VsZCBpbWFn aW5lIHRoYXQgaXQgbWlnaHQgcmV0dXJuIHNvbWV0aGluZyBsaWtlOg0KDQpET0NTSVMgMy4wDQpk b3duc3RyZWFtOiAyNSBNYi9zDQp1cHN0cmVhbTogMTAgTWIvcw0KUHJvdmlkZXI6IFNvbWVJU1Ag SW5jLg0KDQpJbiBzb21lIGNhc2VzLCB0aGlzIG1heSBhbHJlYWR5IGJlIGF2YWlsYWJsZSB0aHJv dWdoIHRoZSBUUjY5IHByb3RvY29sLCBidXQgdGhhdCdzIG9uZSBvZiB0aGUgcXVlc3Rpb25zIHRv IGJlIGFkZHJlc3NlZCBhcyB3ZSBnbyBmb3J3YXJkLg0KDQpIZW5uaW5nDQooY2F0Y2hpbmcgdXAg b24gbGlzdCBlbWFpbCkNCg0KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkZyb206 IGxtYXAtYm91bmNlc0BpZXRmLm9yZyBbbG1hcC1ib3VuY2VzQGlldGYub3JnXSBvbiBiZWhhbGYg b2Ygx+zA2iBbcXFsQGJ1cHQuZWR1LmNuXQ0KU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDI0LCAy MDEyIDg6NTAgQU0NClRvOiBsbWFwQGlldGYub3JnDQpTdWJqZWN0OiBbbG1hcF0gd2hhdCBwYXJh bWV0ZXJzIHByb3ZpZGUgdGhlIG5ldHdvcmsgcGFyYW1ldGVyIHNlcnZlcj8NCg0KSGksIGFsbA0K DQpJIGhhdmUgcmVhZCB0aGUgZHJhZnQgYW5kIGNvbW1lbnQgYWJvdXQgdGhlIG5ldHdvcmsgcGFy YW1ldGVyIHNlcnZlcnMuIEJ1dCBJIHN0aWxsIGhhdmUgYSBxdWVzdGlvbi4gV2hhdCBwYXJhbWV0 ZXJzIHByb3ZpZGUgdGhlIG5ldHdvcmsgcGFyYW1ldGVyIHNlcnZlcj8gQWZ0ZXIgYWxsLCB0aGUg bmFtZSBvZiB0aGUgbmV0d29yayBwYXJhbWV0ZXIgc2VydmVyIGxlYWQgbWUgdG8gYmVsaWV2ZSB0 aGF0IGl0IGNhbiBhbmQgc2hvdWxkIHByb3ZpZGUgc29tZSBwYXJhbWV0ZXIuDQoNClJlZ2FyZHMs IFFpbmdsZWkgUWkNCg== From Henning.Schulzrinne@fcc.gov Sun Nov 4 16:28:25 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C2FB221F85DD for ; Sun, 4 Nov 2012 16:28:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.473 X-Spam-Level: X-Spam-Status: No, score=-5.473 tagged_above=-999 required=5 tests=[AWL=1.126, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kEW2gAoLhV7D for ; Sun, 4 Nov 2012 16:28:25 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id 2087521F85D6 for ; Sun, 4 Nov 2012 16:28:24 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 19:28:23 -0500 From: Henning Schulzrinne To: Al Morton , Robert Kisteleki , "lmap@ietf.org" Thread-Topic: [lmap] About draft-schulzrinne-lmap-requirements-00 Thread-Index: AQHNmvIByk7Rb0svmUavVq/u2HxL9JebSFeAgAAd1ACAAAUdAIABeRUAgD284r0= Date: Mon, 5 Nov 2012 00:28:22 +0000 Message-ID: References: <50616167.4080907@it.uc3m.es> <7.0.1.0.0.20120925085242.03e715b8@att.com> <5061C3B2.2010602@it.uc3m.es> <5061C7FC.1090500@ripe.net>, <7.0.1.0.0.20120926091839.03e71ad8@att.com> In-Reply-To: <7.0.1.0.0.20120926091839.03e71ad8@att.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] About draft-schulzrinne-lmap-requirements-00 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 00:28:25 -0000 Picking up some random short comments:=0A= =0A= For the parameter retrieval, there is an authentication problem - the clien= t (consumer) generally presumably has legitimate access to its own network = parameters, but the ISP has no way to validate a third party and ISPs aren'= t likely to just answer a random query for "Hi, what's the nominal speed fo= r customer 1.2.3.4?". This isn't as much of a problem if the third party is= a national regulatory agency (identification for NATed customers may be), = but is a serious issue if it's a private third-party entity chosen by the e= nd user. Thus, I believe that end system access to this data is required fo= r simplicity and functionality, even if others should also have (authentica= ted) access to the data.=0A= =0A= On NATs, for most active measurements, these are inside-out, so this may no= t require much NAT or firewall interaction. I'm also rather worried about g= iving a measurement client authority to open up firewalls - this can be use= d for mischief. Plus, I'd imagine that we would use any existing NAT/firewa= ll control mechanism for this, rather than building a new one.=0A= =0A= Henning=0A= =0A= ________________________________________=0A= From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al Morton = [acmorton@att.com]=0A= Sent: Wednesday, September 26, 2012 9:34 AM=0A= To: Robert Kisteleki; lmap@ietf.org=0A= Subject: Re: [lmap] About draft-schulzrinne-lmap-requirements-00=0A= =0A= Hi Robert,=0A= =0A= =0A= The details of network parameters would be known at=0A= other testing entities, but not the edge client/agent.=0A= =0A= But the requirement to open the firewall and establish NAT state falls=0A= to the edge client/agent. That's a new wrinkle and needs to be sorted-out= =0A= with the rest of the details.=0A= =0A= regards,=0A= Al=0A= =0A= _______________________________________________=0A= lmap mailing list=0A= lmap@ietf.org=0A= https://www.ietf.org/mailman/listinfo/lmap=0A= From Henning.Schulzrinne@fcc.gov Sun Nov 4 16:45:31 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1200621F880F for ; Sun, 4 Nov 2012 16:45:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.848 X-Spam-Level: X-Spam-Status: No, score=-5.848 tagged_above=-999 required=5 tests=[AWL=0.751, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HKHCToZgbRnU for ; Sun, 4 Nov 2012 16:45:30 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id 39EE821F880D for ; Sun, 4 Nov 2012 16:45:30 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas03.fccnet.win.fcc.gov ([fe80::8db2:ac93:84fc:c36b%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 19:45:29 -0500 From: Henning Schulzrinne To: marcelo bagnulo braun , "lmap@ietf.org" Thread-Topic: [lmap] About draft-schulzrinne-lmap-requirements-00 - controller scheduling Thread-Index: AQHNuu7a0oL1mxgIu0mNNQjOdi7i3g== Date: Mon, 5 Nov 2012 00:45:28 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] About draft-schulzrinne-lmap-requirements-00 - controller scheduling X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 00:45:31 -0000 As a general comment which may color some of the perceptions: =0A= =0A= There are probably two kinds of measurement infrastructures: full-time ("pr= ofessional") and occasional. ATLAS and the current FCC system are examples = of full-time, i.e., the clients are measuring and gathering data all the ti= me. However, as we scale up these systems and build them into every device = (modem, mobile, ...), each individual box is going to do measurements only = rarely: for example, it might get randomly "drafted" if it volunteered to b= e part of a broader measurement study, the user might suspect that somethin= g isn't working right or the ISP is debugging a particular end system. You = need coordination for the full-time case, but for the occasional one, it's = probably sufficient and much less complex for the client that gets asked by= chance by a second controller to just say "sorry, I'm busy already".=0A= =0A= Measurement coordination gets close to some of the GENI protocols and sched= uler work, and might require much fancier algorithms, such as prioritizatio= n, calendar-like scheduling ("please let me use your nodes on Tuesday") and= resource allocation ("Dear controller: I need 500 measurement nodes"). GEN= I is developing and deploying a number of generic experimental resource con= trol protocols and interfaces (ORCA is one), and those may well be applicab= le for those cases.=0A= =0A= May I also suggest that we start separate threads for new topics, to keep t= he messages focused?=0A= =0A= Henning=0A= =0A= ________________________________________=0A= From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of marcelo ba= gnulo braun [marcelo@it.uc3m.es]=0A= Sent: Tuesday, September 25, 2012 3:46 AM=0A= To: lmap@ietf.org=0A= Subject: [lmap] About draft-schulzrinne-lmap-requirements-00=0A= =0A= Hi,=0A= =0A= Thanks for posting the draft and starting the discussion.=0A= =0A= Some comments and questions.=0A= =0A= About the Multi-provider network measurements support. In the draft you=0A= mention that you want to support this, even though it is beyond the=0A= scope of the document. I guess that the way the architecture is defined=0A= will implicitly enable some forms of collaboration and made other forms=0A= of collaboration harder. As per the protocols defined in the current=0A= document, I understand you could support that Measurements clients=0A= (M-clients for short) interacting with M-servers and M-controllers and=0A= even Data collectors from different organizations. This would allow for=0A= different organizations to schedule tests in the M-clients.=0A= However, there is no protocol for the interaction between Data=0A= collectors from different entities. I guess this would be abohter way to=0A= enable interaction between different providers, where they coordiante=0A= through the M-controller to M-controller protocol a set of tests to be=0A= executed in different (disjoint) sets of clients and then the data=0A= collectros exchange the results of the tests. I guess you could achieve=0A= the same thing by allowing the M-clients to report to multiple data=0A= collectors, but i would assume you want to minimize the traffic sent by=0A= the M-clients.=0A= This would allow for a sort of federation of multiple measurement=0A= platforms, that i would find appealing at first sight. So, I guess my=0A= question is whether you discarded this type of scenario for some reason,=0A= or you simply believe that the data collector to data collector protocol=0A= could reuse an existent protocol or else?=0A= =0A= About the Network paramenter server. I am not sure I understand what is=0A= the role you have in mind for this element.=0A= In the definition, you state that:=0A= " In some of the use cases, it is necessary for the analysis to compare= =0A= the measured against the nominal network performance, or correlate=0A= measured parameters with the type and key parameters of the userOs=0A= network connection."=0A= =0A= This may lead to think that the network parameter server provides=0A= information that is relevant for the analysis of the data rather than=0A= for the performance of the tests. However, the only protocol defiend=0A= involving the network parameter server is with the M-client, which=0A= implies that the parameters are needed to actually perform the tests=0A= rather than analyzing the data. I think this should be clarified.=0A= Even in the case the parameters are needed to perform the tests, it is=0A= not clear to me why should the clients retrieve the parameters directly=0A= from the Network parameters server rather than obtain it from the=0A= M-controller along with the tests. I mean, defining a protocol between=0A= the M-controller and the Network parameter server rather than a protocol=0A= between the network parameter server and the M-cllients. Could you=0A= expnad on the rationale for this?=0A= =0A= =0A= Regards, marcelo=0A= =0A= _______________________________________________=0A= lmap mailing list=0A= lmap@ietf.org=0A= https://www.ietf.org/mailman/listinfo/lmap=0A= From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:12:56 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 37B9D21F8817 for ; Sun, 4 Nov 2012 17:12:56 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.036 X-Spam-Level: X-Spam-Status: No, score=-6.036 tagged_above=-999 required=5 tests=[AWL=0.563, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n-i1ujI3ZDo7 for ; Sun, 4 Nov 2012 17:12:55 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id A7A8321F8782 for ; Sun, 4 Nov 2012 17:12:55 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:12:54 -0500 From: Henning Schulzrinne To: Shane Amante , "lmap@ietf.org" Thread-Topic: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - file formats and retrieval Thread-Index: AQHNuvKuzCYBuLMSwU6LbdrrU66zPg== Date: Mon, 5 Nov 2012 01:12:53 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - file formats and retrieval X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:12:56 -0000 Inline, in color, since Outlook doesn't do >. ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Shane Aman= te [shane@castlepoint.net] Sent: Tuesday, September 25, 2012 12:06 PM To: lmap@ietf.org Subject: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 1) It seems critical to look at creation of a data model and/or file format= that allows this architecture to bundle together *both* the measurement pa= rameters/criteria used as input to a measurement test /and/ the measurement= results collected as part of that particular measurement run. This will be= critical if/when measurement results are analyzed historically, particular= ly by third-parties (e.g.: researchers) who may not have originally initiat= ed the tests. If an existing format (e.g., IPFIX) is used for exchange, this might solve = itself. My inclination would be to stick as close as possible to the on-the= -wire format, with record delimiters, as this avoids having to evolve two f= ormats or worry about that new on-the-wire capabilities can't be represente= d in the file, which would obviously be rather unpleasant. 3) The draft doesn't appear to mention anything about providing an ability = for entities to query for and attain historical measurements that have alre= ady been collected. (With respect to "query for", again, it would be nice t= o have a data model that represented the measurement parameters used as inp= ut to a test so those could help winnow down the data sets that match a giv= en criteria of interest that is to analyzed. See #1.) Presumably, this woul= d be a valuable capability. Is this within the purview of the draft/archite= cture? There seem to be two cases here: Retrieve the whole file (for which ftp or = similar would presumably suffice) or a more elaborate query, as in "Give me= all measurements for client X during the 6-7 pm UTC hour, measurement 42" = or even "Give me all measurements ... that have a value greater than 55 ms.= " The latter seems to get awfully close to an SQL-like query language. Inde= ed, not too surprisingly, this is what the current FCC infrastructure uses.= Or did you have something else in mind? Henning From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:18:59 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BCEF121F880D for ; Sun, 4 Nov 2012 17:18:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.149 X-Spam-Level: X-Spam-Status: No, score=-6.149 tagged_above=-999 required=5 tests=[AWL=0.450, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w0ViUdSzVmqv for ; Sun, 4 Nov 2012 17:18:58 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id 5822D21F880B for ; Sun, 4 Nov 2012 17:18:58 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:18:57 -0500 From: Henning Schulzrinne To: Shane Amante , "lmap@ietf.org" Thread-Topic: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - conflict resolution Thread-Index: AQHNuvNjCnx5bQjcUECKMsZt5+u0Qg== Date: Mon, 5 Nov 2012 01:18:55 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - conflict resolution X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:18:59 -0000 4) The draft also doesn't discuss how conflict resolution among tests reque= sted to be scheduled will be performed. Specifically, is the client allowed= to decide, on its own, how to deconflict two tests that may have been sche= duled at/near the same time? If so, how? Presumably a single algorithm coul= d be used for conflict resolution at either the measurement controller or t= he client-end ... or, both ... Likewise, if a measurement controller has as= ked a client to perform a test, but the client is unable to do so because o= f a conflict, then it should report that to the measurement controller so t= he controller can: stop asking the client to do the test, re-schedule it at= a different time or take some other action, e.g.: select a wholly differen= t client. Indeed, that is clearly one of the error cases that need to be handled. A c= ommon case is that the in-home client is busy doing "real" work and doesn't= want to do a measurement and often cannot do so without either interfering= with the non-measurement traffic or being affected by it. Generally, given= that the client is the only entity that knows the local traffic, it seems = to need to make the decision - after all, practically speaking, it's in con= trol. If it doesn't "want to" do measurements, all the pleading of the cont= roller can be ignored by the client and there's nothing the controller can = do about it. I think it would be useful to discuss whether, say, a privileg= ed controller (the ISP?) can tell the client to stop a running measurement,= either because it's causing network harm or because the ISP needs to run a= diagnostic test. Thanks, -shane _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:22:35 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CBF4D21F8817 for ; Sun, 4 Nov 2012 17:22:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.066 X-Spam-Level: X-Spam-Status: No, score=-6.066 tagged_above=-999 required=5 tests=[AWL=0.218, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, SARE_MILLIONSOF=0.315] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V+alQtuVzbUx for ; Sun, 4 Nov 2012 17:22:35 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id E2C8421F880D for ; Sun, 4 Nov 2012 17:22:34 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:22:33 -0500 From: Henning Schulzrinne To: Shane Amante , "lmap@ietf.org" Thread-Topic: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - redundancy Thread-Index: AQHNuvQHovrSV0mV10qPeQrGms4Mwg== Date: Mon, 5 Nov 2012 01:22:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - redundancy X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:22:35 -0000 5) Finally, it might be nice to consider that for redundancy and/or perform= ance reasons a singular measurement test may simultaneously write results t= o several measurement data collectors. Alternatively, is the vision that a = single measurement data collector will receive results and, in the backgrou= nd?, distribute that to multiple measurement data collectors? Ultimately, t= his is a discussion about the necessity of caching of measurement data, whi= ch becomes important to consider as either: a) the data set inevitably grow= s larger (into millions of nodes?); and/or, b) entities wish to consume his= torical data-sets that inevitably will be extremely large. Good point and I see no reason to limit writing the results to one server. = As you note, measurement servers can also back up files in the background. = That can presumably use standard file synchronization mechanisms (e.g., rsy= nc), but would increase the size of the failure unit from a single tuple to= a whole measurement run. The obvious downside is that this increases the t= raffic from the client to the collectors. -shane _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:37:08 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4FA5921F881B for ; Sun, 4 Nov 2012 17:37:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.255 X-Spam-Level: X-Spam-Status: No, score=-6.255 tagged_above=-999 required=5 tests=[AWL=0.344, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OOQBorJSzo4u for ; Sun, 4 Nov 2012 17:37:07 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id EFFFB21F8828 for ; Sun, 4 Nov 2012 17:37:06 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas02.fccnet.win.fcc.gov ([fe80::2d69:114:8552:212f%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:37:05 -0500 From: Henning Schulzrinne To: Brian Trammell , "lmap@ietf.org" Thread-Topic: [lmap] comments on lmap-requirements - re-use Thread-Index: AQHNuvYPmw9A0Jim2ES++QTjgXYduw== Date: Mon, 5 Nov 2012 01:37:05 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] comments on lmap-requirements - re-use X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:37:08 -0000 Comments in-line. ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Brian Tram= mell [trammell@tik.ee.ethz.ch] Sent: Tuesday, September 25, 2012 12:08 PM To: lmap@ietf.org Subject: [lmap] comments on lmap-requirements Is the intention of LMAP to design new measurement methodologies and protoc= ols, to overlay control on an existing set of measurement protocols, or to = provide a framework/control system for controlling existing measurement sys= tems? The assumption seems to be the first, though I'm not sure it's a good= idea to start from scratch. There's a lot of work in internet measurement = (both protocols and tools, including already-deployed active and passive me= asurement systems) that could be leveraged to make an LMAP-based infrastruc= ture more quickly deployable. This is probably something that needs to be made clearer. My goal was suppo= rt integration as much as possible, e.g., the actual probing (for active me= asurements) should be able to use any measurement technique and protocol. You could and should be able to combine LMAP-standard and existing protocol= s for the other elements, e.g., maybe the controller-client protocol is inf= rastructure-specific and only the reporting protocol is LMAP-standardized. Best regards, Brian _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:38:37 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0420921F86DD for ; Sun, 4 Nov 2012 17:38:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.298 X-Spam-Level: X-Spam-Status: No, score=-6.298 tagged_above=-999 required=5 tests=[AWL=0.301, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kQjENW+u6N6w for ; Sun, 4 Nov 2012 17:38:36 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id E380B21F883C for ; Sun, 4 Nov 2012 17:38:33 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas03.fccnet.win.fcc.gov ([fe80::8db2:ac93:84fc:c36b%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:38:33 -0500 From: Henning Schulzrinne To: Brian Trammell , "lmap@ietf.org" Thread-Topic: [lmap] comments on lmap-requirements - terminology Thread-Index: AQHNuvZDKTxOko23kUKUy6DjtAduIA== Date: Mon, 5 Nov 2012 01:38:32 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] comments on lmap-requirements - terminology X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:38:37 -0000 Sent: Tuesday, September 25, 2012 12:08 PM In these cases, as Marcelo and Al point out, the definition of client and s= erver are probably not quite right / too specific. Indeed, if the measureme= nt protocol itself is a "pluggable" thing, then the measurement client and = server aren't really part of the LMAP architecture; there's simply a measur= ement _point_ -- if that's an active client, or an active server, or a pass= ive observation point isn't really relevant, and the other side of the meas= urement is simply a parameter of the request to the measurement point. I agree that the terminology could use work. I see two types of entities - = the large-scale (possibly millions) entity that largely does the bidding of= the controller(s), and the much smaller number of controllers. The measure= ment server (as named currently) is really only necessary for active measur= ements, so suggestions for a name are appreciated. "Active measurement sink= " is one option. Best regards, Brian From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:40:25 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4C46021F8853 for ; Sun, 4 Nov 2012 17:40:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.331 X-Spam-Level: X-Spam-Status: No, score=-6.331 tagged_above=-999 required=5 tests=[AWL=0.268, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TTB8IoqTvcCU for ; Sun, 4 Nov 2012 17:40:24 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id B9B7121F86B7 for ; Sun, 4 Nov 2012 17:40:24 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas03.fccnet.win.fcc.gov ([fe80::8db2:ac93:84fc:c36b%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:40:24 -0500 From: Henning Schulzrinne To: Brian Trammell , "lmap@ietf.org" Thread-Topic: [lmap] comments on lmap-requirements - controller to measurement point Thread-Index: AQHNuvaFIHohmAAAEE+A/thzkN8fRQ== Date: Mon, 5 Nov 2012 01:40:22 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] comments on lmap-requirements - controller to measurement point X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:40:25 -0000 Second, is there a reason that direct connections from a measurement contro= ller to a measurement point are not supported? This argument that this redu= ces DoS potential is not convincing, given that the direction of connection= initiation and the final decision to authorize and initiate a measurement = need not be dependent. What it does do is keep a measurement controller fro= m being able to do iterative measurement: i.e., using the results of a prio= r measurement to direct the parameters of a subsequent measurement, as woul= d be useful when trying to isolate an intermittent connection failure in th= e user diagnostics use case. Good point. My other concern were NAT issues - for agents in home network d= evices, reaching them from the outside is likely to be challenging, but thi= s doesn't apply to (cable|DSL) modems and may be easier in the case of IPv6= networks. Brian From Henning.Schulzrinne@fcc.gov Sun Nov 4 17:52:08 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 01C5321F860D for ; Sun, 4 Nov 2012 17:52:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.358 X-Spam-Level: X-Spam-Status: No, score=-6.358 tagged_above=-999 required=5 tests=[AWL=0.241, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3eiHpt1WvSe3 for ; Sun, 4 Nov 2012 17:52:07 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id E4D3021F8604 for ; Sun, 4 Nov 2012 17:52:06 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0421.002; Sun, 4 Nov 2012 20:52:05 -0500 From: Henning Schulzrinne To: KEN KO , "lmap@ietf.org" Thread-Topic: Comments on draft-schulzrinne-lmap-requirements-00 - capabilities Thread-Index: AQHNuvgn1fW1aM/tCE2oMdmTrAZ/lw== Date: Mon, 5 Nov 2012 01:52:03 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.64] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - capabilities X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 01:52:08 -0000 Picking up a few items that haven't been discussed yet. (I generally agree = with the comments that I cut out.) There is a need to identify different types of measurement clients to the m= easurement controller. Some clients, for instance located at residential or= business gateways adjacent to UNIs, are primary candidates for scheduled t= est activities. Other clients located at intermediate nodes within the netw= ork may be used only for troubleshooting, or at least may have a different = set of tests scheduled than a gateway. Still others such as software-based = clients installed on fixed or mobile endpoint devices might serve either pu= rpose, depending on whether there the associated gateway is a measurement c= lient and whether the endpoint is considered =93reliable=94 for the purpose= s of a specific test program. Classification of measurement client both by = feature set and by network location may be desirable. Would you envision that this requires more than just returning "sorry, I ca= n't do this" error messages, e.g., if there's a request for a scheduled tes= t for nodes that only do on-demand tests? Do you see fundamental difference= s in control protocols? I do envision a discovery query where measurement c= lients (or whatever we end up calling them) can describe their capabilities= , primarily in terms of tests they can run, including passive ones, but may= be also in terms of overall capability. Beyond scheduled tests and the abil= ity to use a publish/subscribe mechanism, are there other examples? Best regards, Ken Ko From robert@ripe.net Mon Nov 5 00:14:47 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 077AB21F8A53 for ; Mon, 5 Nov 2012 00:14:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o8gOyA+ccC5U for ; Mon, 5 Nov 2012 00:14:46 -0800 (PST) Received: from postgirl.ripe.net (postgirl.ipv6.ripe.net [IPv6:2001:67c:2e8:11::c100:1342]) by ietfa.amsl.com (Postfix) with ESMTP id EC9E221F8A25 for ; Mon, 5 Nov 2012 00:14:45 -0800 (PST) Received: from dodo.ripe.net ([193.0.23.4]) by postgirl.ripe.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1TVHpR-0004zF-Vs; Mon, 05 Nov 2012 09:14:43 +0100 Received: from baboon.ripe.net ([193.0.1.208] helo=Kistel-Mac.local) by dodo.ripe.net with esmtp (Exim 4.72) (envelope-from ) id 1TVHpR-0000KD-Se; Mon, 05 Nov 2012 09:14:41 +0100 Message-ID: <50977571.9030007@ripe.net> Date: Mon, 05 Nov 2012 09:14:41 +0100 From: Robert Kisteleki Organization: RIPE NCC User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: Henning Schulzrinne References: In-Reply-To: X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Anti-Virus: Kaspersky Anti-Virus for Linux Mail Server 5.6.48/RELEASE, bases: 20120425 #7816575, check: 20121105 clean X-RIPE-Spam-Level: --- X-RIPE-Spam-Report: Spam Total Points: -3.3 points pts rule name description ---- ---------------------- ------------------------------------ -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -0.4 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-RIPE-Signature: 72e00e6d7601fa19264e98abc238a27412703fbb43a717d98037555e68b1dc7f Cc: Shane Amante , "lmap@ietf.org" Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - file formats and retrieval X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 08:14:47 -0000 As a meta-thread: either the mailing list is wonky, or Henning wrote about a dozen or so emails lately with *many* different signatures :-) Also: without quoting it's very difficult to follow which parts are from the original mails. See below. Colors don't work well in text mails. Robert On 2012.11.05. 2:12, Henning Schulzrinne wrote: > Inline, in color, since Outlook doesn't do >. > > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Shane Amante [shane@castlepoint.net] > Sent: Tuesday, September 25, 2012 12:06 PM > To: lmap@ietf.org > Subject: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 > > 1) It seems critical to look at creation of a data model and/or file format that allows this architecture to bundle together *both* the measurement parameters/criteria used as input to a measurement test /and/ the measurement results collected as part of that particular measurement run. This will be critical if/when measurement results are analyzed historically, particularly by third-parties (e.g.: researchers) who may not have originally initiated the tests. > > If an existing format (e.g., IPFIX) is used for exchange, this might solve itself. My inclination would be to stick as close as possible to the on-the-wire format, with record delimiters, as this avoids having to evolve two formats or worry about that new on-the-wire capabilities can't be represented in the file, which would obviously be rather unpleasant. > > > 3) The draft doesn't appear to mention anything about providing an ability for entities to query for and attain historical measurements that have already been collected. (With respect to "query for", again, it would be nice to have a data model that represented the measurement parameters used as input to a test so those could help winnow down the data sets that match a given criteria of interest that is to analyzed. See #1.) Presumably, this would be a valuable capability. Is this within the purview of the draft/architecture? > > There seem to be two cases here: Retrieve the whole file (for which ftp or similar would presumably suffice) or a more elaborate query, as in "Give me all measurements for client X during the 6-7 pm UTC hour, measurement 42" or even "Give me all measurements ... that have a value greater than 55 ms." The latter seems to get awfully close to an SQL-like query language. Indeed, not too surprisingly, this is what the current FCC infrastructure uses. Or did you have something else in mind? > > Henning > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > From Henning.Schulzrinne@fcc.gov Mon Nov 5 05:08:37 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6E48D21F85DD for ; Mon, 5 Nov 2012 05:08:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.38 X-Spam-Level: X-Spam-Status: No, score=-6.38 tagged_above=-999 required=5 tests=[AWL=0.219, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iq53QDJ-eaat for ; Mon, 5 Nov 2012 05:08:36 -0800 (PST) Received: from dmz-mail2.fcc.gov (dmz-mail2.fcc.gov [192.104.54.11]) by ietfa.amsl.com (Postfix) with ESMTP id 145E421F8518 for ; Mon, 5 Nov 2012 05:08:32 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas03.fccnet.win.fcc.gov ([fe80::8db2:ac93:84fc:c36b%13]) with mapi id 14.01.0421.002; Mon, 5 Nov 2012 08:08:31 -0500 From: Henning Schulzrinne To: Robert Kisteleki Thread-Topic: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - file formats and retrieval Thread-Index: AQHNuvKuzCYBuLMSwU6LbdrrU66zPpfbOSKA///8kAc= Date: Mon, 5 Nov 2012 13:08:29 +0000 Message-ID: References: , <50977571.9030007@ripe.net> In-Reply-To: <50977571.9030007@ripe.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.63] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Shane Amante , "lmap@ietf.org" Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - file formats and retrieval X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 13:08:37 -0000 My apologies - our Outlook web client doesn't do standard > quoting, so I u= sed HTML mark-up, which obviously did not survive the trip through the mail= ing list manager. I will try to re-send with more textual quoting.=0A= =0A= (Private) suggestions on how to make responses work within the constraints = of Outlook are appreciated.=0A= =0A= ________________________________________=0A= From: Robert Kisteleki [robert@ripe.net]=0A= Sent: Monday, November 05, 2012 3:14 AM=0A= To: Henning Schulzrinne=0A= Cc: Shane Amante; lmap@ietf.org=0A= Subject: Re: [lmap] Comments on draft-schulzrinne-lmap-requirements-00 - fi= le formats and retrieval=0A= =0A= As a meta-thread: either the mailing list is wonky, or Henning wrote about = a=0A= dozen or so emails lately with *many* different signatures :-)=0A= =0A= Also: without quoting it's very difficult to follow which parts are from th= e=0A= original mails. See below. Colors don't work well in text mails.=0A= =0A= Robert=0A= =0A= =0A= On 2012.11.05. 2:12, Henning Schulzrinne wrote:=0A= > Inline, in color, since Outlook doesn't do >.=0A= >=0A= > ________________________________________=0A= > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Shane Am= ante [shane@castlepoint.net]=0A= > Sent: Tuesday, September 25, 2012 12:06 PM=0A= > To: lmap@ietf.org=0A= > Subject: [lmap] Comments on draft-schulzrinne-lmap-requirements-00=0A= >=0A= > 1) It seems critical to look at creation of a data model and/or file form= at that allows this architecture to bundle together *both* the measurement = parameters/criteria used as input to a measurement test /and/ the measureme= nt results collected as part of that particular measurement run. This will = be critical if/when measurement results are analyzed historically, particul= arly by third-parties (e.g.: researchers) who may not have originally initi= ated the tests.=0A= >=0A= > If an existing format (e.g., IPFIX) is used for exchange, this might solv= e itself. My inclination would be to stick as close as possible to the on-t= he-wire format, with record delimiters, as this avoids having to evolve two= formats or worry about that new on-the-wire capabilities can't be represen= ted in the file, which would obviously be rather unpleasant.=0A= >=0A= >=0A= > 3) The draft doesn't appear to mention anything about providing an abilit= y for entities to query for and attain historical measurements that have al= ready been collected. (With respect to "query for", again, it would be nice= to have a data model that represented the measurement parameters used as i= nput to a test so those could help winnow down the data sets that match a g= iven criteria of interest that is to analyzed. See #1.) Presumably, this wo= uld be a valuable capability. Is this within the purview of the draft/archi= tecture?=0A= >=0A= > There seem to be two cases here: Retrieve the whole file (for which ftp o= r similar would presumably suffice) or a more elaborate query, as in "Give = me all measurements for client X during the 6-7 pm UTC hour, measurement 42= " or even "Give me all measurements ... that have a value greater than 55 m= s." The latter seems to get awfully close to an SQL-like query language. In= deed, not too surprisingly, this is what the current FCC infrastructure use= s. Or did you have something else in mind?=0A= >=0A= > Henning=0A= > _______________________________________________=0A= > lmap mailing list=0A= > lmap@ietf.org=0A= > https://www.ietf.org/mailman/listinfo/lmap=0A= >=0A= =0A= From nweaver@icsi.berkeley.edu Mon Nov 5 10:02:18 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 62E6921F89BC for ; Mon, 5 Nov 2012 10:02:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5ltfb+iYyiXO for ; Mon, 5 Nov 2012 10:02:17 -0800 (PST) Received: from rock.ICSI.Berkeley.EDU (rock.ICSI.Berkeley.EDU [192.150.186.19]) by ietfa.amsl.com (Postfix) with ESMTP id 164C521F8451 for ; Mon, 5 Nov 2012 09:58:47 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id 9B5F92C4010 for ; Mon, 5 Nov 2012 09:58:46 -0800 (PST) X-Virus-Scanned: amavisd-new at ICSI.Berkeley.EDU Received: from rock.ICSI.Berkeley.EDU ([127.0.0.1]) by localhost (maihub.ICSI.Berkeley.EDU [127.0.0.1]) (amavisd-new, port 10024) with LMTP id SzYhhF-Buxw4; Mon, 5 Nov 2012 09:58:46 -0800 (PST) Received: from wifi243.sys.icsi.berkeley.edu (wifi243.sys.ICSI.Berkeley.EDU [192.168.6.243]) (Authenticated sender: nweaver) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id 4CA0F2C4002; Mon, 5 Nov 2012 09:58:46 -0800 (PST) From: Nicholas Weaver Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 5 Nov 2012 09:58:47 -0800 Message-Id: <11D91A1D-C47E-49DC-82B7-FB37EDB9E732@icsi.berkeley.edu> To: lmap@ietf.org Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) Cc: Nicholas Weaver Subject: [lmap] My reflections on draft-schulzrinne-lmap-requirements-00 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 18:02:18 -0000 It appears to be a good document, but I feel that it is putting the cart = before the horse. The difficult problem is coming up with meaningful tests, especially for = a given level of system privilege, and ensuring that these tests are = meaningful and well calibrated. Even something "simple" like bandwidth testing becomes a huge can of = worms: look at the effort that Ookla has gone through in developing a = bandwidth test that is capable of operating across a wide variety of OSs = (including OS-limitations such as the lack of window scaling) in the = limited privilege context of a Flash applet. Then the question becomes how to turn a bandwidth test into a realistic = latency-under-load test (reliably measuring the bufferboat problem). Then the question becomes identifying the bottleneck which is the source = of the latency-under-load... And all ideally in a world where the client is running with the = permissions of Flash or HTML5 (although the server can be full = custom)... In comparison, the infrastructure for running measurements is trivia: = one-and-done browser measurements work just fine by simply reporting the = data back to the web server, and platforms for persistent measurements = are more often defined by the hardware cost and installation on the = client side, rather than the control software. =20 And the nature of such systems having "one-off infrastructure" (e.g. = SamKnows uses different infrastructure than BisMark than Ripe Atlas than = Netalyzr) isn't that big of a problem compared with making sure the = tests are sound and correct, within various client capability = restrictions. Overall, where I'd like to see this group go is focus on developing and = standardizing tests, of the form: Given a client with a given set of capabilities and a given persistence = model, can one construct a test to measure X in an efficient and = reliable manner? Because there are a lot of tests that need to be done, and it is the = tests themselves that is the problem area. From philip.eardley@bt.com Thu Nov 8 08:32:01 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 71AA121F852C for ; Thu, 8 Nov 2012 08:32:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.599 X-Spam-Level: X-Spam-Status: No, score=-103.599 tagged_above=-999 required=5 tests=[AWL=-0.001, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u47eVa+jNXCv for ; Thu, 8 Nov 2012 08:32:00 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 2FCB521F8433 for ; Thu, 8 Nov 2012 08:32:00 -0800 (PST) Received: from EVMHT64-UKRD.domain1.systemhost.net (10.36.3.101) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.279.1; Thu, 8 Nov 2012 16:31:59 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.2.94]) by EVMHT64-UKRD.domain1.systemhost.net ([10.36.3.101]) with mapi; Thu, 8 Nov 2012 16:31:59 +0000 From: To: Date: Thu, 8 Nov 2012 16:31:58 +0000 Thread-Topic: some thoughts about the LMAP activity Thread-Index: AQHNvc6SR21nqO1H7k6c74GvKmjf1w== Message-ID: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> Accept-Language: en-US, en-GB Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_9510D26531EF184D9017DF24659BB87F33F336A0ADEMV65UKRDdoma_" MIME-Version: 1.0 Subject: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 16:32:01 -0000 --_000_9510D26531EF184D9017DF24659BB87F33F336A0ADEMV65UKRDdoma_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Couple of thoughts after the LMAP bar bof last night. Thank-you to Henning and James for organising and initiating the meeting an= d the I-D about this important work. I think we should distinguish several topics: 1. Use cases -From an IETF perspective the aim (I think) is motivational an= d to show the broad scope. they don't have to be completely exhaustive. 2. Functional architecture - I think it's important for us to agree the hig= h level functions (starting from something like Henning's picture in the pl= enary, or the discussion Marcelo triggered about 'Starting the work descrip= tion') with enough detail to confirm consensus). Ideally we would agree a f= unctional architecture with other standards bodies working in the area. May= be this can be achieved informally, via the people who are active in severa= l standards bodies. 3. Which pieces of this functional architecture are a. standardised by the ietf b. standardised by some other body c. not appropriate for standardisation Ideally other standards bodies would have the same view of this breakdown, = to avoid the danger that there end up being 0 or 2 standards where there sh= ould be 1. I think the I-D and discussion are great start on 1 and 2. To get a BoF we = need a really clear answer on 3 as well. But we don't need a complete answe= r before starting any work at the IETF - eg there's already seems good agre= ement about doing some metrics related activity in ippm. Perhaps we can def= ine that already, and define other work a little later. Personally I'd find it very useful to have an overview of where other stand= ards bodies have got to and what work that they're planning. For example, I= remember that Michael B offered to give an overview of the Broadand forum'= s framework (=3D functional architecture?), is it possible to share it plea= se? 802.16 (?) was also mentioned last night, is any info available about t= hat please? One comment about the I-D. I find the Requirements section a bit premature.= For example, I don't think it's that useful for us to agree requirements a= bout pieces of the functional architecture that are in 3b & 3c, ie outside = the IETF's remit. Finally, I was wondering about how Henning and the authors of the I-D want = to handle input. Do you want comments that you'll gradually work in, extra = authors/editors, split the doc into different bits....? Best wishes, phil --_000_9510D26531EF184D9017DF24659BB87F33F336A0ADEMV65UKRDdoma_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Couple = of thoughts after the LMAP bar bof last night.
=  
Thank-y= ou to Henning and James for organising and initiating the meeting and the I= -D about this important work.
 
I think we should distinguish severa= l topics:
 
1. Use cases -From an IETF perspecti= ve the aim (I think) is motivational and to show the broad scope. they= don't have to be completely exhaustive. 
 
2. Functional architecture - I think= it's important for us to agree the high level functions (starting from som= ething like Henning's picture in the plenary, or the discussion Marcelo tri= ggered about 'Starting the work description') with enough detail to confirm consensus). Ideally we would agree a functio= nal architecture with other standards bodies working in the area. Maybe thi= s can be achieved informally, via the people who are active in several stan= dards bodies.
 
3. Which pieces of this functional a= rchitecture are
a. standardised by the ietf
b. standardised by some other body
c. not appropriate for standardisati= on
Ideally other standards bodies would= have the same view of this breakdown, to avoid the danger that there end u= p being 0 or 2 standards where there should be 1.
 
I think the I-D and discussion are g= reat start on 1 and 2. To get a BoF we need a really clear answer on 3= as well. But we don't need a complete answer before starting any work at t= he IETF - eg there's already seems good agreement about doing some metrics related activity in ippm. Perhaps we can define t= hat already, and define other work a little later.
 
Personally I'd find it very useful t= o have an overview of where other standards bodies have got to and what wor= k that they're planning. For example, I remember that Michael B offered to = give an overview of the Broadand forum's framework (=3D functional architecture?), is it possible to share it pleas= e? 802.16 (?) was also mentioned last night, is any info available about th= at please?
 
One comment about the I-D. I find th= e Requirements section a bit premature. For example, I don't think it's tha= t useful for us to agree requirements about pieces of the functional archit= ecture that are in 3b & 3c, ie outside the IETF's remit.
 
Finally, I was wondering about how H= enning and the authors of the I-D want to handle input. Do you want comment= s that you'll gradually work in, extra authors/editors, split the doc into = different bits....?
 
Best wishes,
phil
 
--_000_9510D26531EF184D9017DF24659BB87F33F336A0ADEMV65UKRDdoma_-- From jamesmilleresquire@gmail.com Thu Nov 8 09:12:53 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A14F621F87C7 for ; Thu, 8 Nov 2012 09:12:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GxoFx--CZW5B for ; Thu, 8 Nov 2012 09:12:52 -0800 (PST) Received: from mail-vc0-f172.google.com (mail-vc0-f172.google.com [209.85.220.172]) by ietfa.amsl.com (Postfix) with ESMTP id 266D621F87C0 for ; Thu, 8 Nov 2012 09:12:52 -0800 (PST) Received: by mail-vc0-f172.google.com with SMTP id fl11so3219337vcb.31 for ; Thu, 08 Nov 2012 09:12:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=k0bo3lU6aRs+wxsy0RXb/l9WH9SBTmefGXEPP8ExYU0=; b=fkcuXRnhmOUrjPXg4SuJq0FdGY3hh+LV2xBB3P0jVQ+nJACHTqa7CtFeboRt0oFVxl W0Y6xBkY7HBeoN1AeHfF5ZKOG7erapO8Zv32HbKbMjeW96sFANG9lQNLiNKy3IQD2sPj L+2wYvudhtEZw2xo4b7EhdYHFqEb4k1nK8VQdNz8r5xLHR9onfWlKSt+oXc4PaFqBbBe PN8kejObHXmwHGu++/foRMrLwwp8axK86lUjSQzeRLw+O7gjFmxdZgKA3a1/VIKS5R5d E3y8tZFun2g8OiwYpZO6eqI6rIYgIwAv+8GYo0mD9houWLNuhMGmKJBpQmJCaCb7fJRz VUXg== Received: by 10.58.33.66 with SMTP id p2mr2717326vei.24.1352394771580; Thu, 08 Nov 2012 09:12:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.58.200.226 with HTTP; Thu, 8 Nov 2012 09:12:20 -0800 (PST) In-Reply-To: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> From: James Miller Date: Thu, 8 Nov 2012 12:12:20 -0500 Message-ID: To: philip.eardley@bt.com Content-Type: multipart/alternative; boundary=047d7b418145f4c67e04cdfef1ee Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 17:12:53 -0000 --047d7b418145f4c67e04cdfef1ee Content-Type: text/plain; charset=ISO-8859-1 Thanks to everyone for the contributions on the list so far and the attendance and interesting feedback at the BAR BOF, and to Philip for the below. On Thu, Nov 8, 2012 at 11:31 AM, wrote: > Couple of thoughts after the LMAP bar bof last night. > > Thank-you to Henning and James for organising and initiating the meeting > and the I-D about this important work. > > I think we should distinguish several topics: > > 1. Use cases -From an IETF perspective the aim (I think) is motivational > and to show the broad scope. they don't have to be completely exhaustive. > In answer to your comments point below, I would appreciate specific references to the documents section 2 discussing use cases. We did not intend to provide a highly specific description of examples but wanted to capture at a high level what sorts of use cases may exist in a way that would be relevant for understanding how the architcture and metrics would need to be developed. If there are categories of use cases that are not captured we would like to make those changes. I think it's important that we provide the use cases at a high-level but more specific discussions may clutter the focus of the draft if the references aren't necessary to provide background for the goal of defining the architecture in the document. Currently we have the following three cases: - Provider network measurements - User network diagnostics - Multi-provider network measurements Some specific references on the list and in discussion last night related to: - Can a user perform a test and get data? Under this formulation, yes it is the "user network diagnostics" case. - Could different parties, contracting or volunteering together, operate a test together, such as a small ISP contracting out work with a third-party? Yes, we think that the authentication/security and data communications flow that we map out try to capture this. In fact the FCC's case is similar is a variant of this case where we work with many parties including subscribers who volunteer to join the program, SamKnows who is the contractor running the test, and measurement lab who also works with us to support our measurement servers solution. If people think that more specific examples that would demonstrate the use cases would be helpful in the categorical sections, we would be happy to include suggestions. > > 2. Functional architecture - I think it's important for us to agree the > high level functions (starting from something like Henning's picture in the > plenary, or the discussion Marcelo triggered about 'Starting the work > description') with enough detail to confirm consensus). Ideally we would > agree a functional architecture with other standards bodies working in the > area. Maybe this can be achieved informally, via the people who are active > in several standards bodies. > > The ID currently maps this in section 3 defining the overview of the architecture. The ID text tracks the diagrams and approaches in both Henning's plenary talk and the approaches that have been employed with the FCC measurement programs. Would there be changes to the ID that could help facilitate the discussion? 3 . Architecture Overview . . . . . . . . . . . . . . . . . . . . 8 3.1 . Measurement client . . . . . . . . . . . . . . . . . . . . 8 3.2 . Measurement server . . . . . . . . . . . . . . . . . . . . 8 3.3 . Measurement controller . . . . . . . . . . . . . . . . . . 8 3.4 . Data collector . . . . . . . . . . . . . . . . . . . . . . 8 3.5 . Network parameter server . . . . . . . . . . . . . . . . . 9 On this and the points below, I would like to suggest that anyone interesting this topic call in or attend our next Generation meeting at FCC Headquarters currently Scheduled for November 28 in the morning. We could have an overview from folks doing work in the IEEE and Broadband Forum and offer a platform for discussion on the architecture and how groups might currently be heading. We typically make these meeting available via WebEx but if folks are in DC, I'm happy to brew some coffee. 3. Which pieces of this functional architecture are > a. standardised by the ietf > b. standardised by some other body > c. not appropriate for standardisation > Ideally other standards bodies would have the same view of this breakdown, > to avoid the danger that there end up being 0 or 2 standards where there > should be 1. > > I think the I-D and discussion are great start on 1 and 2. To get a BoF we > need a really clear answer on 3 as well. But we don't need a complete > answer before starting any work at the IETF - eg there's already seems good > agreement about doing some metrics related activity in ippm. Perhaps we can > define that already, and define other work a little later. > > Personally I'd find it very useful to have an overview of where other > standards bodies have got to and what work that they're planning. For > example, I remember that Michael B offered to give an overview of the > Broadand forum's framework (= functional architecture?), is it possible to > share it please? 802.16 (?) was also mentioned last night, is any info > available about that please? > > One comment about the I-D. I find the Requirements section a bit > premature. For example, I don't think it's that useful for us to agree > requirements about pieces of the functional architecture that are in 3b & > 3c, ie outside the IETF's remit. > > Finally, I was wondering about how Henning and the authors of the I-D want > to handle input. Do you want comments that you'll gradually work in, extra > authors/editors, split the doc into different bits....? > > Best wishes, > phil > > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > > --047d7b418145f4c67e04cdfef1ee Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks to everyone for the contributions on the list so far and the attenda= nce and interesting feedback at the BAR BOF, and to Philip for the below.
On Thu, Nov 8, 2012 at 11:31 AM, <p= hilip.eardley@bt.com> wrote:
Couple of thoughts= after the LMAP bar bof last night.
=A0
Thank-you to Henni= ng and James for organising and initiating the meeting and the I-D about th= is important work.
=A0
I think we should distinguish severa= l topics:
=A0
1. Use cases -From an IETF perspecti= ve=A0the aim (I think) is motivational and to show the broad scope. they do= n't have to be completely exhaustive.=A0

In answer to your comments point below,=A0I would appre= ciate specific references to the documents section 2 discussing use cases. = =A0We did not intend to provide a highly specific description of examples b= ut wanted to capture at a high level what sorts of use cases may exist in a= way that would be relevant for understanding how the architcture and metri= cs would need to be developed.

If there are categories of use cases that are not captu= red we would like to make those changes. =A0I think it's important that= we provide the use cases at a high-level but more specific discussions may= clutter the focus of the draft if the references aren't necessary to p= rovide background for the goal of defining the architecture in the document= . =A0Currently we have the following three cases:

  • Provider net=
work measurements
  • User network=
 diagnostics
    • 


  • Multi-provider network measurements

Some specific references on the = list and in discussion last night related to:

  • Can a user perform a test and get data? =A0
Under t= his formulation, yes it is the "user network diagnostics" case.

  • Could different parties, contracting or vol= unteering together, operate a test together, such as a small ISP contractin= g out work with a third-party?=A0
Yes, we think that the authentication/security and data com= munications flow that we map out try to capture this. =A0In fact the FCC= 9;s case is=A0similar=A0is a=A0variant=A0of this case where we work with ma= ny parties including subscribers who volunteer to join the program, SamKnow= s who is the contractor running the test, and measurement lab who also work= s with us to support our measurement servers solution.

If people think that more specific examples that would = demonstrate the use cases would be helpful in the categorical sections, we = would be happy to include suggestions.
=A0
=A0
2. Functional architecture - I think= it's important for us to agree the high level functions (starting from= something like Henning's picture in the plenary, or the discussion Mar= celo triggered about 'Starting the work description') with enough detail to confirm consensus). Ideally we would agree a functio= nal architecture with other standards bodies working in the area. Maybe thi= s can be achieved informally, via the people who are active in several stan= dards bodies.
=A0

The ID currently maps this in section 3 defining t= he overview of the architecture. =A0The ID text tracks the diagrams and app= roaches in both Henning's plenary talk and the approaches that have bee= n employed with the FCC measurement programs. =A0Would there be changes to = the ID that could help=A0facilitate=A0the discussion?

  3.  Architecture Overview  . . =
. . . . . . . . . . . . . . . . . .  8
     3.1.  Measurement client . . . . . . . . . . . . . =
. . . . . . .  8
     3.2.  Measurement server . . . . . . . . . . . . . =
. . . . . . .  8
     3.3.  Measurement controller . . . . . . . . . . . =
. . . . . . .  8
     3.4.  Data collector . . . . . . . . . . . . . . . =
. . . . . . .  8
     3.5.  Network parameter server . . . . . . . . . . =
. . . . . . .  9

On this and the points below, I would like to sug= gest that anyone interesting this topic call in or attend our next Generati= on meeting at FCC Headquarters currently Scheduled for November 28 in the m= orning. =A0We could have an overview from folks doing work in the IEEE and = Broadband Forum and offer a platform for discussion on the architecture and= how groups might currently be heading.

We typically make these meeting available via WebEx but= if folks are in DC, I'm happy to brew some coffee.



3. Which pieces of this functional a= rchitecture are
a. standardised by the ietf
b. standardised by some other body
c. not appropriate for standardisati= on
Ideally other standards bodies would= have the same view of this breakdown, to avoid the danger that there end u= p being 0 or 2 standards where there should be 1.
=A0
I think the I-D and discussion are g= reat=A0start on 1 and 2. To get a BoF we need a really clear answer on 3 as= well. But we don't need a complete answer before starting any work at = the IETF - eg there's already seems=A0good agreement about doing some metrics related activity in ippm. Perhaps we can define t= hat already, and define other work a little later.
=A0
Personally I'd find it very usef= ul to have an overview of where other standards bodies have got to and what= work that they're planning. For example, I remember that Michael B off= ered to give an overview of the Broadand forum's framework (=3D functional architecture?), is it possible to share it pleas= e? 802.16 (?) was also mentioned last night, is any info available about th= at please?
=A0
One comment about the I-D. I find th= e Requirements section a bit premature. For example, I don't think it&#= 39;s that useful for us to agree requirements about pieces of the functiona= l architecture that are in 3b & 3c, ie outside the IETF's remit.
=A0
Finally, I was wondering about how H= enning and the authors of the I-D want to handle input. Do you want comment= s that you'll gradually work in, extra authors/editors, split the doc i= nto different bits....?
=A0
Best wishes,
phil
=A0

_______________________________________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap


--047d7b418145f4c67e04cdfef1ee-- From bs7652@att.com Thu Nov 8 10:39:42 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C2CC321F8433 for ; Thu, 8 Nov 2012 10:39:42 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.299 X-Spam-Level: X-Spam-Status: No, score=-106.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_44=0.6, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xyzKvqkD7P6m for ; Thu, 8 Nov 2012 10:39:42 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id 0165121F842B for ; Thu, 8 Nov 2012 10:39:41 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id d6cfb905.0.1654779.00-459.4562472.nbfkord-smmo05.seg.att.com (envelope-from ); Thu, 08 Nov 2012 18:39:41 +0000 (UTC) X-MXL-Hash: 509bfc6d5b14846c-081586fabbef4418c4213d91e0c71edbcb030611 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8IddX0016144 for ; Thu, 8 Nov 2012 13:39:41 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8IdPfc014912 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 8 Nov 2012 13:39:34 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint03.pst.cso.att.com (RSA Interceptor) for ; Thu, 8 Nov 2012 13:39:09 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Thu, 8 Nov 2012 13:39:09 -0500 From: "STARK, BARBARA H" To: "lmap@ietf.org" Thread-Topic: Recommendation for Path Forward Thread-Index: Ac294FSYyYlC5TBWQzmqeQ1zCkoe1Q== Date: Thu, 8 Nov 2012 18:39:08 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5C8DB8@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.158.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=a66HAzuF c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=8BrlyXNxoVEA:10 a=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=kj9] X-AnalysisOut: [zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32RAAAA:8 a=IbbpEMglt] X-AnalysisOut: [IsA:10 a=ytYbQtBQOIuKIKBfOmUA:9 a=CjuIK1q_8ugA:10] Subject: [lmap] Recommendation for Path Forward X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 18:39:42 -0000 I agree with Phil's sentiments, and here are my thoughts on the topic of pa= th forward. After sleeping on last night's discussion, I recommend the following path f= orward: 1. Scope the problem space. 2. Identify use cases of interest. Use cases need to clearly identify the p= rimary "user" of the use case. 3. Determine which use cases we "must" solve vs. "may/should" solve. 4. Determine "Business Requirements" for the use cases. [It's ok to accept = input of biz requirements for all use cases, so long as we keep in mind tha= t the primary focus is to meet the needs of mandatory use cases.] 5. Business Requirements drive high level functional requirements and crite= ria for assessing various possible architectural solutions. 6. Allow for proposals to be submitted for functional architectures that wo= uld meet the needs of the Business Requirements. Compare the proposed funct= ional architectures as to their ability to satisfy the functional requireme= nts and other assessment criteria. 7. Select a functional architecture. 8. Determine gaps in the architecture (functional requirements for which so= lutions are not obvious or do not exist) that require additional effort, an= d determine which of these the IETF wants to tackle and is well-suited to t= ackle. Other SDOs can make similar assessments. 9. Create the needed gap-filling solutions. In my experience (as a primary use case "user" who has needed solutions cre= ated in the past), it is in this "primary user's" best interest to consider= themselves to be the owner of the process as a whole. It may be possible t= o hand off ownership to a SIG (Special Interest Group), if there is a SIG w= hose interests are primarily to meet the needs of a group of such users. IE= TF is not a SIG, and I do not recommend trying to hand off ownership of the= holistic solution to the IETF. An example of a SIG as an owner, consider t= he way the US and Canadian governments use NENA to create architectures for= emergency services, and use IETF to create protocols where there are gaps. I think we should be able to get items 1 through 3 done really quickly, and= then proceed directly to 4. I think the current draft is assuming an architecture (step 7), and trying = to back into 5, without first having tackled 1-4. As such, I'm not willing = to provide comments on the current draft. I think it's the wrong architectu= re.=20 I think that it may be possible to do steps 1-8 within the context of lmap+= IETF, if others are willing. I think 1-4 could be done using this email ref= lector. Once we do 1-4, I think we will have what we need to proceed with e= stablishing a more formal environment in which to work 5-7. My next email will propose what I think are answers for items 1-3. Barbara From bs7652@att.com Thu Nov 8 10:41:59 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6578C21F8433 for ; Thu, 8 Nov 2012 10:41:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.524 X-Spam-Level: X-Spam-Status: No, score=-106.524 tagged_above=-999 required=5 tests=[AWL=0.075, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P2m1SELKGLJd for ; Thu, 8 Nov 2012 10:41:58 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id A644A21F842B for ; Thu, 8 Nov 2012 10:41:58 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 6fcfb905.0.1643894.00-153.4546948.nbfkord-smmo07.seg.att.com (envelope-from ); Thu, 08 Nov 2012 18:41:58 +0000 (UTC) X-MXL-Hash: 509bfcf60ff1ce14-ec487fbfac42a412490c92210e1c818808a77c52 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8IfvkF022252 for ; Thu, 8 Nov 2012 13:41:58 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8IfoF3022139 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 8 Nov 2012 13:41:52 -0500 Received: from GAALPA1MSGHUB9B.ITServices.sbc.com (gaalpa1msghub9b.itservices.sbc.com [130.8.36.88]) by sflint04.pst.cso.att.com (RSA Interceptor) for ; Thu, 8 Nov 2012 13:41:27 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9B.ITServices.sbc.com ([130.8.36.88]) with mapi id 14.02.0318.001; Thu, 8 Nov 2012 13:41:11 -0500 From: "STARK, BARBARA H" To: "lmap@ietf.org" Thread-Topic: Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQ== Date: Thu, 8 Nov 2012 18:41:10 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.158.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=KI7t+i5o c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=8BrlyXNxoVEA:10 a=mT2U6pLIDLoA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=YUwTYMFjZNgA:10 a=2jEgpgA8oQdiscA9zk8A:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 18:41:59 -0000 Following are my suggestions for problem scope, listing of use cases that h= ave been mentioned to date, and my thoughts on which we need to focus on so= lving. Proposed Scope: Performance measurements for mass market broadband connecti= vity services. Use Cases=20 1. Regulators want performance data for "mass market" broadband access netw= ork connectivity services. Regulators are the primary user in this use case= . Regulators may have business requirements that involve access by others t= o the data collected for this use case (e.g., researchers are to be allowed= access to data, or broadband customers should be allowed to access data fo= r their own service), but these others are not considered primary users in = this case. 2. Researchers want to be able to run and collect data from a variety of te= sts performed against broadband connections (end-to-end, segment by segment= , maybe or maybe not limited to mass market or the access network). Researc= hers are the primary user. 3. Broadband "mass market" customers want to run tests against their broadb= and connections and see results of tests performed against their broadband = connection. Regulators, researchers, and customers are the 3 sets of "users= " that are covered by these 3 use cases. Broadband customers are the primar= y user. 4. ISPs want tests to assist in troubleshooting, and network architecting a= nd engineering. [Note that I'm just listing this as a use case that has bee= n mentioned, and not as something that is being requested by all ISPs -- no= value judgment is implied by listing it here.] ISPs are the primary user. Which Use Cases Are Mandatory Based on last night's discussion, I believe that only the first use case is= mandatory. The others are fine to consider, if they don't get in the way o= f solving use case #1. If we can agree on these points, then I think the next step is to identify = the Business Requirements of regulators. Ideally, we would get input from r= egulators around the world. We certainly need to try to solicit their input= . However, I think it's a good idea to start by documenting the business re= quirements of the one regulator who we do have. Getting started may incite = others to join in. Other "users" (or proxies of other users) are also welco= me to provide their business requirements -- but they need to understand th= at satisfying their requirements is considered "nice to have" and not manda= tory. I'm going to work next on putting together a start for Business Requirement= s. Barbara From steve.baillargeon@ericsson.com Thu Nov 8 11:58:59 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E071521F84F8 for ; Thu, 8 Nov 2012 11:58:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GQ6RgcO5BWMn for ; Thu, 8 Nov 2012 11:58:59 -0800 (PST) Received: from imr4.ericy.com (imr4.ericy.com [198.24.6.9]) by ietfa.amsl.com (Postfix) with ESMTP id 2BE7021F84BF for ; Thu, 8 Nov 2012 11:58:59 -0800 (PST) Received: from eusaamw0711.eamcs.ericsson.se ([147.117.20.178]) by imr4.ericy.com (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id qA8K4BDv024623; Thu, 8 Nov 2012 14:04:17 -0600 Received: from EUSAAHC004.ericsson.se (147.117.188.84) by eusaamw0711.eamcs.ericsson.se (147.117.20.178) with Microsoft SMTP Server (TLS) id 8.3.279.1; Thu, 8 Nov 2012 14:58:50 -0500 Received: from EUSAAMB105.ericsson.se ([147.117.188.122]) by EUSAAHC004.ericsson.se ([147.117.188.84]) with mapi id 14.02.0318.001; Thu, 8 Nov 2012 14:58:50 -0500 From: Steve Baillargeon To: "STARK, BARBARA H" , "lmap@ietf.org" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQACMnlQ Date: Thu, 8 Nov 2012 19:58:49 +0000 Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [147.117.188.135] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 19:59:00 -0000 Hi=20 I believe use case #4 is critical, at least for mobile broadband operators. I would like to see an architecture that can accommodate both cases as much= as possible including operator-initiated active measurements. In theory, the metrics measured and reported by regulators/customers should= be "consistent" with the metrics measured and collected by operators. -Steve =20 -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of STA= RK, BARBARA H Sent: November-08-12 1:41 PM To: lmap@ietf.org Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nic= e to Have Following are my suggestions for problem scope, listing of use cases that h= ave been mentioned to date, and my thoughts on which we need to focus on so= lving. Proposed Scope: Performance measurements for mass market broadband connecti= vity services. Use Cases 1. Regulators want performance data for "mass market" broadband access netw= ork connectivity services. Regulators are the primary user in this use case= . Regulators may have business requirements that involve access by others t= o the data collected for this use case (e.g., researchers are to be allowed= access to data, or broadband customers should be allowed to access data fo= r their own service), but these others are not considered primary users in = this case. 2. Researchers want to be able to run and collect data from a variety of te= sts performed against broadband connections (end-to-end, segment by segment= , maybe or maybe not limited to mass market or the access network). Researc= hers are the primary user. 3. Broadband "mass market" customers want to run tests against their broadb= and connections and see results of tests performed against their broadband = connection. Regulators, researchers, and customers are the 3 sets of "users= " that are covered by these 3 use cases. Broadband customers are the primar= y user. 4. ISPs want tests to assist in troubleshooting, and network architecting a= nd engineering. [Note that I'm just listing this as a use case that has bee= n mentioned, and not as something that is being requested by all ISPs -- no= value judgment is implied by listing it here.] ISPs are the primary user. Which Use Cases Are Mandatory Based on last night's discussion, I believe that only the first use case is= mandatory. The others are fine to consider, if they don't get in the way o= f solving use case #1. If we can agree on these points, then I think the next step is to identify = the Business Requirements of regulators. Ideally, we would get input from r= egulators around the world. We certainly need to try to solicit their input= . However, I think it's a good idea to start by documenting the business re= quirements of the one regulator who we do have. Getting started may incite = others to join in. Other "users" (or proxies of other users) are also welco= me to provide their business requirements -- but they need to understand th= at satisfying their requirements is considered "nice to have" and not manda= tory. I'm going to work next on putting together a start for Business Requirement= s. Barbara _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From mlinsner@cisco.com Thu Nov 8 12:38:59 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D13F921F8683 for ; Thu, 8 Nov 2012 12:38:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.25 X-Spam-Level: X-Spam-Status: No, score=-10.25 tagged_above=-999 required=5 tests=[AWL=0.349, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aZmhfrBEuKtA for ; Thu, 8 Nov 2012 12:38:59 -0800 (PST) Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) by ietfa.amsl.com (Postfix) with ESMTP id E629921F867C for ; Thu, 8 Nov 2012 12:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3235; q=dns/txt; s=iport; t=1352407139; x=1353616739; h=date:subject:from:to:message-id:in-reply-to:mime-version: content-transfer-encoding; bh=F72nZdvK0syiGhBusPzrFk03Mgm245cCBA98p6X13Cw=; b=GWFigMaK02Uy6b5tsNEoniAfr0GG52coXMSzTy50vSTFOZAjuYyrWT3h PVb6BinqIqhW6Qmg9Mh/y4LfIHze1bxHl8FMFORuP3uD5Ia5mFixM4Q1d iUqkhB7sV9NV9MdeMcutXL9sfP8XnYDMvRQbUdDA0XTEQZM46q+w5777Q k=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av8EALEXnFCtJV2c/2dsb2JhbAA6CsNCgQiCHgEBAQQBAQEPAScCATEXBwgRAwECVigIBgESIoVugXoLnA6gLASMEhAEhjMDiFqNIYVriG2Ba4MNgT0JFw X-IronPort-AV: E=Sophos;i="4.80,739,1344211200"; d="scan'208";a="140271717" Received: from rcdn-core-5.cisco.com ([173.37.93.156]) by rcdn-iport-2.cisco.com with ESMTP; 08 Nov 2012 20:38:58 +0000 Received: from [10.82.217.137] (rtp-vpn3-391.cisco.com [10.82.217.137]) by rcdn-core-5.cisco.com (8.14.5/8.14.5) with ESMTP id qA8KcuAY023131; Thu, 8 Nov 2012 20:38:57 GMT User-Agent: Microsoft-MacOutlook/14.2.3.120616 Date: Thu, 08 Nov 2012 15:38:57 -0500 From: Marc Linsner To: "STARK, BARBARA H" , "lmap@ietf.org" Message-ID: Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 20:38:59 -0000 Barbara, Thanks for documenting the use cases below. I would like to submit use case #5. 5. OTT providers of services. OTT providers are interested in the QOE of their customers and may instanciate a measurement client/controller/server of their own accord. -Marc- -----Original Message----- From: "STARK, BARBARA H" Date: Thursday, November 8, 2012 1:41 PM To: "lmap@ietf.org" Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have >Following are my suggestions for problem scope, listing of use cases that >have been mentioned to date, and my thoughts on which we need to focus on >solving. > >Proposed Scope: Performance measurements for mass market broadband >connectivity services. > >Use Cases >1. Regulators want performance data for "mass market" broadband access >network connectivity services. Regulators are the primary user in this >use case. Regulators may have business requirements that involve access >by others to the data collected for this use case (e.g., researchers are >to be allowed access to data, or broadband customers should be allowed to >access data for their own service), but these others are not considered >primary users in this case. > >2. Researchers want to be able to run and collect data from a variety of >tests performed against broadband connections (end-to-end, segment by >segment, maybe or maybe not limited to mass market or the access >network). Researchers are the primary user. > >3. Broadband "mass market" customers want to run tests against their >broadband connections and see results of tests performed against their >broadband connection. Regulators, researchers, and customers are the 3 >sets of "users" that are covered by these 3 use cases. Broadband >customers are the primary user. > >4. ISPs want tests to assist in troubleshooting, and network architecting >and engineering. [Note that I'm just listing this as a use case that has >been mentioned, and not as something that is being requested by all ISPs >-- no value judgment is implied by listing it here.] ISPs are the primary >user. > >Which Use Cases Are Mandatory >Based on last night's discussion, I believe that only the first use case >is mandatory. The others are fine to consider, if they don't get in the >way of solving use case #1. > >If we can agree on these points, then I think the next step is to >identify the Business Requirements of regulators. Ideally, we would get >input from regulators around the world. We certainly need to try to >solicit their input. However, I think it's a good idea to start by >documenting the business requirements of the one regulator who we do >have. Getting started may incite others to join in. Other "users" (or >proxies of other users) are also welcome to provide their business >requirements -- but they need to understand that satisfying their >requirements is considered "nice to have" and not mandatory. > >I'm going to work next on putting together a start for Business >Requirements. >Barbara >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Thu Nov 8 13:01:19 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4D84021F88B4 for ; Thu, 8 Nov 2012 13:01:19 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.599 X-Spam-Level: X-Spam-Status: No, score=-103.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6g2RGV9i3iW3 for ; Thu, 8 Nov 2012 13:01:18 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 17DCC21F889E for ; Thu, 8 Nov 2012 13:01:17 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.279.1; Thu, 8 Nov 2012 21:01:16 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.2.94]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Thu, 8 Nov 2012 21:01:16 +0000 From: To: , , Date: Thu, 8 Nov 2012 21:01:15 +0000 Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac298RfvuR57JZu1TRy4fUZvdHWEQAAAZHgo Message-ID: <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, In-Reply-To: Accept-Language: en-US, en-GB Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 21:01:19 -0000 similar to #1=20 Operators want performance data for "mass market" broadband access network = connectivity services. One difference between the use cases is the timescale of interest. For exam= ple regulators are probably more interested in monthly averages over large = numbers of users whereas in #4 ISPs are probably more interested in finer g= rain information. Barbara, I'm interested in why you want to narrow the focus to #1 (is it that your s= ceptical about the importance of the other use cases, or that they don't ne= ed standardisation, or something else?) thanks phil ________________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of Marc Linsn= er [mlinsner@cisco.com] Sent: 08 November 2012 20:38 To: STARK, BARBARA H; lmap@ietf.org Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs.= Nice to Have Barbara, Thanks for documenting the use cases below. I would like to submit use case #5. 5. OTT providers of services. OTT providers are interested in the QOE of their customers and may instanciate a measurement client/controller/server of their own accord. -Marc- -----Original Message----- From: "STARK, BARBARA H" Date: Thursday, November 8, 2012 1:41 PM To: "lmap@ietf.org" Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have >Following are my suggestions for problem scope, listing of use cases that >have been mentioned to date, and my thoughts on which we need to focus on >solving. > >Proposed Scope: Performance measurements for mass market broadband >connectivity services. > >Use Cases >1. Regulators want performance data for "mass market" broadband access >network connectivity services. Regulators are the primary user in this >use case. Regulators may have business requirements that involve access >by others to the data collected for this use case (e.g., researchers are >to be allowed access to data, or broadband customers should be allowed to >access data for their own service), but these others are not considered >primary users in this case. > >2. Researchers want to be able to run and collect data from a variety of >tests performed against broadband connections (end-to-end, segment by >segment, maybe or maybe not limited to mass market or the access >network). Researchers are the primary user. > >3. Broadband "mass market" customers want to run tests against their >broadband connections and see results of tests performed against their >broadband connection. Regulators, researchers, and customers are the 3 >sets of "users" that are covered by these 3 use cases. Broadband >customers are the primary user. > >4. ISPs want tests to assist in troubleshooting, and network architecting >and engineering. [Note that I'm just listing this as a use case that has >been mentioned, and not as something that is being requested by all ISPs >-- no value judgment is implied by listing it here.] ISPs are the primary >user. > >Which Use Cases Are Mandatory >Based on last night's discussion, I believe that only the first use case >is mandatory. The others are fine to consider, if they don't get in the >way of solving use case #1. > >If we can agree on these points, then I think the next step is to >identify the Business Requirements of regulators. Ideally, we would get >input from regulators around the world. We certainly need to try to >solicit their input. However, I think it's a good idea to start by >documenting the business requirements of the one regulator who we do >have. Getting started may incite others to join in. Other "users" (or >proxies of other users) are also welcome to provide their business >requirements -- but they need to understand that satisfying their >requirements is considered "nice to have" and not mandatory. > >I'm going to work next on putting together a start for Business >Requirements. >Barbara >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap= From bs7652@att.com Thu Nov 8 13:16:21 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8A18721F8B98 for ; Thu, 8 Nov 2012 13:16:21 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.539 X-Spam-Level: X-Spam-Status: No, score=-106.539 tagged_above=-999 required=5 tests=[AWL=0.060, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2oW+hTHAivQw for ; Thu, 8 Nov 2012 13:16:20 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 2A98921F8487 for ; Thu, 8 Nov 2012 13:16:20 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 4212c905.2aaaf3c45940.1736189.00-507.4784395.nbfkord-smmo06.seg.att.com (envelope-from ); Thu, 08 Nov 2012 21:16:20 +0000 (UTC) X-MXL-Hash: 509c2124410b9850-00b20b96bac4bd4330fc87e70b038a86eced022d Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id f112c905.0.1736168.00-254.4784325.nbfkord-smmo06.seg.att.com (envelope-from ); Thu, 08 Nov 2012 21:16:17 +0000 (UTC) X-MXL-Hash: 509c21210311f21a-ffcd67e9fce79667224b87f08a1a54d5c9e7e550 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8LGExL005434; Thu, 8 Nov 2012 16:16:15 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8LG67E005368 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 8 Nov 2012 16:16:12 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint04.pst.cso.att.com (RSA Interceptor); Thu, 8 Nov 2012 16:15:43 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Thu, 8 Nov 2012 16:15:42 -0500 From: "STARK, BARBARA H" To: Steve Baillargeon , "lmap@ietf.org" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQACMnlQAADK6mA= Date: Thu, 8 Nov 2012 21:15:41 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5C9336@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.158.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=DIM4FVxb c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=8BrlyXNxoVEA:10 a=WUZhNrZLDssA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bbd-PDihl1AA:10 a=48vgC7mUAAAA:8 a=55qSIGMjQ7uqWR] X-AnalysisOut: [D9r1oA:9 a=CjuIK1q_8ugA:10 a=lZB815dzVvQA:10 a=_JwY0JK3tcf] X-AnalysisOut: [yhX91:21 a=zzGR5DSsZawcSGbk:21] Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 21:16:21 -0000 > Hi > I believe use case #4 is critical, at least for mobile broadband operator= s. > I would like to see an architecture that can accommodate both cases as mu= ch > as possible including operator-initiated active measurements. > In theory, the metrics measured and reported by regulators/customers > should be "consistent" with the metrics measured and collected by > operators. >=20 > -Steve Hmm. While I agree that operators consider collecting such metrics to= be important, I disagree that this should be a mandatory-to-consider drive= r for this "lmap" architecture. On our wireline side, we already have an architecture for managing devices,= running tests, collecting metrics, and measurement data collection. We don= 't need lmap to define something different for us. I'm not an expert on our wireless architecture, but I haven't heard any req= uests from that side, for lmap to define an architecture for them. I do kno= w that we do extensive mobile device management using OMA-DM. And I know th= at OMA-DM data models include elements for tests and some performance metri= c data collection. I know that if we need OM-DM extended to cover other dat= a elements, that we have people capable of making this happen. I'm aware of separate (from lmap) standards work that some operators are pu= rsuing to improve their performance measurement capabilities. I completely = support such efforts. And I believe that they are distinct from lmap and th= at their requirements to not, in any way, impose requirements on lmap. I'm aware that there are operators who aren't already collecting metrics an= d do no device management. For these -- if they find the regulator-required= metrics to be useful, that's great. If they want more than what the regula= tors want, then I consider trying to make sure that lmap meets their needs = to be optional.=20 But though I think it's optional, I fully support bringing the requirements= in for us to consider. I think it's great to have people provide business = requirements for all of the use cases, and not just the regulator use case.= If you're able to provide business requirements for the operator use case = that fit in well with the regulator requirements, then I may well support t= heir inclusion in lmap. Optional doesn't mean that they can't be included. = But if they don't fit in well, I, for one, will consider them to be unneces= sary for lmap. Barbara > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of > STARK, BARBARA H > Sent: November-08-12 1:41 PM > To: lmap@ietf.org > Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. > Nice to Have >=20 > Following are my suggestions for problem scope, listing of use cases that > have been mentioned to date, and my thoughts on which we need to focus > on solving. >=20 > Proposed Scope: Performance measurements for mass market broadband > connectivity services. >=20 > Use Cases > 1. Regulators want performance data for "mass market" broadband access > network connectivity services. Regulators are the primary user in this us= e > case. Regulators may have business requirements that involve access by > others to the data collected for this use case (e.g., researchers are to = be > allowed access to data, or broadband customers should be allowed to acces= s > data for their own service), but these others are not considered primary > users in this case. >=20 > 2. Researchers want to be able to run and collect data from a variety of = tests > performed against broadband connections (end-to-end, segment by > segment, maybe or maybe not limited to mass market or the access > network). Researchers are the primary user. >=20 > 3. Broadband "mass market" customers want to run tests against their > broadband connections and see results of tests performed against their > broadband connection. Regulators, researchers, and customers are the 3 > sets of "users" that are covered by these 3 use cases. Broadband customer= s > are the primary user. >=20 > 4. ISPs want tests to assist in troubleshooting, and network architecting= and > engineering. [Note that I'm just listing this as a use case that has been > mentioned, and not as something that is being requested by all ISPs -- no > value judgment is implied by listing it here.] ISPs are the primary user. >=20 > Which Use Cases Are Mandatory > Based on last night's discussion, I believe that only the first use case = is > mandatory. The others are fine to consider, if they don't get in the way = of > solving use case #1. >=20 > If we can agree on these points, then I think the next step is to identif= y the > Business Requirements of regulators. Ideally, we would get input from > regulators around the world. We certainly need to try to solicit their in= put. > However, I think it's a good idea to start by documenting the business > requirements of the one regulator who we do have. Getting started may > incite others to join in. Other "users" (or proxies of other users) are a= lso > welcome to provide their business requirements -- but they need to > understand that satisfying their requirements is considered "nice to have= " > and not mandatory. >=20 > I'm going to work next on putting together a start for Business > Requirements. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From trammell@tik.ee.ethz.ch Thu Nov 8 13:42:04 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B816121F86AD for ; Thu, 8 Nov 2012 13:42:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Us2Vd75v68Wl for ; Thu, 8 Nov 2012 13:42:03 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 99AE021F86D4 for ; Thu, 8 Nov 2012 13:42:03 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id 62EB3D930D; Thu, 8 Nov 2012 22:42:02 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id UJ5hHcYi3mnA; Thu, 8 Nov 2012 22:42:02 +0100 (MET) Received: from dhcp-447b.meeting.ietf.org (dhcp-447b.meeting.ietf.org [130.129.68.123]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id 8AE86D9308; Thu, 8 Nov 2012 22:42:01 +0100 (MET) Content-Type: multipart/alternative; boundary="Apple-Mail=_9A3E47AD-E9CF-4575-A4E5-08B63801FFAC" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Brian Trammell In-Reply-To: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> Date: Thu, 8 Nov 2012 16:41:59 -0500 Message-Id: <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> To: philip.eardley@bt.com X-Mailer: Apple Mail (2.1499) Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 21:42:04 -0000 --Apple-Mail=_9A3E47AD-E9CF-4575-A4E5-08B63801FFAC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hi, Philip, all, On 8 Nov 2012, at 11:31, philip.eardley@bt.com wrote: > Couple of thoughts after the LMAP bar bof last night. > =20 > Thank-you to Henning and James for organising and initiating the = meeting and the I-D about this important work. > =20 > I think we should distinguish several topics: > =20 > 1. Use cases -=46rom an IETF perspective the aim (I think) is = motivational and to show the broad scope. they don't have to be = completely exhaustive.=20 > =20 > 2. Functional architecture - I think it's important for us to agree = the high level functions (starting from something like Henning's picture = in the plenary, or the discussion Marcelo triggered about 'Starting the = work description') with enough detail to confirm consensus). Ideally we = would agree a functional architecture with other standards bodies = working in the area. Maybe this can be achieved informally, via the = people who are active in several standards bodies. > =20 > 3. Which pieces of this functional architecture are > a. standardised by the ietf > b. standardised by some other body > c. not appropriate for standardisation > Ideally other standards bodies would have the same view of this = breakdown, to avoid the danger that there end up being 0 or 2 standards = where there should be 1. > =20 > I think the I-D and discussion are great start on 1 and 2. To get a = BoF we need a really clear answer on 3 as well. But we don't need a = complete answer before starting any work at the IETF - eg there's = already seems good agreement about doing some metrics related activity = in ippm. Perhaps we can define that already, and define other work a = little later. First I should say that I think we should be careful here: unless it's = very tightly scoped, standardization of functional architecture can IMO = turn into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to prove. We're here to specify protocols such that components and systems can = interoperate toward meeting a common requirement,=20 > Personally I'd find it very useful to have an overview of where other = standards bodies have got to and what work that they're planning. For = example, I remember that Michael B offered to give an overview of the = Broadand forum's framework (=3D functional architecture?), is it = possible to share it please? 802.16 (?) was also mentioned last night, = is any info available about that please? > =20 > One comment about the I-D. I find the Requirements section a bit = premature. For example, I don't think it's that useful for us to agree = requirements about pieces of the functional architecture that are in 3b = & 3c, ie outside the IETF's remit. Agreed. > Finally, I was wondering about how Henning and the authors of the I-D = want to handle input. Do you want comments that you'll gradually work = in, extra authors/editors, split the doc into different bits....? > =20 > Best wishes, > phil > =20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap --Apple-Mail=_9A3E47AD-E9CF-4575-A4E5-08B63801FFAC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Hi, Philip, = all,

On 8 Nov 2012, at 11:31, philip.eardley@bt.com = wrote:

Couple of = thoughts after the LMAP bar bof last night.
 
Thank-you to Henning and = James for organising and initiating the meeting and the I-D about this = important work.
 
I think we should distinguish several = topics:
 
1. Use cases -=46rom an IETF perspective the aim (I = think) is motivational and to show the broad scope. they don't have to = be completely exhaustive. 
 
2. Functional architecture - I think it's important for = us to agree the high level functions (starting from something like = Henning's picture in the plenary, or the discussion Marcelo triggered = about 'Starting the work description') with enough detail to confirm = consensus). Ideally we would agree a functional architecture with other = standards bodies working in the area. Maybe this can be achieved = informally, via the people who are active in several standards = bodies.
 
3. Which pieces of this functional architecture = are
a. standardised = by the ietf
b. = standardised by some other body
c. not appropriate for standardisation
Ideally other standards bodies would = have the same view of this breakdown, to avoid the danger that there end = up being 0 or 2 standards where there should be 1.
 
I think the I-D and discussion are = great start on 1 and 2. To get a BoF we need a really clear answer = on 3 as well. But we don't need a complete answer before starting any = work at the IETF - eg there's already seems good agreement about = doing some metrics related activity in ippm. Perhaps we can define that = already, and define other work a little = later.

First I = should say that I think we should be careful here: unless it's very = tightly scoped, standardization of functional architecture can IMO turn = into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to = prove.

We're here to specify protocols such = that components and systems can interoperate toward meeting a common = requirement, 

Personally I'd find it very = useful to have an overview of where other standards bodies have got to = and what work that they're planning. For example, I remember that = Michael B offered to give an overview of the Broadand forum's framework = (=3D functional architecture?), is it possible to share it please? = 802.16 (?) was also mentioned last night, is any info available about = that please?
 
One comment about the I-D. I find the Requirements = section a bit premature. For example, I don't think it's that useful for = us to agree requirements about pieces of the functional architecture = that are in 3b & 3c, ie outside the IETF's = remit.

Agreed.

Finally, I was wondering about how Henning and the = authors of the I-D want to handle input. Do you want comments that = you'll gradually work in, extra authors/editors, split the doc into = different bits....?
 
Best wishes,
phil
 
____________________________________________= ___
lmap mailing list
lmap@ietf.org
https://www.ietf.org/m= ailman/listinfo/lmap

= --Apple-Mail=_9A3E47AD-E9CF-4575-A4E5-08B63801FFAC-- From trammell@tik.ee.ethz.ch Thu Nov 8 13:44:53 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A4FD721F855E for ; Thu, 8 Nov 2012 13:44:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uugqCeWn2yzD for ; Thu, 8 Nov 2012 13:44:52 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 7280921F8944 for ; Thu, 8 Nov 2012 13:44:51 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id 4F5ADD930A; Thu, 8 Nov 2012 22:44:50 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Yoduc26DiF9V; Thu, 8 Nov 2012 22:44:50 +0100 (MET) Received: from dhcp-447b.meeting.ietf.org (dhcp-447b.meeting.ietf.org [130.129.68.123]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id 45EEAD9308; Thu, 8 Nov 2012 22:44:49 +0100 (MET) Content-Type: multipart/alternative; boundary="Apple-Mail=_C4C3439C-C768-4FF2-A86D-8197F7FF456A" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Brian Trammell In-Reply-To: <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> Date: Thu, 8 Nov 2012 16:44:47 -0500 Message-Id: References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> To: philip.eardley@bt.com X-Mailer: Apple Mail (2.1499) Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 21:44:53 -0000 --Apple-Mail=_C4C3439C-C768-4FF2-A86D-8197F7FF456A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 (Oops, hit send instead of save :/ please stand by for the rest of this = message...) On 8 Nov 2012, at 16:41, Brian Trammell wrote: > Hi, Philip, all, >=20 > On 8 Nov 2012, at 11:31, philip.eardley@bt.com wrote: >=20 >> Couple of thoughts after the LMAP bar bof last night. >> =20 >> Thank-you to Henning and James for organising and initiating the = meeting and the I-D about this important work. >> =20 >> I think we should distinguish several topics: >> =20 >> 1. Use cases -=46rom an IETF perspective the aim (I think) is = motivational and to show the broad scope. they don't have to be = completely exhaustive.=20 >> =20 >> 2. Functional architecture - I think it's important for us to agree = the high level functions (starting from something like Henning's picture = in the plenary, or the discussion Marcelo triggered about 'Starting the = work description') with enough detail to confirm consensus). Ideally we = would agree a functional architecture with other standards bodies = working in the area. Maybe this can be achieved informally, via the = people who are active in several standards bodies. >> =20 >> 3. Which pieces of this functional architecture are >> a. standardised by the ietf >> b. standardised by some other body >> c. not appropriate for standardisation >> Ideally other standards bodies would have the same view of this = breakdown, to avoid the danger that there end up being 0 or 2 standards = where there should be 1. >> =20 >> I think the I-D and discussion are great start on 1 and 2. To get a = BoF we need a really clear answer on 3 as well. But we don't need a = complete answer before starting any work at the IETF - eg there's = already seems good agreement about doing some metrics related activity = in ippm. Perhaps we can define that already, and define other work a = little later. >=20 > First I should say that I think we should be careful here: unless it's = very tightly scoped, standardization of functional architecture can IMO = turn into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to prove. >=20 > We're here to specify protocols such that components and systems can = interoperate toward meeting a common requirement,=20 >=20 >> Personally I'd find it very useful to have an overview of where other = standards bodies have got to and what work that they're planning. For = example, I remember that Michael B offered to give an overview of the = Broadand forum's framework (=3D functional architecture?), is it = possible to share it please? 802.16 (?) was also mentioned last night, = is any info available about that please? >> =20 >> One comment about the I-D. I find the Requirements section a bit = premature. For example, I don't think it's that useful for us to agree = requirements about pieces of the functional architecture that are in 3b = & 3c, ie outside the IETF's remit. >=20 > Agreed. >=20 >> Finally, I was wondering about how Henning and the authors of the I-D = want to handle input. Do you want comments that you'll gradually work = in, extra authors/editors, split the doc into different bits....? >> =20 >> Best wishes, >> phil >> =20 >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap --Apple-Mail=_C4C3439C-C768-4FF2-A86D-8197F7FF456A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 trammell@tik.ee.ethz.ch> = wrote:
Hi, Philip, all,

On 8 = Nov 2012, at 11:31, philip.eardley@bt.com = wrote:

Couple of = thoughts after the LMAP bar bof last night.
 
Thank-you to Henning and = James for organising and initiating the meeting and the I-D about this = important work.
 
I think we should distinguish several = topics:
 
1. Use cases -=46rom an IETF perspective the aim (I = think) is motivational and to show the broad scope. they don't have to = be completely exhaustive. 
 
2. Functional architecture - I think it's important for = us to agree the high level functions (starting from something like = Henning's picture in the plenary, or the discussion Marcelo triggered = about 'Starting the work description') with enough detail to confirm = consensus). Ideally we would agree a functional architecture with other = standards bodies working in the area. Maybe this can be achieved = informally, via the people who are active in several standards = bodies.
 
3. Which pieces of this functional architecture = are
a. standardised = by the ietf
b. = standardised by some other body
c. not appropriate for standardisation
Ideally other standards bodies would = have the same view of this breakdown, to avoid the danger that there end = up being 0 or 2 standards where there should be 1.
 
I think the I-D and discussion are = great start on 1 and 2. To get a BoF we need a really clear answer = on 3 as well. But we don't need a complete answer before starting any = work at the IETF - eg there's already seems good agreement about = doing some metrics related activity in ippm. Perhaps we can define that = already, and define other work a little = later.

First I = should say that I think we should be careful here: unless it's very = tightly scoped, standardization of functional architecture can IMO turn = into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to = prove.

We're here to specify protocols such = that components and systems can interoperate toward meeting a common = requirement, 

Personally I'd find it very = useful to have an overview of where other standards bodies have got to = and what work that they're planning. For example, I remember that = Michael B offered to give an overview of the Broadand forum's framework = (=3D functional architecture?), is it possible to share it please? = 802.16 (?) was also mentioned last night, is any info available about = that please?
 
One comment about the I-D. I find the Requirements = section a bit premature. For example, I don't think it's that useful for = us to agree requirements about pieces of the functional architecture = that are in 3b & 3c, ie outside the IETF's = remit.

Agreed.
Finally, I was wondering about how Henning and the = authors of the I-D want to handle input. Do you want comments that = you'll gradually work in, extra authors/editors, split the doc into = different bits....?
 
Best wishes,
phil
 
____________________________________________= ___
lmap mailing list
lmap@ietf.org
https://www.ietf.org/m= ailman/listinfo/lmap

_________= ______________________________________
lmap mailing list
lmap@ietf.org
https://www.ietf.org/ma= ilman/listinfo/lmap

= --Apple-Mail=_C4C3439C-C768-4FF2-A86D-8197F7FF456A-- From bs7652@att.com Thu Nov 8 14:01:36 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 303C621F8860 for ; Thu, 8 Nov 2012 14:01:36 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.549 X-Spam-Level: X-Spam-Status: No, score=-106.549 tagged_above=-999 required=5 tests=[AWL=0.050, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rbv4BMZA6T1f for ; Thu, 8 Nov 2012 14:01:35 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id 1F1E021F880A for ; Thu, 8 Nov 2012 14:01:35 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo07.seg.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id fbb2c905.2aab09007940.41899.00-595.118538.nbfkord-smmo07.seg.att.com (envelope-from ); Thu, 08 Nov 2012 22:01:35 +0000 (UTC) X-MXL-Hash: 509c2bbf5e61be0d-5931ad4baec286c90aa85316ed8d1a2e2d5d721e Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 5bb2c905.0.41799.00-471.118286.nbfkord-smmo07.seg.att.com (envelope-from ); Thu, 08 Nov 2012 22:01:26 +0000 (UTC) X-MXL-Hash: 509c2bb64eec5347-48a6f5598faaf695bb525d267af67155ba416952 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8M1O9N017668; Thu, 8 Nov 2012 17:01:25 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA8M1EJI017187 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 8 Nov 2012 17:01:19 -0500 Received: from GAALPA1MSGHUB9B.ITServices.sbc.com (gaalpa1msghub9b.itservices.sbc.com [130.8.36.88]) by sflint04.pst.cso.att.com (RSA Interceptor); Thu, 8 Nov 2012 17:00:50 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9B.ITServices.sbc.com ([130.8.36.88]) with mapi id 14.02.0318.001; Thu, 8 Nov 2012 17:00:49 -0500 From: "STARK, BARBARA H" To: "philip.eardley@bt.com" , "mlinsner@cisco.com" , "lmap@ietf.org" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQAOl1eAAADHYIAACT7MoA== Date: Thu, 8 Nov 2012 22:00:48 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> In-Reply-To: <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.158.133] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=H6BsNZki c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=8BrlyXNxoVEA:10 a=WUZhNrZLDssA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bbd-PDihl1AA:10 a=e9qsufxtAAAA:8 a=48vgC7mUAAAA:8] X-AnalysisOut: [ a=AUd_NHdVAAAA:8 a=BtMpnLeseNAVD7ousOoA:9 a=CjuIK1q_8ugA:] X-AnalysisOut: [10 a=W1qU_X6G3J8A:10 a=lZB815dzVvQA:10 a=JfD0Fch1gWkA:10 a] X-AnalysisOut: [=Hz7IrDYlS0cA:10 a=NWVoK91CQyQA:10 a=tXbx8Cdvhw5MKZ_8:21 a] X-AnalysisOut: [=UdV-8vKON4oGfG0f:21] Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 22:01:36 -0000 To be painfully blunt --=20 Measurements for regulatory purposes are a cost. They generate no revenue. We heard regulators telling us yesterday that they want to mandate operator= s provide them with performance measurements. I need to know what the minimum set of requirements is that an operator wou= ld need to comply with, in order to meet the regulatory requirements. The g= oal is to minimize the cost of the regulation.=20 If the goals of other parties can be met without adding cost to the regulat= orily-mandated architecture, then that's wonderful, and I'm happy to accomm= odate them. If the needs of others add cost to the mandated architecture, t= hen that's very, very bad. To me, this means that all other use cases are o= ptional.=20 As I mentioned in my other email, we do extensive measurements today. We do= this to increase customer satisfaction (which decreases churn and keeps cu= stomers loyal) and to decrease help desk and other operational costs, by ha= ving effective test mechanisms and data that allows us to do proactive netw= ork maintenance and management. But we don't need lmap to help us with this= . If others do, ok, but it better not add to the cost of the solution that = the regulators are going to make operators do to meet regulatory mandates. Barbara > -----Original Message----- > From: philip.eardley@bt.com [mailto:philip.eardley@bt.com] >=20 > similar to #1 > Operators want performance data for "mass market" broadband access > network connectivity services. >=20 > One difference between the use cases is the timescale of interest. For > example regulators are probably more interested in monthly averages over > large numbers of users whereas in #4 ISPs are probably more interested in > finer grain information. >=20 > Barbara, > I'm interested in why you want to narrow the focus to #1 (is it that your > sceptical about the importance of the other use cases, or that they don't > need standardisation, or something else?) >=20 > thanks > phil >=20 > ________________________________________ > From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of Marc > Linsner [mlinsner@cisco.com] > Sent: 08 November 2012 20:38 > To: STARK, BARBARA H; lmap@ietf.org > Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory > vs. Nice to Have >=20 > Barbara, >=20 > Thanks for documenting the use cases below. I would like to submit use ca= se > #5. >=20 > 5. OTT providers of services. OTT providers are interested in the QOE of= their > customers and may instanciate a measurement client/controller/server of > their own accord. >=20 > -Marc- >=20 > -----Original Message----- > From: "STARK, BARBARA H" > Date: Thursday, November 8, 2012 1:41 PM > To: "lmap@ietf.org" > Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. > Nice to Have >=20 > >Following are my suggestions for problem scope, listing of use cases > >that have been mentioned to date, and my thoughts on which we need to > >focus on solving. > > > >Proposed Scope: Performance measurements for mass market broadband > >connectivity services. > > > >Use Cases > >1. Regulators want performance data for "mass market" broadband access > >network connectivity services. Regulators are the primary user in this > >use case. Regulators may have business requirements that involve access > >by others to the data collected for this use case (e.g., researchers > >are to be allowed access to data, or broadband customers should be > >allowed to access data for their own service), but these others are not > >considered primary users in this case. > > > >2. Researchers want to be able to run and collect data from a variety > >of tests performed against broadband connections (end-to-end, segment > >by segment, maybe or maybe not limited to mass market or the access > >network). Researchers are the primary user. > > > >3. Broadband "mass market" customers want to run tests against their > >broadband connections and see results of tests performed against their > >broadband connection. Regulators, researchers, and customers are the 3 > >sets of "users" that are covered by these 3 use cases. Broadband > >customers are the primary user. > > > >4. ISPs want tests to assist in troubleshooting, and network > >architecting and engineering. [Note that I'm just listing this as a use > >case that has been mentioned, and not as something that is being > >requested by all ISPs > >-- no value judgment is implied by listing it here.] ISPs are the > >primary user. > > > >Which Use Cases Are Mandatory > >Based on last night's discussion, I believe that only the first use > >case is mandatory. The others are fine to consider, if they don't get > >in the way of solving use case #1. > > > >If we can agree on these points, then I think the next step is to > >identify the Business Requirements of regulators. Ideally, we would get > >input from regulators around the world. We certainly need to try to > >solicit their input. However, I think it's a good idea to start by > >documenting the business requirements of the one regulator who we do > >have. Getting started may incite others to join in. Other "users" (or > >proxies of other users) are also welcome to provide their business > >requirements -- but they need to understand that satisfying their > >requirements is considered "nice to have" and not mandatory. > > > >I'm going to work next on putting together a start for Business > >Requirements. > >Barbara > >_______________________________________________ > >lmap mailing list > >lmap@ietf.org > >https://www.ietf.org/mailman/listinfo/lmap >=20 >=20 > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From philip.eardley@bt.com Thu Nov 8 14:04:37 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6A20821F87F2 for ; Thu, 8 Nov 2012 14:04:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.598 X-Spam-Level: X-Spam-Status: No, score=-103.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uDdIIuv4dpzO for ; Thu, 8 Nov 2012 14:04:36 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp61.intersmtp.com [62.239.224.234]) by ietfa.amsl.com (Postfix) with ESMTP id 4D98721F8681 for ; Thu, 8 Nov 2012 14:04:36 -0800 (PST) Received: from EVMHT61-UKRD.domain1.systemhost.net (10.36.3.127) by RDW083A005ED61.smtp-e1.hygiene.service (10.187.98.10) with Microsoft SMTP Server (TLS) id 8.3.279.1; Thu, 8 Nov 2012 22:04:35 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.2.94]) by EVMHT61-UKRD.domain1.systemhost.net ([10.36.3.127]) with mapi; Thu, 8 Nov 2012 22:04:35 +0000 From: To: Date: Thu, 8 Nov 2012 22:04:34 +0000 Thread-Topic: [lmap] some thoughts about the LMAP activity Thread-Index: Ac29+eeJYWiv9Ta8Qf6btHaFFsDFoQAAmJt3 Message-ID: <9510D26531EF184D9017DF24659BB87F33F336A0B4@EMV65-UKRD.domain1.systemhost.net> References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net>, <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> In-Reply-To: <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> Accept-Language: en-US, en-GB Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_9510D26531EF184D9017DF24659BB87F33F336A0B4EMV65UKRDdoma_" MIME-Version: 1.0 Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 22:04:37 -0000 --_000_9510D26531EF184D9017DF24659BB87F33F336A0B4EMV65UKRDdoma_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable [cut] [Brian] First I should say that I think we should be careful here: unless it's very= tightly scoped, standardization of functional architecture can IMO turn in= to a bit of a rathole. I may be generalizing a bit much from my experience,= but the closer the functional architecture comes to simply being a descrip= tion of the minimum set of boxes and lines you need to illustrate or explai= n the terminology and operation of a protocol, the better. The closer it ge= ts to being an implementation report of a particular attempt at solving a p= articular set of requirements, the less resilient and less useful it is lik= ely to prove. We're here to specify protocols such that components and systems can intero= perate toward meeting a common requirement, [phil] I agree. For me, a functional architecture can be a good way of gene= rating a common view on what a WG is trying to do in its specification phas= e. simplicity is generally good as it's easier to understand, so easier to = see if there's a common view - but not so simple that it hides where there'= s actually some fundamental disagreement about what the WG is trying to spe= cify phil --_000_9510D26531EF184D9017DF24659BB87F33F336A0B4EMV65UKRDdoma_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
[cut]
 
[Brian]
First I should say that I think we should be careful here: unless it's= very tightly scoped, standardization of functional architecture can IMO tu= rn into a bit of a rathole. I may be generalizing a bit much from my experi= ence, but the closer the functional architecture comes to simply being a description of the minimum set of box= es and lines you need to illustrate or explain the terminology and operatio= n of a protocol, the better. The closer it gets to being an implementation = report of a particular attempt at solving a particular set of requirements, the less resilient and less usef= ul it is likely to prove.

We're here to specify protocols such that components and systems can i= nteroperate toward meeting a common requirement, 
 
[phil] I agree. For me, a functional architectur= e can be a good way of generating a common view on what a WG is trying to d= o in its specification phase. simplicity is generally good as it's easier t= o understand, so easier to see if there's a common view - but not so simple that it hides where there's actu= ally some fundamental disagreement about what the WG is trying to specify
phil
--_000_9510D26531EF184D9017DF24659BB87F33F336A0B4EMV65UKRDdoma_-- From trammell@tik.ee.ethz.ch Thu Nov 8 14:07:19 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0BDC121F8860 for ; Thu, 8 Nov 2012 14:07:19 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7Z1kqyhacfJW for ; Thu, 8 Nov 2012 14:07:18 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 1D6B721F8836 for ; Thu, 8 Nov 2012 14:07:18 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id 735B2D930A; Thu, 8 Nov 2012 23:07:17 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id cnSFAxF2+z0Y; Thu, 8 Nov 2012 23:07:17 +0100 (MET) Received: from dhcp-549e.meeting.ietf.org (dhcp-549e.meeting.ietf.org [130.129.84.158]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id 9BF5ED9308; Thu, 8 Nov 2012 23:07:16 +0100 (MET) Content-Type: multipart/alternative; boundary="Apple-Mail=_433948A6-2C2F-4975-902E-C1E63F19C44A" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Brian Trammell In-Reply-To: <9510D26531EF184D9017DF24659BB87F33F336A0B4@EMV65-UKRD.domain1.systemhost.net> Date: Thu, 8 Nov 2012 17:07:14 -0500 Message-Id: <5C544B36-F904-4119-A5DA-7CCCF09F944B@tik.ee.ethz.ch> References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net>, <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> <9510D26531EF184D9017DF24659BB87F33F336A0B4@EMV65-UKRD.domain1.systemhost.net> To: philip.eardley@bt.com X-Mailer: Apple Mail (2.1499) Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 22:07:19 -0000 --Apple-Mail=_433948A6-2C2F-4975-902E-C1E63F19C44A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 hi Phil, all, On 8 Nov 2012, at 17:04, philip.eardley@bt.com wrote: > [cut] > =20 > [Brian] > First I should say that I think we should be careful here: unless it's = very tightly scoped, standardization of functional architecture can IMO = turn into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to prove. >=20 > We're here to specify protocols such that components and systems can = interoperate toward meeting a common requirement,=20 > =20 > [phil] I agree. For me, a functional architecture can be a good way of = generating a common view on what a WG is trying to do in its = specification phase. simplicity is generally good as it's easier to = understand, so easier to see if there's a common view - but not so = simple that it hides where there's actually some fundamental = disagreement about what the WG is trying to specify Absolutely. I think maybe I'm getting caught up on terminology here; = perhaps I'd prefer to call it a "reference architecture", to capture = that it's a description of how a protocol or set of protocols are = designed to work, as opposed to a proscription for how they must be = deployed.=20 (The rest of my thought in a moment...) cheers, Brian --Apple-Mail=_433948A6-2C2F-4975-902E-C1E63F19C44A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 hi Phil, = all,

On 8 Nov 2012, at 17:04, philip.eardley@bt.com = wrote:

 
[Brian]
First I should say that I = think we should be careful here: unless it's very tightly scoped, = standardization of functional architecture can IMO turn into a bit of a = rathole. I may be generalizing a bit much from my experience, but the = closer the functional architecture comes to simply being a description = of the minimum set of boxes and lines you need to illustrate or explain = the terminology and operation of a protocol, the better. The closer it = gets to being an implementation report of a particular attempt at = solving a particular set of requirements, the less resilient and less = useful it is likely to prove.

We're here = to specify protocols such that components and systems can interoperate = toward meeting a common requirement, 
 
[phil] I = agree. For me, a functional architecture can be a good way of generating = a common view on what a WG is trying to do in its specification phase. = simplicity is generally good as it's easier to understand, so easier to = see if there's a common view - but not so simple that it hides where = there's actually some fundamental disagreement about what the WG is = trying to = specify

Abs= olutely. I think maybe I'm getting caught up on terminology here; =  perhaps I'd prefer to call it a "reference architecture", to = capture that it's a description of how a protocol or set of protocols = are designed to work, as opposed to a proscription for how they must be = deployed. 

(The rest of my thought in a = moment...)

cheers,

Brian=

= --Apple-Mail=_433948A6-2C2F-4975-902E-C1E63F19C44A-- From trammell@tik.ee.ethz.ch Thu Nov 8 14:32:10 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A603121F894D for ; Thu, 8 Nov 2012 14:32:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=-0.000, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OmTlFTHNB+mF for ; Thu, 8 Nov 2012 14:32:09 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 42E1421F88B4 for ; Thu, 8 Nov 2012 14:32:09 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id 9F71ED9321; Thu, 8 Nov 2012 23:32:07 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id UMawJUUlQmLk; Thu, 8 Nov 2012 23:32:07 +0100 (MET) Received: from dhcp-549e.meeting.ietf.org (dhcp-549e.meeting.ietf.org [130.129.84.158]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id C0584D9308; Thu, 8 Nov 2012 23:32:06 +0100 (MET) Content-Type: multipart/alternative; boundary="Apple-Mail=_8B1C3904-9180-4656-BBE1-617E2858FF28" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Brian Trammell In-Reply-To: <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> Date: Thu, 8 Nov 2012 17:32:03 -0500 Message-Id: References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> To: philip.eardley@bt.com X-Mailer: Apple Mail (2.1499) Cc: lmap@ietf.org Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 22:32:10 -0000 --Apple-Mail=_8B1C3904-9180-4656-BBE1-617E2858FF28 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hi, Phil, all, Apologies for the early send; my thought got fragmented by lack of = sleep, apparently. Here it is reassembled. On 8 Nov 2012, at 11:31, philip.eardley@bt.com wrote: > Couple of thoughts after the LMAP bar bof last night. > =20 > Thank-you to Henning and James for organising and initiating the = meeting and the I-D about this important work. > =20 > I think we should distinguish several topics: > =20 > 1. Use cases -=46rom an IETF perspective the aim (I think) is = motivational and to show the broad scope. they don't have to be = completely exhaustive.=20 > =20 > 2. Functional architecture - I think it's important for us to agree = the high level functions (starting from something like Henning's picture = in the plenary, or the discussion Marcelo triggered about 'Starting the = work description') with enough detail to confirm consensus). Ideally we = would agree a functional architecture with other standards bodies = working in the area. Maybe this can be achieved informally, via the = people who are active in several standards bodies. > =20 > 3. Which pieces of this functional architecture are > a. standardised by the ietf > b. standardised by some other body > c. not appropriate for standardisation > Ideally other standards bodies would have the same view of this = breakdown, to avoid the danger that there end up being 0 or 2 standards = where there should be 1. > =20 > I think the I-D and discussion are great start on 1 and 2. To get a = BoF we need a really clear answer on 3 as well. But we don't need a = complete answer before starting any work at the IETF - eg there's = already seems good agreement about doing some metrics related activity = in ippm. Perhaps we can define that already, and define other work a = little later. First I should say that I think we should be careful here: unless it's = very tightly scoped, standardization of functional architecture can IMO = turn into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to prove. We're here to specify protocols such that components and systems can = interoperate toward meeting a common requirement: if there's nothing to = be gained by having independent implementations working with each other, = then a standard protocol is useful as a form of documentation for the = community, but is no more useful than the implementors' making their = interface specifications available. I believe in the use cases presented in the draft, and especially the = regulatory use case presented in the tsvarea/plenary presentations, = standard protocols both for control as well as for metrics data exchange = would have a lot of value. For regulatory/verification purposes it is = _absolutely_ essential that the definition of the metrics produced by = different vendors' equipment and operators' implementations are = transparently defined and openly comparable (within a known and reported = margin of error); it is also vitally important for troubleshooting of = complex problems with multiple partial causes.=20 On the control side, this is also vitally important. Whether for = regulatory, research, or troubleshooting use cases, measurement becomes = much easier to deploy when it's possible to grab a (tested) component = off the shelf, whether that be simple active delay probing software on = an end system or a box in the core estimating performance = characteristics passively, and know that it will interoperate with = existing data collection and measurement control already deployed. So I guess I'd come at this same question from a slightly different, = perhaps too IETF-centric angle: what is this venue (the IETF) good for, = and how much does work alluded to so far in the LMAP effort, as = outlined, overlap with work already underway in the IETF? Does there a = exist a venue where there is more overlap?=20 Here, it's clear that standard definitions of performance metrics fit = very well with IPPM. It's a little less clear where the control = interactions would fit. I don't see a lot of overlap here with NETCONF, = for example, but at the same time, I would submit that it probably = doesn't make sense to standardize measurement control in one venue and = metrics in another, especially as they're most probably done at the same = layer. > Personally I'd find it very useful to have an overview of where other = standards bodies have got to and what work that they're planning. For = example, I remember that Michael B offered to give an overview of the = Broadand forum's framework (=3D functional architecture?), is it = possible to share it please? 802.16 (?) was also mentioned last night, = is any info available about that please? > =20 > One comment about the I-D. I find the Requirements section a bit = premature. For example, I don't think it's that useful for us to agree = requirements about pieces of the functional architecture that are in 3b = & 3c, ie outside the IETF's remit. Agreed. >=20 Cheers, Brian --Apple-Mail=_8B1C3904-9180-4656-BBE1-617E2858FF28 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Hi, Phil, all,

Apologies for the early send; = my thought got fragmented by lack of sleep, apparently. Here it is = reassembled.

On 8 Nov 2012, at 11:31, = philip.eardley@bt.com = wrote:

Couple of = thoughts after the LMAP bar bof last night.
 
Thank-you to Henning and = James for organising and initiating the meeting and the I-D about this = important work.
 
I think we should distinguish several = topics:
 
1. Use cases -=46rom an IETF perspective the aim (I = think) is motivational and to show the broad scope. they don't have to = be completely exhaustive. 
 
2. Functional architecture - I think it's important for = us to agree the high level functions (starting from something like = Henning's picture in the plenary, or the discussion Marcelo triggered = about 'Starting the work description') with enough detail to confirm = consensus). Ideally we would agree a functional architecture with other = standards bodies working in the area. Maybe this can be achieved = informally, via the people who are active in several standards = bodies.
 
3. Which pieces of this functional architecture = are
a. standardised = by the ietf
b. = standardised by some other body
c. not appropriate for standardisation
Ideally other standards bodies would = have the same view of this breakdown, to avoid the danger that there end = up being 0 or 2 standards where there should be 1.
 
I think the I-D and discussion are = great start on 1 and 2. To get a BoF we need a really clear answer = on 3 as well. But we don't need a complete answer before starting any = work at the IETF - eg there's already seems good agreement about = doing some metrics related activity in ippm. Perhaps we can define that = already, and define other work a little = later.

First I = should say that I think we should be careful here: unless it's very = tightly scoped, standardization of functional architecture can IMO turn = into a bit of a rathole. I may be generalizing a bit much from my = experience, but the closer the functional architecture comes to simply = being a description of the minimum set of boxes and lines you need to = illustrate or explain the terminology and operation of a protocol, the = better. The closer it gets to being an implementation report of a = particular attempt at solving a particular set of requirements, the less = resilient and less useful it is likely to = prove.

We're here to specify protocols such = that components and systems can interoperate toward meeting a common = requirement: if there's nothing to be gained by having independent = implementations working with each other, then a standard protocol is = useful as a form of documentation for the community, but is no more = useful than the implementors' making their interface specifications = available.

I believe in the use cases presented = in the draft, and especially the regulatory use case presented in the = tsvarea/plenary presentations, standard protocols both for control as = well as for metrics data exchange would have a lot of value. For = regulatory/verification purposes it is _absolutely_ essential that the = definition of the metrics produced by different vendors' equipment and = operators' implementations are transparently defined and openly = comparable (within a known and reported margin of error); it is also = vitally important for troubleshooting of complex problems with multiple = partial causes. 

On the control side, this = is also vitally important. Whether for regulatory, research, or = troubleshooting use cases, measurement becomes much easier to deploy = when it's possible to grab a (tested) component off the shelf, whether = that be simple active delay probing software on an end system or a box = in the core estimating performance characteristics passively, and know = that it will interoperate with existing data collection and measurement = control already deployed.

So I guess I'd come = at this same question from a slightly different, perhaps too = IETF-centric angle: what is this venue (the IETF) good for, and how much = does work alluded to so far in the LMAP effort, as outlined, overlap = with work already underway in the IETF? Does there a exist a venue where = there is more overlap? 

Here, it's clear = that standard definitions of performance metrics fit very well with = IPPM. It's a little less clear where the control interactions would fit. = I don't see a lot of overlap here with NETCONF, for example, but at the = same time, I would submit that it probably doesn't make sense to = standardize measurement control in one venue and metrics in another, = especially as they're most probably done at the same = layer.

Personally I'd find it very = useful to have an overview of where other standards bodies have got to = and what work that they're planning. For example, I remember that = Michael B offered to give an overview of the Broadand forum's framework = (=3D functional architecture?), is it possible to share it please? = 802.16 (?) was also mentioned last night, is any info available about = that please?
 
One comment about the I-D. I find the Requirements = section a bit premature. For example, I don't think it's that useful for = us to agree requirements about pieces of the functional architecture = that are in 3b & 3c, ie outside the IETF's = remit.

Agreed.
Cheers,

Brian

= --Apple-Mail=_8B1C3904-9180-4656-BBE1-617E2858FF28-- From alessandro.capello@telecomitalia.it Thu Nov 8 16:04:28 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8CDB121F8B69 for ; Thu, 8 Nov 2012 16:04:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.882 X-Spam-Level: X-Spam-Status: No, score=0.882 tagged_above=-999 required=5 tests=[BAYES_50=0.001, HELO_EQ_IT=0.635, HOST_EQ_IT=1.245, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Kr97Wl6ufvS for ; Thu, 8 Nov 2012 16:04:27 -0800 (PST) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by ietfa.amsl.com (Postfix) with ESMTP id D340121F8B57 for ; Thu, 8 Nov 2012 16:04:26 -0800 (PST) Content-Type: multipart/mixed; boundary="_4dfe5bbb-72c4-40af-bf48-e267012f3706_" Received: from GRFHUB702BA020.griffon.local (10.188.101.112) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.3.279.5; Fri, 9 Nov 2012 01:04:11 +0100 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by GRFHUB702BA020.griffon.local ([10.188.101.112]) with mapi; Fri, 9 Nov 2012 01:04:11 +0100 From: Capello Alessandro To: "lmap@ietf.org" Date: Fri, 9 Nov 2012 01:04:06 +0100 Thread-Topic: [lmap] some thoughts about the LMAP activity Thread-Index: Ac2+AOpf4Gf3XhiEQheREQjghil1TAACDWRQ Message-ID: <36A93B31228D3B49B691AD31652BCAE9A5A6F5FA7A@GRFMBX702BA020.griffon.local> References: <9510D26531EF184D9017DF24659BB87F33F336A0AD@EMV65-UKRD.domain1.systemhost.net> <7AC51366-EBAD-4E16-950A-C08029C39939@tik.ee.ethz.ch> In-Reply-To: Accept-Language: en-US, it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, it-IT x-ti-disclaimer: Disclaimer1 MIME-Version: 1.0 Subject: Re: [lmap] some thoughts about the LMAP activity X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 00:04:28 -0000 --_4dfe5bbb-72c4-40af-bf48-e267012f3706_ Content-Type: multipart/alternative; boundary="_000_36A93B31228D3B49B691AD31652BCAE9A5A6F5FA7AGRFMBX702BA02_" --_000_36A93B31228D3B49B691AD31652BCAE9A5A6F5FA7AGRFMBX702BA02_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, I support Brian's point on interworking. Looking at the use cases, you can envision a residential user that: * has a measurement probe embedded in the home gateway managed by the ISP * has a dedicated appliance in the home network to run tests for regulators * has a software probe on the PC that runs tests on behalf of an OTT player * ... I think that the controlled exchange of data, requests, etc. between differ= ent administrative domains (ISPs, regulators, OTT players, ...) should be p= art of the functions required (something like peering relationships but app= lied to measurement instead of routing). In my (probably biased) mind ISPs = have a key role here but it looks really like a win-win game to me (larger = data set, better accuracy, etc.). Regards, Alessandro Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. [cid:00000000000000000000000000000003@TI.Disclaimer]Rispetta l'ambiente. No= n stampare questa mail se non ? necessario. --_000_36A93B31228D3B49B691AD31652BCAE9A5A6F5FA7AGRFMBX702BA02_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi all,

 

I support Brian's point on interworking.

 

Looking at the use cases, you can envision a residential user that:=

• has a measurement probe embedded in the home gateway manage= d by the ISP

• has a dedicated appliance in the home network to run tests = for regulators

• has a software probe on the PC that runs tests on behalf of= an OTT player

• ...

 

I think that the controlled exchange of data, requests, etc. betwee= n different administrative domains (ISPs, regulators, OTT players, ...) sho= uld be part of the functions required (something like peering relationships but applied to measurement = instead of routing). In my (probably biased) mind ISPs have a key role here= but it looks really like a win-win game to me (larger data set, better acc= uracy, etc.).

 

Regards,

Alessandro

 

= Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigoro= samente vietate. Qualora abbiate ricevuto questo documento per errore siete= cortesemente pregati di darne immediata comunicazione al mittente e di pro= vvedere alla sua distruzione, Grazie.

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only.= Dissemination, copying, printing or use by anybody else is unauthorised. I= f you are not the intended recipient, please delete this message and any at= tachments and advise the sender by return e-mail, Thanks.

3D"rispettaRispetta= l'ambiente. Non stampare questa mail se non è necessario.

--_000_36A93B31228D3B49B691AD31652BCAE9A5A6F5FA7AGRFMBX702BA02_-- --_4dfe5bbb-72c4-40af-bf48-e267012f3706_ Content-Description: logo Ambiente_foglia2.jpg Content-Type: image/jpeg; name="logo Ambiente_foglia2.jpg" Content-Disposition: inline; filename="logo Ambiente_foglia2.jpg" Content-Transfer-Encoding: base64 Content-ID: 00000000000000000000000000000003@TI.Disclaimer R0lGODlhGgAoANU5AEiFNnikNyRvNcvYOafCOEOEW3DO3jB2NqjGs9ny9o+zOIOrN+L1+G+ggbzo 8GCUN1SNNv///zx+NrPJOL/ROYPV44zY5YuzmrfQwCZxQlKNaMXZzOfy8NTi2TV6TuLs5vX8/ez5 +4yzmtTj2cXr8mCXdKni62ycN5/f6aDf6X2qjrPl7rLl7Zu6OJbb53nS4PH188bs8sXZzfH18pq9 p0SDWxhnNWbL3NfgOf///wAAAAAAAAAAAAAAAAAAAAAAACH5BAEAADkALAAAAAAaACgAAAb/wJxw SBQ6WMWkMmm6kZbQ5OvmjFoT1JuBYYWmsrcXqJvEgm8WctFypnI66pyjfXMhCmqGgR4r2S5dIBV0 FjI2h3BRX20VHAUCEjZ4UHNtFo42BA+HCEskbQYOIwU2CjgBNgAZM0kMbSYcIjYCBDg4LTYLf0V6 YCghNBk2DwO2OASZqkS9VBYMCB6pE8bGucidOYJUoaOptdQTFDg2ATgHGkIs2wyyAqbUtgICCwfl uh8ge1sNNhDF8LYiHYKAg4INBJUS8FsAEF6AA6lsHWjg4gYKDDZONGwIAIAtCAX2hPAgYSNHj6ds 3KiA8V3DAQGm2epoS4HKFQ0EmMRhc97Mwge2kN1IoIGgyQECD5wQUO6YygTkdtpCdSgqDl0VoDaV OmDBpm+oTCQoABQgBZfGbIrDAUBDAgcNDnDs98/WA504Buxa0RKgzVkB/h0oi+pDjgQhHtU1RgDi oY6l8gpAJyTBCBsSFtuC6XjWTBuJhIRAgFkmwAmoAkM4mCQCBmEB1sIDIKAFRGytYfDrt4CAuAkn qnrYYCXCBxGkqlYtgbtLhOcbMNSw0SBOEmg2VFj/sGHDhRLCNBC3fqFqARWhowQBADs= --_4dfe5bbb-72c4-40af-bf48-e267012f3706_-- From trevor.burbridge@bt.com Fri Nov 9 01:02:33 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 086BA21F85CB for ; Fri, 9 Nov 2012 01:02:33 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.999 X-Spam-Level: X-Spam-Status: No, score=-2.999 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hCmLp3HVoync for ; Fri, 9 Nov 2012 01:02:32 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp61.intersmtp.com [62.239.224.234]) by ietfa.amsl.com (Postfix) with ESMTP id A5A6B21F851F for ; Fri, 9 Nov 2012 01:02:31 -0800 (PST) Received: from EVMHT64-UKRD.domain1.systemhost.net (10.36.3.101) by RDW083A005ED61.smtp-e1.hygiene.service (10.187.98.10) with Microsoft SMTP Server (TLS) id 8.3.279.1; Fri, 9 Nov 2012 09:02:30 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.92]) by EVMHT64-UKRD.domain1.systemhost.net ([10.36.3.101]) with mapi; Fri, 9 Nov 2012 09:02:30 +0000 From: To: , , , Date: Fri, 9 Nov 2012 09:02:29 +0000 Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQAOl1eAAADHYIAACT7MoAAFCFiQ Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 09:02:33 -0000 To give a different perspective, I think that #4 (including both fine per l= ine measurement and overall sampling of products for aggregate network perf= ormance and benchmarking against other ISP/products) is a critical use case= . Some ISPs will benefit, others may not. I's not that all regulators in use = case #1 will want to (or be able to) mandate that this measurement has to t= ake place either. While there are many currently deployed measurement techniques - deploying = and managing tests, for example to home CPE, is not standardised. A harmoni= sed way to run end-to-end tests that can be better indicative of user QoE i= s lacking. A way of detecting performance problems at any point and at any = layer in diverse networks is lacking. Furthermore if the ISP manages or owns much of this equipment, then it is v= ital that they are able to get benefit from running tests on these devices.= In my view if the ISP is not the primary beneficiary (along with ultimatel= y the broadband user) of this activity then these tests will never get depl= oyed on a large scale but will always be restricted to limited and costly p= anels. Trevor. >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >STARK, BARBARA H >Sent: 08 November 2012 22:01 >To: Eardley,PL,Philip,DUB8 R; mlinsner@cisco.com; lmap@ietf.org >Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory >vs. Nice to Have > >To be painfully blunt -- >Measurements for regulatory purposes are a cost. They generate no revenue. >We heard regulators telling us yesterday that they want to mandate >operators provide them with performance measurements. >I need to know what the minimum set of requirements is that an operator >would need to comply with, in order to meet the regulatory requirements. >The goal is to minimize the cost of the regulation. > >If the goals of other parties can be met without adding cost to the >regulatorily-mandated architecture, then that's wonderful, and I'm happy t= o >accommodate them. If the needs of others add cost to the mandated >architecture, then that's very, very bad. To me, this means that all other= use >cases are optional. > >As I mentioned in my other email, we do extensive measurements today. We >do this to increase customer satisfaction (which decreases churn and keeps >customers loyal) and to decrease help desk and other operational costs, by >having effective test mechanisms and data that allows us to do proactive >network maintenance and management. But we don't need lmap to help us >with this. If others do, ok, but it better not add to the cost of the solu= tion >that the regulators are going to make operators do to meet regulatory >mandates. >Barbara > >> -----Original Message----- >> From: philip.eardley@bt.com [mailto:philip.eardley@bt.com] >> >> similar to #1 >> Operators want performance data for "mass market" broadband access >> network connectivity services. >> >> One difference between the use cases is the timescale of interest. For >> example regulators are probably more interested in monthly averages over >> large numbers of users whereas in #4 ISPs are probably more interested i= n >> finer grain information. >> >> Barbara, >> I'm interested in why you want to narrow the focus to #1 (is it that you= r >> sceptical about the importance of the other use cases, or that they don'= t >> need standardisation, or something else?) >> >> thanks >> phil >> >> ________________________________________ >> From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of Marc >> Linsner [mlinsner@cisco.com] >> Sent: 08 November 2012 20:38 >> To: STARK, BARBARA H; lmap@ietf.org >> Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory >> vs. Nice to Have >> >> Barbara, >> >> Thanks for documenting the use cases below. I would like to submit use >case >> #5. >> >> 5. OTT providers of services. OTT providers are interested in the QOE o= f >their >> customers and may instanciate a measurement client/controller/server of >> their own accord. >> >> -Marc- >> >> -----Original Message----- >> From: "STARK, BARBARA H" >> Date: Thursday, November 8, 2012 1:41 PM >> To: "lmap@ietf.org" >> Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. >> Nice to Have >> >> >Following are my suggestions for problem scope, listing of use cases >> >that have been mentioned to date, and my thoughts on which we need to >> >focus on solving. >> > >> >Proposed Scope: Performance measurements for mass market >broadband >> >connectivity services. >> > >> >Use Cases >> >1. Regulators want performance data for "mass market" broadband >access >> >network connectivity services. Regulators are the primary user in this >> >use case. Regulators may have business requirements that involve access >> >by others to the data collected for this use case (e.g., researchers >> >are to be allowed access to data, or broadband customers should be >> >allowed to access data for their own service), but these others are not >> >considered primary users in this case. >> > >> >2. Researchers want to be able to run and collect data from a variety >> >of tests performed against broadband connections (end-to-end, segment >> >by segment, maybe or maybe not limited to mass market or the access >> >network). Researchers are the primary user. >> > >> >3. Broadband "mass market" customers want to run tests against their >> >broadband connections and see results of tests performed against their >> >broadband connection. Regulators, researchers, and customers are the 3 >> >sets of "users" that are covered by these 3 use cases. Broadband >> >customers are the primary user. >> > >> >4. ISPs want tests to assist in troubleshooting, and network >> >architecting and engineering. [Note that I'm just listing this as a use >> >case that has been mentioned, and not as something that is being >> >requested by all ISPs >> >-- no value judgment is implied by listing it here.] ISPs are the >> >primary user. >> > >> >Which Use Cases Are Mandatory >> >Based on last night's discussion, I believe that only the first use >> >case is mandatory. The others are fine to consider, if they don't get >> >in the way of solving use case #1. >> > >> >If we can agree on these points, then I think the next step is to >> >identify the Business Requirements of regulators. Ideally, we would get >> >input from regulators around the world. We certainly need to try to >> >solicit their input. However, I think it's a good idea to start by >> >documenting the business requirements of the one regulator who we do >> >have. Getting started may incite others to join in. Other "users" (or >> >proxies of other users) are also welcome to provide their business >> >requirements -- but they need to understand that satisfying their >> >requirements is considered "nice to have" and not mandatory. >> > >> >I'm going to work next on putting together a start for Business >> >Requirements. >> >Barbara >> >_______________________________________________ >> >lmap mailing list >> >lmap@ietf.org >> >https://www.ietf.org/mailman/listinfo/lmap >> >> >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap From trammell@tik.ee.ethz.ch Fri Nov 9 04:14:50 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 84E4521F8599 for ; Fri, 9 Nov 2012 04:14:50 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.299 X-Spam-Level: X-Spam-Status: No, score=-6.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PJQZUDXer91g for ; Fri, 9 Nov 2012 04:14:49 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 3DA3321F8546 for ; Fri, 9 Nov 2012 04:14:48 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id 1347FD9308; Fri, 9 Nov 2012 13:14:46 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id qmR4asVVKe-w; Fri, 9 Nov 2012 13:14:38 +0100 (MET) Received: from dhcp-447b.meeting.ietf.org (dhcp-447b.meeting.ietf.org [130.129.68.123]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id BA7B3D930A; Fri, 9 Nov 2012 13:14:37 +0100 (MET) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Brian Trammell In-Reply-To: Date: Fri, 9 Nov 2012 07:14:33 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> To: trevor.burbridge@bt.com X-Mailer: Apple Mail (2.1499) Cc: philip.eardley@bt.com, lmap@ietf.org, bs7652@att.com, mlinsner@cisco.com Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 12:14:50 -0000 hi Trevor, +1 I get that the way some companies account for such things may make this = sound terribly na=EFve, but I think the approach I'd take here is if the = deployment of measurement infrastructure within the network will be = mandated in order to allow regulators to to verify that operators follow = laws and regulations pertinent to delivered network performance, thereby = forcing a cost on network operators, then it makes sense to explore = whether this same infrastructure can be used to other (potentially = profitable) purposes, thereby at least recovering some of the capital = outlay therefor. Now, if we end up in a situation that it is _clear_ that the regulatory = use cases are far simpler than any useful troubleshooting use case, or = any value-added "measure my network" use case, such that a regulatory = infrastructure would be measurably cheaper than one capable of multiple = uses, then I would buy the argument that these should be treated = separately. But that just doesn't seem plausible to me. Unless the regulatory use = case is scoped to be severely restricted (say, one trivially computable = metric with a tiny population sample, which would IMO be questionably = useful), the basic metrics (measurement point capabilities), analyses = (collector and post-collector capabilities), data volumes and deployment = patterns (system scale) seem to me to be pretty much the same. (The only = thing from the LMAP draft architecture I don't see as universally = applicable to all of these is the network parameters server, but it's = clear from earlier list discussion that a lot of people seem to be = confused about what that's for in the first place.) Indeed, I'd look at this whole argument from the other side: is it = possible to define a useful reference architecture and protocol for = measurement infrastructures which have as their _primary_ use the = within-the-ISP and from-the-customer-POV troubleshooting use cases, = which is _also_ useful for regulatory verification purposes, which is = incrementally implementable, such that when the mandate does come down, = the regulatory requirements can be met by infrastructure already being = deployed for operations? Best regards, Brian On 9 Nov 2012, at 4:02, trevor.burbridge@bt.com wrote: > To give a different perspective, I think that #4 (including both fine = per line measurement and overall sampling of products for aggregate = network performance and benchmarking against other ISP/products) is a = critical use case. >=20 > Some ISPs will benefit, others may not. I's not that all regulators in = use case #1 will want to (or be able to) mandate that this measurement = has to take place either. >=20 > While there are many currently deployed measurement techniques - = deploying and managing tests, for example to home CPE, is not = standardised. A harmonised way to run end-to-end tests that can be = better indicative of user QoE is lacking. A way of detecting performance = problems at any point and at any layer in diverse networks is lacking. >=20 > Furthermore if the ISP manages or owns much of this equipment, then it = is vital that they are able to get benefit from running tests on these = devices. In my view if the ISP is not the primary beneficiary (along = with ultimately the broadband user) of this activity then these tests = will never get deployed on a large scale but will always be restricted = to limited and costly panels. >=20 > Trevor. >=20 >=20 >> -----Original Message----- >> From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf = Of >> STARK, BARBARA H >> Sent: 08 November 2012 22:01 >> To: Eardley,PL,Philip,DUB8 R; mlinsner@cisco.com; lmap@ietf.org >> Subject: Re: [lmap] Scope, Use Cases and Users, and Which are = Mandatory >> vs. Nice to Have >>=20 >> To be painfully blunt -- >> Measurements for regulatory purposes are a cost. They generate no = revenue. >> We heard regulators telling us yesterday that they want to mandate >> operators provide them with performance measurements. >> I need to know what the minimum set of requirements is that an = operator >> would need to comply with, in order to meet the regulatory = requirements. >> The goal is to minimize the cost of the regulation. >>=20 >> If the goals of other parties can be met without adding cost to the >> regulatorily-mandated architecture, then that's wonderful, and I'm = happy to >> accommodate them. If the needs of others add cost to the mandated >> architecture, then that's very, very bad. To me, this means that all = other use >> cases are optional. >>=20 >> As I mentioned in my other email, we do extensive measurements today. = We >> do this to increase customer satisfaction (which decreases churn and = keeps >> customers loyal) and to decrease help desk and other operational = costs, by >> having effective test mechanisms and data that allows us to do = proactive >> network maintenance and management. But we don't need lmap to help us >> with this. If others do, ok, but it better not add to the cost of the = solution >> that the regulators are going to make operators do to meet regulatory >> mandates. >> Barbara >>=20 >>> -----Original Message----- >>> From: philip.eardley@bt.com [mailto:philip.eardley@bt.com] >>>=20 >>> similar to #1 >>> Operators want performance data for "mass market" broadband access >>> network connectivity services. >>>=20 >>> One difference between the use cases is the timescale of interest. = For >>> example regulators are probably more interested in monthly averages = over >>> large numbers of users whereas in #4 ISPs are probably more = interested in >>> finer grain information. >>>=20 >>> Barbara, >>> I'm interested in why you want to narrow the focus to #1 (is it that = your >>> sceptical about the importance of the other use cases, or that they = don't >>> need standardisation, or something else?) >>>=20 >>> thanks >>> phil >>>=20 >>> ________________________________________ >>> From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of = Marc >>> Linsner [mlinsner@cisco.com] >>> Sent: 08 November 2012 20:38 >>> To: STARK, BARBARA H; lmap@ietf.org >>> Subject: Re: [lmap] Scope, Use Cases and Users, and Which are = Mandatory >>> vs. Nice to Have >>>=20 >>> Barbara, >>>=20 >>> Thanks for documenting the use cases below. I would like to submit = use >> case >>> #5. >>>=20 >>> 5. OTT providers of services. OTT providers are interested in the = QOE of >> their >>> customers and may instanciate a measurement client/controller/server = of >>> their own accord. >>>=20 >>> -Marc- >>>=20 >>> -----Original Message----- >>> From: "STARK, BARBARA H" >>> Date: Thursday, November 8, 2012 1:41 PM >>> To: "lmap@ietf.org" >>> Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory = vs. >>> Nice to Have >>>=20 >>>> Following are my suggestions for problem scope, listing of use = cases >>>> that have been mentioned to date, and my thoughts on which we need = to >>>> focus on solving. >>>>=20 >>>> Proposed Scope: Performance measurements for mass market >> broadband >>>> connectivity services. >>>>=20 >>>> Use Cases >>>> 1. Regulators want performance data for "mass market" broadband >> access >>>> network connectivity services. Regulators are the primary user in = this >>>> use case. Regulators may have business requirements that involve = access >>>> by others to the data collected for this use case (e.g., = researchers >>>> are to be allowed access to data, or broadband customers should be >>>> allowed to access data for their own service), but these others are = not >>>> considered primary users in this case. >>>>=20 >>>> 2. Researchers want to be able to run and collect data from a = variety >>>> of tests performed against broadband connections (end-to-end, = segment >>>> by segment, maybe or maybe not limited to mass market or the access >>>> network). Researchers are the primary user. >>>>=20 >>>> 3. Broadband "mass market" customers want to run tests against = their >>>> broadband connections and see results of tests performed against = their >>>> broadband connection. Regulators, researchers, and customers are = the 3 >>>> sets of "users" that are covered by these 3 use cases. Broadband >>>> customers are the primary user. >>>>=20 >>>> 4. ISPs want tests to assist in troubleshooting, and network >>>> architecting and engineering. [Note that I'm just listing this as a = use >>>> case that has been mentioned, and not as something that is being >>>> requested by all ISPs >>>> -- no value judgment is implied by listing it here.] ISPs are the >>>> primary user. >>>>=20 >>>> Which Use Cases Are Mandatory >>>> Based on last night's discussion, I believe that only the first use >>>> case is mandatory. The others are fine to consider, if they don't = get >>>> in the way of solving use case #1. >>>>=20 >>>> If we can agree on these points, then I think the next step is to >>>> identify the Business Requirements of regulators. Ideally, we would = get >>>> input from regulators around the world. We certainly need to try to >>>> solicit their input. However, I think it's a good idea to start by >>>> documenting the business requirements of the one regulator who we = do >>>> have. Getting started may incite others to join in. Other "users" = (or >>>> proxies of other users) are also welcome to provide their business >>>> requirements -- but they need to understand that satisfying their >>>> requirements is considered "nice to have" and not mandatory. >>>>=20 >>>> I'm going to work next on putting together a start for Business >>>> Requirements. >>>> Barbara >>>> _______________________________________________ >>>> lmap mailing list >>>> lmap@ietf.org >>>> https://www.ietf.org/mailman/listinfo/lmap >>>=20 >>>=20 >>> _______________________________________________ >>> lmap mailing list >>> lmap@ietf.org >>> https://www.ietf.org/mailman/listinfo/lmap >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap From bs7652@att.com Fri Nov 9 08:08:00 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CA08D21F8759 for ; Fri, 9 Nov 2012 08:08:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.556 X-Spam-Level: X-Spam-Status: No, score=-106.556 tagged_above=-999 required=5 tests=[AWL=0.043, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JZoM-8qozVqv for ; Fri, 9 Nov 2012 08:07:59 -0800 (PST) Received: from nbfkord-smmo04.seg.att.com (nbfkord-smmo04.seg.att.com [209.65.160.86]) by ietfa.amsl.com (Postfix) with ESMTP id B609221F874C for ; Fri, 9 Nov 2012 08:07:59 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo04.seg.att.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id f5a2d905.2aaaf61af940.1989466.00-598.5496206.nbfkord-smmo04.seg.att.com (envelope-from ); Fri, 09 Nov 2012 16:07:59 +0000 (UTC) X-MXL-Hash: 509d2a5f55d3249b-05e549124766ecf801b90bad0041393711983130 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 15a2d905.0.1989368.00-498.5495893.nbfkord-smmo04.seg.att.com (envelope-from ); Fri, 09 Nov 2012 16:07:46 +0000 (UTC) X-MXL-Hash: 509d2a5269b8f044-71d5896cb2128e48fcaf28a891402a36cf0dc7cf Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA9G7iNk003171; Fri, 9 Nov 2012 11:07:45 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qA9G7Vo8002820 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 9 Nov 2012 11:07:34 -0500 Received: from GAALPA1MSGHUB9D.ITServices.sbc.com (gaalpa1msghub9d.itservices.sbc.com [130.8.36.90]) by sflint04.pst.cso.att.com (RSA Interceptor); Fri, 9 Nov 2012 11:07:09 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9D.ITServices.sbc.com ([130.8.36.90]) with mapi id 14.02.0318.001; Fri, 9 Nov 2012 11:06:42 -0500 From: "STARK, BARBARA H" To: Brian Trammell , "trevor.burbridge@bt.com" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQAOl1eAAADHYIAACT7MoAAFCFiQABGeZoAABA7fUA== Date: Fri, 9 Nov 2012 16:06:42 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.74.153] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=Na1RIR/4 c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=0tZWebmd2HoA:10 a=WUZhNrZLDssA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=8nJEP1OIZ-IA:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bbd-PDihl1AA:10 a=_ACSte9SGPBagGZzFysA:9 a=wPNLvf] X-AnalysisOut: [GTeEIA:10] Cc: "philip.eardley@bt.com" , "mlinsner@cisco.com" , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 16:08:00 -0000 > I get that the way some companies account for such things may make this > sound terribly na=EFve, but I think the approach I'd take here is if the > deployment of measurement infrastructure within the network will be > mandated in order to allow regulators to to verify that operators follow = laws > and regulations pertinent to delivered network performance, thereby > forcing a cost on network operators, then it makes sense to explore wheth= er > this same infrastructure can be used to other (potentially profitable) > purposes, thereby at least recovering some of the capital outlay therefor= . >=20 > Now, if we end up in a situation that it is _clear_ that the regulatory u= se cases > are far simpler than any useful troubleshooting use case, or any value-ad= ded > "measure my network" use case, such that a regulatory infrastructure woul= d > be measurably cheaper than one capable of multiple uses, then I would buy > the argument that these should be treated separately. >=20 > But that just doesn't seem plausible to me.=20 As I said, I think understanding the needs of others is good, and *if* the = needs of others can be accommodated without adding complexity, then we shou= ld do that. So as long as we allow for this to be a decision factor, and al= low that a provider who only wants to meet the requirements of the regulato= r must be *allowed* to only meet those requirements, and should not be *req= uired* to implement something that meets the needs of all others, in order = to satisfy a regulatory mandate. If we can agree on this, I suggest we proceed with identifying the business= requirements of the various use cases so we have something to actually ass= ess as to what it does or does not imply. After assessing business requirements and identifying the functions they im= ply -- if there is a consensus desire to define the mega-"satisfies everyon= e"-functional-architecture and create various needed piece-parts, I wouldn'= t stand in the way of this, as long as it is very clear which are the parts= that the regulator has an interest in mandating ("function Y must be accom= plished by doing X"), vs. being a regulatory "don't care". If the regulator= doesn't care how function Y is accomplished, and 20 different orgs define = 20 different ways of accomplishing Y, then a provider should be free to cho= ose which way to do Y. If the IETF wants to be one of those 20, and the IET= F's solution makes sense to a provider (e.g., because it allows for all use= cases in a single cohesive architecture and is brilliant in its design), t= hen the provider can certainly choose that as their approach. But it should= be understood that it would not be a regulatory mandate to select that app= roach. Barbara From j.schoenwaelder@jacobs-university.de Fri Nov 9 08:23:51 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 32CF021F85CD for ; Fri, 9 Nov 2012 08:23:48 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.166 X-Spam-Level: X-Spam-Status: No, score=-103.166 tagged_above=-999 required=5 tests=[AWL=0.083, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P7ByMwBCLeJp for ; Fri, 9 Nov 2012 08:23:47 -0800 (PST) Received: from hermes.jacobs-university.de (hermes.jacobs-university.de [212.201.44.23]) by ietfa.amsl.com (Postfix) with ESMTP id 2418721F85C3 for ; Fri, 9 Nov 2012 08:23:47 -0800 (PST) Received: from localhost (demetrius4.jacobs-university.de [212.201.44.49]) by hermes.jacobs-university.de (Postfix) with ESMTP id B0CF120BF8; Fri, 9 Nov 2012 17:23:43 +0100 (CET) X-Virus-Scanned: amavisd-new at jacobs-university.de Received: from hermes.jacobs-university.de ([212.201.44.23]) by localhost (demetrius4.jacobs-university.de [212.201.44.32]) (amavisd-new, port 10024) with ESMTP id FfTWA4ES_0QP; Fri, 9 Nov 2012 17:23:43 +0100 (CET) Received: from elstar.local (elstar.jacobs.jacobs-university.de [10.50.231.133]) by hermes.jacobs-university.de (Postfix) with ESMTP id 49D5020BEB; Fri, 9 Nov 2012 17:23:43 +0100 (CET) Received: by elstar.local (Postfix, from userid 501) id 5C64A22BA000; Fri, 9 Nov 2012 17:23:46 +0100 (CET) Date: Fri, 9 Nov 2012 17:23:46 +0100 From: Juergen Schoenwaelder To: "STARK, BARBARA H" Message-ID: <20121109162346.GA82261@elstar.local> Mail-Followup-To: "STARK, BARBARA H" , "lmap@ietf.org" References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com> <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: Juergen Schoenwaelder List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2012 16:23:51 -0000 On Fri, Nov 09, 2012 at 04:06:42PM +0000, STARK, BARBARA H wrote: > After assessing business requirements and identifying the functions > they imply -- if there is a consensus desire to define the > mega-"satisfies everyone"-functional-architecture and create various > needed piece-parts, I wouldn't stand in the way of this, as long as > it is very clear which are the parts that the regulator has an > interest in mandating ("function Y must be accomplished by doing > X"), vs. being a regulatory "don't care". If the regulator doesn't > care how function Y is accomplished, and 20 different orgs define 20 > different ways of accomplishing Y, then a provider should be free to > choose which way to do Y. If the IETF wants to be one of those 20, > and the IETF's solution makes sense to a provider (e.g., because it > allows for all use cases in a single cohesive architecture and is > brilliant in its design), then the provider can certainly choose > that as their approach. But it should be understood that it would > not be a regulatory mandate to select that approach. Barbara, I think we heard your concerns. That said, lets also take into account that we talk likely about multiple regulators if we take a more global perspective and the regulators might agree on a single set of requirements globally (not too likely if you ask me but possible) or slightly different overlapping sets (perhaps more likely). This is perhaps clear but I thought I spell this out since people always write about _the_ regulator (as if it is only the FCC). /js -- Juergen Schoenwaelder Jacobs University Bremen gGmbH Phone: +49 421 200 3587 Campus Ring 1, 28759 Bremen, Germany Fax: +49 421 200 3103 From Michael.K.Bugenhagen@centurylink.com Mon Nov 12 07:49:14 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6080A21F859D for ; Mon, 12 Nov 2012 07:49:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.998 X-Spam-Level: X-Spam-Status: No, score=-0.998 tagged_above=-999 required=5 tests=[AWL=-1.599, BAYES_50=0.001, J_CHICKENPOX_72=0.6] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l12bkWOzJZ2v for ; Mon, 12 Nov 2012 07:49:13 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id 11A3721F85C4 for ; Mon, 12 Nov 2012 07:49:12 -0800 (PST) Received: from lxomavmpc030.qintra.com (lxomavmpc030.qintra.com [151.117.207.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id qACFn5nY018441 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 Nov 2012 08:49:06 -0700 (MST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 7E0881E0069; Mon, 12 Nov 2012 09:49:00 -0600 (CST) Received: from suomp61i.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id 5F7D91E005D; Mon, 12 Nov 2012 09:49:00 -0600 (CST) Received: from suomp61i.qintra.com (localhost [127.0.0.1]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id qACFmxB1025863; Mon, 12 Nov 2012 09:48:59 -0600 (CST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.qintra.com [151.117.206.28]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id qACFmxxw025850 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 12 Nov 2012 09:48:59 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Mon, 12 Nov 2012 09:48:59 -0600 From: "Bugenhagen, Michael K" To: "'Brian Trammell'" , "trevor.burbridge@bt.com" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: AQHNvnPUyR6x4BrnVU2cSUkEqtNg05fmW9ZA Date: Mon, 12 Nov 2012 15:48:59 +0000 Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "mlinsner@cisco.com" , "philip.eardley@bt.com" , "bs7652@att.com" , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2012 15:49:14 -0000 Just a FYI - The Broadband forum scope, and Project requirements are actually in the = liaison they sent over in March this year when that project started. Just as a reminder I'll ask them liaison the latest draft they have to the = group as well. These would need to be identified as the scope as an earlier poster noted. https://datatracker.ietf.org/liaison/1179/ - Mike -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Bri= an Trammell Sent: Friday, November 09, 2012 6:15 AM To: trevor.burbridge@bt.com Cc: philip.eardley@bt.com; lmap@ietf.org; bs7652@att.com; mlinsner@cisco.co= m Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs.= Nice to Have hi Trevor, +1 I get that the way some companies account for such things may make this sou= nd terribly na=EFve, but I think the approach I'd take here is if the deplo= yment of measurement infrastructure within the network will be mandated in = order to allow regulators to to verify that operators follow laws and regul= ations pertinent to delivered network performance, thereby forcing a cost o= n network operators, then it makes sense to explore whether this same infra= structure can be used to other (potentially profitable) purposes, thereby a= t least recovering some of the capital outlay therefor. Now, if we end up in a situation that it is _clear_ that the regulatory use= cases are far simpler than any useful troubleshooting use case, or any val= ue-added "measure my network" use case, such that a regulatory infrastructu= re would be measurably cheaper than one capable of multiple uses, then I wo= uld buy the argument that these should be treated separately. But that just doesn't seem plausible to me. Unless the regulatory use case = is scoped to be severely restricted (say, one trivially computable metric w= ith a tiny population sample, which would IMO be questionably useful), the = basic metrics (measurement point capabilities), analyses (collector and pos= t-collector capabilities), data volumes and deployment patterns (system sca= le) seem to me to be pretty much the same. (The only thing from the LMAP dr= aft architecture I don't see as universally applicable to all of these is t= he network parameters server, but it's clear from earlier list discussion t= hat a lot of people seem to be confused about what that's for in the first = place.) Indeed, I'd look at this whole argument from the other side: is it possible= to define a useful reference architecture and protocol for measurement inf= rastructures which have as their _primary_ use the within-the-ISP and from-= the-customer-POV troubleshooting use cases, which is _also_ useful for regu= latory verification purposes, which is incrementally implementable, such th= at when the mandate does come down, the regulatory requirements can be met = by infrastructure already being deployed for operations? Best regards, Brian On 9 Nov 2012, at 4:02, trevor.burbridge@bt.com wrote: > To give a different perspective, I think that #4 (including both fine per= line measurement and overall sampling of products for aggregate network pe= rformance and benchmarking against other ISP/products) is a critical use ca= se. >=20 > Some ISPs will benefit, others may not. I's not that all regulators in us= e case #1 will want to (or be able to) mandate that this measurement has to= take place either. >=20 > While there are many currently deployed measurement techniques - deployin= g and managing tests, for example to home CPE, is not standardised. A harmo= nised way to run end-to-end tests that can be better indicative of user QoE= is lacking. A way of detecting performance problems at any point and at an= y layer in diverse networks is lacking. >=20 > Furthermore if the ISP manages or owns much of this equipment, then it is= vital that they are able to get benefit from running tests on these device= s. In my view if the ISP is not the primary beneficiary (along with ultimat= ely the broadband user) of this activity then these tests will never get de= ployed on a large scale but will always be restricted to limited and costly= panels. >=20 > Trevor. >=20 >=20 >> -----Original Message----- >> From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf=20 >> Of STARK, BARBARA H >> Sent: 08 November 2012 22:01 >> To: Eardley,PL,Philip,DUB8 R; mlinsner@cisco.com; lmap@ietf.org >> Subject: Re: [lmap] Scope, Use Cases and Users, and Which are=20 >> Mandatory vs. Nice to Have >>=20 >> To be painfully blunt -- >> Measurements for regulatory purposes are a cost. They generate no revenu= e. >> We heard regulators telling us yesterday that they want to mandate=20 >> operators provide them with performance measurements. >> I need to know what the minimum set of requirements is that an=20 >> operator would need to comply with, in order to meet the regulatory requ= irements. >> The goal is to minimize the cost of the regulation. >>=20 >> If the goals of other parties can be met without adding cost to the=20 >> regulatorily-mandated architecture, then that's wonderful, and I'm=20 >> happy to accommodate them. If the needs of others add cost to the=20 >> mandated architecture, then that's very, very bad. To me, this means=20 >> that all other use cases are optional. >>=20 >> As I mentioned in my other email, we do extensive measurements today.=20 >> We do this to increase customer satisfaction (which decreases churn=20 >> and keeps customers loyal) and to decrease help desk and other=20 >> operational costs, by having effective test mechanisms and data that=20 >> allows us to do proactive network maintenance and management. But we=20 >> don't need lmap to help us with this. If others do, ok, but it better=20 >> not add to the cost of the solution that the regulators are going to=20 >> make operators do to meet regulatory mandates. >> Barbara >>=20 >>> -----Original Message----- >>> From: philip.eardley@bt.com [mailto:philip.eardley@bt.com] >>>=20 >>> similar to #1 >>> Operators want performance data for "mass market" broadband access=20 >>> network connectivity services. >>>=20 >>> One difference between the use cases is the timescale of interest.=20 >>> For example regulators are probably more interested in monthly=20 >>> averages over large numbers of users whereas in #4 ISPs are probably=20 >>> more interested in finer grain information. >>>=20 >>> Barbara, >>> I'm interested in why you want to narrow the focus to #1 (is it that=20 >>> your sceptical about the importance of the other use cases, or that=20 >>> they don't need standardisation, or something else?) >>>=20 >>> thanks >>> phil >>>=20 >>> ________________________________________ >>> From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] On Behalf Of=20 >>> Marc Linsner [mlinsner@cisco.com] >>> Sent: 08 November 2012 20:38 >>> To: STARK, BARBARA H; lmap@ietf.org >>> Subject: Re: [lmap] Scope, Use Cases and Users, and Which are=20 >>> Mandatory vs. Nice to Have >>>=20 >>> Barbara, >>>=20 >>> Thanks for documenting the use cases below. I would like to submit=20 >>> use >> case >>> #5. >>>=20 >>> 5. OTT providers of services. OTT providers are interested in the=20 >>> QOE of >> their >>> customers and may instanciate a measurement client/controller/server=20 >>> of their own accord. >>>=20 >>> -Marc- >>>=20 >>> -----Original Message----- >>> From: "STARK, BARBARA H" >>> Date: Thursday, November 8, 2012 1:41 PM >>> To: "lmap@ietf.org" >>> Subject: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. >>> Nice to Have >>>=20 >>>> Following are my suggestions for problem scope, listing of use=20 >>>> cases that have been mentioned to date, and my thoughts on which we=20 >>>> need to focus on solving. >>>>=20 >>>> Proposed Scope: Performance measurements for mass market >> broadband >>>> connectivity services. >>>>=20 >>>> Use Cases >>>> 1. Regulators want performance data for "mass market" broadband >> access >>>> network connectivity services. Regulators are the primary user in=20 >>>> this use case. Regulators may have business requirements that=20 >>>> involve access by others to the data collected for this use case=20 >>>> (e.g., researchers are to be allowed access to data, or broadband=20 >>>> customers should be allowed to access data for their own service),=20 >>>> but these others are not considered primary users in this case. >>>>=20 >>>> 2. Researchers want to be able to run and collect data from a=20 >>>> variety of tests performed against broadband connections=20 >>>> (end-to-end, segment by segment, maybe or maybe not limited to mass=20 >>>> market or the access network). Researchers are the primary user. >>>>=20 >>>> 3. Broadband "mass market" customers want to run tests against=20 >>>> their broadband connections and see results of tests performed=20 >>>> against their broadband connection. Regulators, researchers, and=20 >>>> customers are the 3 sets of "users" that are covered by these 3 use=20 >>>> cases. Broadband customers are the primary user. >>>>=20 >>>> 4. ISPs want tests to assist in troubleshooting, and network=20 >>>> architecting and engineering. [Note that I'm just listing this as a=20 >>>> use case that has been mentioned, and not as something that is=20 >>>> being requested by all ISPs >>>> -- no value judgment is implied by listing it here.] ISPs are the=20 >>>> primary user. >>>>=20 >>>> Which Use Cases Are Mandatory >>>> Based on last night's discussion, I believe that only the first use=20 >>>> case is mandatory. The others are fine to consider, if they don't=20 >>>> get in the way of solving use case #1. >>>>=20 >>>> If we can agree on these points, then I think the next step is to=20 >>>> identify the Business Requirements of regulators. Ideally, we would=20 >>>> get input from regulators around the world. We certainly need to=20 >>>> try to solicit their input. However, I think it's a good idea to=20 >>>> start by documenting the business requirements of the one regulator=20 >>>> who we do have. Getting started may incite others to join in. Other=20 >>>> "users" (or proxies of other users) are also welcome to provide=20 >>>> their business requirements -- but they need to understand that=20 >>>> satisfying their requirements is considered "nice to have" and not man= datory. >>>>=20 >>>> I'm going to work next on putting together a start for Business=20 >>>> Requirements. >>>> Barbara >>>> _______________________________________________ >>>> lmap mailing list >>>> lmap@ietf.org >>>> https://www.ietf.org/mailman/listinfo/lmap >>>=20 >>>=20 >>> _______________________________________________ >>> lmap mailing list >>> lmap@ietf.org >>> https://www.ietf.org/mailman/listinfo/lmap >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From rolf.winter@hs-augsburg.de Mon Nov 12 08:21:06 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0223521F85F0 for ; Mon, 12 Nov 2012 08:21:06 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.312 X-Spam-Level: X-Spam-Status: No, score=-0.312 tagged_above=-999 required=5 tests=[BAYES_05=-1.11, HELO_EQ_DE=0.35, HELO_MISMATCH_DE=1.448, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 81AVRCYmq8KM for ; Mon, 12 Nov 2012 08:21:05 -0800 (PST) Received: from bee.RZ.HS-Augsburg.DE (bee.RZ.HS-Augsburg.DE [141.82.25.26]) by ietfa.amsl.com (Postfix) with ESMTP id 2C82E21F8529 for ; Mon, 12 Nov 2012 08:21:05 -0800 (PST) Received: from Rolfs-MacBook-Pro.local ([141.82.51.130]) (authenticated bits=0) by bee.RZ.HS-Augsburg.DE (8.14.3/8.14.3) with ESMTP id qACGKwe3013501 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 12 Nov 2012 17:20:59 +0100 (MET) Message-ID: <50A121EA.7080303@hs-augsburg.de> Date: Mon, 12 Nov 2012 17:20:58 +0100 From: Rolf Winter User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: lmap@ietf.org References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (bee.RZ.HS-Augsburg.DE [141.82.16.16]); Mon, 12 Nov 2012 17:20:59 +0100 (MET) X-HSA-RZ-MailScanner-Information: Please contact the ISP for more information X-HSA-RZ-MailScanner-ID: qACGKwe3013501 X-HSA-RZ-MailScanner: Found to be clean X-HSA-RZ-MailScanner-From: rolf.winter@hs-augsburg.de X-HSA-RZ-MailScanner-Watermark: 1353342060.62823@z3XtfT2a/VS2xBezuinMrQ Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2012 16:21:06 -0000 Hi, I really wonder what exactly you mean with business requirements here and what the IETF can or should do about those. Could you spell some of these requirements out? Best, Rolf On 11/9/12 5:06 PM, STARK, BARBARA H wrote: >> I get that the way some companies account for such things may make this >> sound terribly naïve, but I think the approach I'd take here is if the >> deployment of measurement infrastructure within the network will be >> mandated in order to allow regulators to to verify that operators follow laws >> and regulations pertinent to delivered network performance, thereby >> forcing a cost on network operators, then it makes sense to explore whether >> this same infrastructure can be used to other (potentially profitable) >> purposes, thereby at least recovering some of the capital outlay therefor. >> >> Now, if we end up in a situation that it is _clear_ that the regulatory use cases >> are far simpler than any useful troubleshooting use case, or any value-added >> "measure my network" use case, such that a regulatory infrastructure would >> be measurably cheaper than one capable of multiple uses, then I would buy >> the argument that these should be treated separately. >> >> But that just doesn't seem plausible to me. > > As I said, I think understanding the needs of others is good, and *if* the needs of others can be accommodated without adding complexity, then we should do that. So as long as we allow for this to be a decision factor, and allow that a provider who only wants to meet the requirements of the regulator must be *allowed* to only meet those requirements, and should not be *required* to implement something that meets the needs of all others, in order to satisfy a regulatory mandate. > > If we can agree on this, I suggest we proceed with identifying the business requirements of the various use cases so we have something to actually assess as to what it does or does not imply. > > After assessing business requirements and identifying the functions they imply -- if there is a consensus desire to define the mega-"satisfies everyone"-functional-architecture and create various needed piece-parts, I wouldn't stand in the way of this, as long as it is very clear which are the parts that the regulator has an interest in mandating ("function Y must be accomplished by doing X"), vs. being a regulatory "don't care". If the regulator doesn't care how function Y is accomplished, and 20 different orgs define 20 different ways of accomplishing Y, then a provider should be free to choose which way to do Y. If the IETF wants to be one of those 20, and the IETF's solution makes sense to a provider (e.g., because it allows for all use cases in a single cohesive architecture and is brilliant in its design), then the provider can certainly choose that as their approach. But it should be understood that it would not be a regulatory mandate to select that approach. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > From bs7652@att.com Mon Nov 12 10:33:24 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 07B2D21F874A for ; Mon, 12 Nov 2012 10:33:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.562 X-Spam-Level: X-Spam-Status: No, score=-106.562 tagged_above=-999 required=5 tests=[AWL=0.038, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gtleP7FE+9Xx for ; Mon, 12 Nov 2012 10:33:17 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id 40D7521F8780 for ; Mon, 12 Nov 2012 10:33:16 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo07.seg.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id de041a05.2aaafd254940.361807.00-560.987118.nbfkord-smmo07.seg.att.com (envelope-from ); Mon, 12 Nov 2012 18:33:17 +0000 (UTC) X-MXL-Hash: 50a140ed6c01403d-1d8bc278973c616d5ea535772f49ee18b1845d0f Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 1e041a05.0.361713.00-446.986830.nbfkord-smmo07.seg.att.com (envelope-from ); Mon, 12 Nov 2012 18:33:08 +0000 (UTC) X-MXL-Hash: 50a140e42b057380-037710e03cd8558dcf36e07b77a3f3a82f3bb3ad Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qACIX5Tk014135; Mon, 12 Nov 2012 13:33:05 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qACIWlb9013350 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 12 Nov 2012 13:32:57 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint03.pst.cso.att.com (RSA Interceptor); Mon, 12 Nov 2012 13:32:13 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Mon, 12 Nov 2012 13:32:13 -0500 From: "STARK, BARBARA H" To: Rolf Winter , "lmap@ietf.org" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: Ac294J6lTJJLQ1x3S5C5rqnfFr7PzQAOl1eAAADHYIAACT7MoAAFCFiQABGeZoAABA7fUACbbBMAAAi5WmA= Date: Mon, 12 Nov 2012 18:32:12 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5CC293@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> <50A121EA.7080303@hs-augsburg.de> In-Reply-To: <50A121EA.7080303@hs-augsburg.de> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.23.105] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=Q/plFfKa c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=fQwc9AsTUDYA:10 a=WUZhNrZLDssA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bbd-PDihl1AA:10 a=0QeGv_Ny73cuUC7u1iAA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2012 18:33:24 -0000 > I really wonder what exactly you mean with business requirements here and > what the IETF can or should do about those. Could you spell some of these > requirements out? Yes. I've been trying to write down what I think I heard as being the basic= "business" requirements coming from Henning and James at the IETF. I haven't managed to write it up into something formal, but here are some h= igh-level statements: Regulators must be able to=20 1) specify what tests get run; the set of tests needs to cover all access = technologies and allow "apples to apples" comparison of different access ne= twork providers, the technologies they use, and among results of different = customers; the set of tests is expected to change over time (but hopefully = not too quickly, since some tests may require considerable lead time to ens= ure that test infrastructure is in place, and to understand any performance= /security risks that the tests my incur); note that different regulators ma= y choose to require different sets of tests. 2) ensure that there is sufficient sampling of customers (of access network= providers) to ensure coverage of all access network providers, geographies= , and access technologies, where the sample size is large enough to ensure = anonymity of results and limit the effect of statistical variation 3) be confident that data needed to provide meaningful test results on a pe= r-customer level will exist, be collected, and collated=20 4) have per-customer anonymous test results be available to the regulator a= nd researchers The first item implies to me that there is a need for a set of recommended = tests. The set of tests will drive requirements not only for endpoints with= in the customer premises network, but also for test endpoints in the access= network or beyond. The second item, to me, looks less like an architectural element and more l= ike a recommendation for what constitutes an appropriate sample size.=20 The third implies to me that access network providers will need to be able = to ensure that this is the case. The fourth implies to me that access network providers will need to make su= re that they have some way of making results available, and that the result= s are per-customer but anonymous. So when I look at items 1, 3 and 4, I see that these cause access network p= roviders to have requirements. Not all access network providers are alike. Some access network providers may want to have a 100% proprietary test cont= rol and management and internal data collection architecture. Some access network providers may want to re-use their existing TR-069/TR-1= 43/IPDR/OMA-DM/SNMP/etc. architectures. Some access network providers may want to put a new box (like SamKnows) or = an app (on a PC/smart phone/USB stick/etc.) in the customer premises networ= k, and have the option of doing this themselves or contracting it out to a = 3rd party. Some access network providers may want a new architecture that perhaps re-u= ses some existing architectural elements, but maybe adds a new "control pla= ne" protocol. Some may want to use a proprietary control plane protocol, an= d some may want to get together with other access network providers and sta= ndardize such a protocol. If I have accurately represented the James/Henning "business requirements" = that I heard, then I see nothing in them that would prohibit any of the acc= ess network provider options that I've listed. Which would make me happy. O= n the other hand, if a regulator is coming forward and saying "I have a req= uirement that the architecture for providing metrics must use a specific co= ntrol plane protocol", then I'm concerned; because it would prohibit all ar= chitectures that do not use it. If an access network provider is coming for= ward and saying "I want a new control plane protocol to exist because I wan= t to meet the regulatory requirements with an architecture that makes use o= f it", then I'm happy to support them (well, if they can get other provider= s to also want it -- I dislike creating "standards" that are only used by o= ne provider). I'm also happy if the regulators express a desire to help the= providers achieve this standardized control plane protocol. I just get wo= rried when the regulators state that a new standardized control plane proto= col *must* exist to meet the regulatory use case. Anyway, from "business requirement" #1, I see the need for a set of tests t= o be identified. It's clear that some PHY/access technologies and other spe= cial interest groups and standards orgs are interested in providing input t= o this set. It's also clear that the IETF has expertise in Layer 3 and abov= e metrics. =20 I've heard some access network providers say that they want a new control p= lane protocol for the architecture they want to use to meet #3 and #4. The = access providers who want such a protocol should be free to have this proto= col worked in whatever organization they wish. As it relates to other use cases (customer wants to run tests, application = service provider wants to run tests, service provider wants test environmen= t to meet a variety of non-regulatory needs), I need to leave it up to the = proponents of those use cases to provide the business requirements. Barbara From shane@castlepoint.net Mon Nov 12 20:24:30 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0A39621F887E for ; Mon, 12 Nov 2012 20:24:30 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.357 X-Spam-Level: X-Spam-Status: No, score=-0.357 tagged_above=-999 required=5 tests=[AWL=0.080, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h5sGbB7dkw7P for ; Mon, 12 Nov 2012 20:24:29 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 0228621F8870 for ; Mon, 12 Nov 2012 20:24:29 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 91FEE20E4 for ; Tue, 13 Nov 2012 04:24:27 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id EF20F20C7; Mon, 12 Nov 2012 21:24:25 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CC293@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Mon, 12 Nov 2012 21:24:24 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> <50A121EA.7080303@hs-augsburg.de> <2D09D61DDFA73D4C884805CC7865E6112F5CC293@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Mon Nov 12 21:24:27 2012 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 50a1cb7b199639717911945 X-DSPAM-Factors: 27, happy+#+the, 0.40000, happy+#+the, 0.40000, want+it, 0.40000, premature+#+#+#+there, 0.40000, satisfy+#+#+partially, 0.40000, concerns+in, 0.40000, then+time, 0.40000, you+#+#+#+just, 0.40000, specify+#+#+IETF, 0.40000, should+#+#+to, 0.40000, those+#+you, 0.40000, proposal+is, 0.40000, tests+#+expected, 0.40000, network+#+#+#+a, 0.40000, network+#+#+#+a, 0.40000, 00+#+only, 0.40000, all+#+#+#+into, 0.40000, performance+#+#+#+the, 0.40000, and+#+The, 0.40000, and+#+The, 0.40000, dislike+#+#+#+are, 0.40000, ensure+coverage, 0.40000, there+#+#+you, 0.40000, requirements+#+an, 0.40000, an+#+#+#+is, 0.40000, customers+#+#+network, 0.40000, for+#+set, 0.40000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2012 04:24:30 -0000 Barbara, On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" wrote: >> I really wonder what exactly you mean with business requirements here = and >> what the IETF can or should do about those. Could you spell some of = these >> requirements out? >=20 > Yes. I've been trying to write down what I think I heard as being the = basic "business" requirements coming from Henning and James at the IETF. > I haven't managed to write it up into something formal, but here are = some high-level statements: >=20 > Regulators must be able to=20 > 1) specify what tests get run; the set of tests needs to cover all = access technologies and allow "apples to apples" comparison of different = access network providers, the technologies they use, and among results = of different customers; the set of tests is expected to change over time = (but hopefully not too quickly, since some tests may require = considerable lead time to ensure that test infrastructure is in place, = and to understand any performance/security risks that the tests my = incur); note that different regulators may choose to require different = sets of tests. > 2) ensure that there is sufficient sampling of customers (of access = network providers) to ensure coverage of all access network providers, = geographies, and access technologies, where the sample size is large = enough to ensure anonymity of results and limit the effect of = statistical variation > 3) be confident that data needed to provide meaningful test results on = a per-customer level will exist, be collected, and collated=20 > 4) have per-customer anonymous test results be available to the = regulator and researchers >=20 > The first item implies to me that there is a need for a set of = recommended tests. The set of tests will drive requirements not only for = endpoints within the customer premises network, but also for test = endpoints in the access network or beyond. > The second item, to me, looks less like an architectural element and = more like a recommendation for what constitutes an appropriate sample = size.=20 > The third implies to me that access network providers will need to be = able to ensure that this is the case. > The fourth implies to me that access network providers will need to = make sure that they have some way of making results available, and that = the results are per-customer but anonymous. >=20 > So when I look at items 1, 3 and 4, I see that these cause access = network providers to have requirements. > Not all access network providers are alike. > Some access network providers may want to have a 100% proprietary test = control and management and internal data collection architecture. > Some access network providers may want to re-use their existing = TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures. > Some access network providers may want to put a new box (like = SamKnows) or an app (on a PC/smart phone/USB stick/etc.) in the customer = premises network, and have the option of doing this themselves or = contracting it out to a 3rd party. > Some access network providers may want a new architecture that perhaps = re-uses some existing architectural elements, but maybe adds a new = "control plane" protocol. Some may want to use a proprietary control = plane protocol, and some may want to get together with other access = network providers and standardize such a protocol. First, I think the correct term to use, above, is: "management protocol" = or OAM protocol, not "control plane protocol". Examples of the former = in the IETF are: SNMP, NETCONF. In the IETF, I think if/when people see = "control plane" protocol, they will instantly assume you mean: BGP, = OSPF, LDP, etc. Sorry to be the "word police" :-). Second, I think it's important to clarify your comments in light of = "straw man" architectural proposal in = draft-schulzrinne-lmap-requirements-00. I only say this because, IMO, I = think that provides a decent starting point to ensure we share the same = terminology and points-of-reference in this dialogue. Specifically, it = sounds like you're solely (?) focused on the "measurement client" in the = above, correct? > If I have accurately represented the James/Henning "business = requirements" that I heard, then I see nothing in them that would = prohibit any of the access network provider options that I've listed. = Which would make me happy. On the other hand, if a regulator is coming = forward and saying "I have a requirement that the architecture for = providing metrics must use a specific control plane protocol", then I'm = concerned; because it would prohibit all architectures that do not use = it. If an access network provider is coming forward and saying "I want a = new control plane protocol to exist because I want to meet the = regulatory requirements with an architecture that makes use of it", then = I'm happy to support them (well, if they can get other providers to also = want it -- I dislike creating "standards" that are only used by one = provider). I'm also happy if the regulators express a desire to help the = providers achieve this standardized control plane protocol. I just get = worried when the=20 > regulators state that a new standardized control plane protocol *must* = exist to meet the regulatory use case. I would point out a few things: 1) I don't think anyone has stated the "must" word, even lower-case :-) = -- but, I would welcome being corrected if you believe that to be the = case. Furthermore, regulators can't state things in the IETF or in any = IETF documents, since the IETF is a global standards organization and = regulations differ (widely) all across the world. 2) Second, I believe the main thrust of the present straw man proposal = is to point out the need for a _unified_ methodology in which to conduct = network performance measurements on a much broader scale than exists, at = the moment. Personally, I read the straw man more as a "call to action" = than any specific, detailed proposal of "a solution", but maybe that's = me. 3) Finally, without an objective set of requirements it will be = difficult to impossible to objectively evaluate existing protocols, = which you mention above, to see if all of them (or only some of them or = none of them) will wholly satisfy or only partially satisfy the needs = outlined in draft-schulzrinne. IMO, I think it's premature to assume = that there can only exist a single OAM protocol, at this point. = Obviously, from an Engineering PoV, I think many would agree that there = are benefits to a single OAM protocol (development is focused on one = thing); however, there are also downsides, as well. Namely, that it = *will* take quite a bit of time to specify in the IETF, procure and = longer still to get adopted. Thus, if it's /feasible/ to fit most/all = the requirements incrementally into an existing protocol(s), then time = to develop, procure and deploy is decreased (considerably). Regardless, = I think those are questions to be answered when evaluating the solution = space against a widely agreed upon set of objective requirements, but = we're not there yet. In light of the above, it would be helpful to characterize your concerns = in light of draft-schulzrinne-lmap-requirements-00, since that's the = only proposal on the table at the moment. If there is text you can cite = from that draft that concerns you, it would be helpful to get it = clarified, ask to get it *out* of that draft or whatever. :-) If you = really want, focus just on Section 6, "Requirements", which is a mere 3 = pages long, at this point. :-) -shane > Anyway, from "business requirement" #1, I see the need for a set of = tests to be identified. It's clear that some PHY/access technologies and = other special interest groups and standards orgs are interested in = providing input to this set. It's also clear that the IETF has expertise = in Layer 3 and above metrics. =20 >=20 > I've heard some access network providers say that they want a new = control plane protocol for the architecture they want to use to meet #3 = and #4. The access providers who want such a protocol should be free to = have this protocol worked in whatever organization they wish. >=20 > As it relates to other use cases (customer wants to run tests, = application service provider wants to run tests, service provider wants = test environment to meet a variety of non-regulatory needs), I need to = leave it up to the proponents of those use cases to provide the business = requirements. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap >=20 From Michael.K.Bugenhagen@centurylink.com Tue Nov 13 08:05:18 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CD04D21F8845 for ; Tue, 13 Nov 2012 08:05:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.799 X-Spam-Level: X-Spam-Status: No, score=-1.799 tagged_above=-999 required=5 tests=[AWL=0.800, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ma7iiT86L5vu for ; Tue, 13 Nov 2012 08:05:17 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id DA1D421F8455 for ; Tue, 13 Nov 2012 08:05:15 -0800 (PST) Received: from lxomavmpc030.qintra.com (lxomavmpc030.qintra.com [151.117.207.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id qADG4pH1025186 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Nov 2012 09:04:51 -0700 (MST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id DCD8F1E006F; Tue, 13 Nov 2012 10:04:45 -0600 (CST) Received: from suomp61i.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id BEB0C1E005A; Tue, 13 Nov 2012 10:04:45 -0600 (CST) Received: from suomp61i.qintra.com (localhost [127.0.0.1]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id qADG4jVP001813; Tue, 13 Nov 2012 10:04:45 -0600 (CST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.qintra.com [151.117.206.28]) by suomp61i.qintra.com (8.14.4/8.14.4) with ESMTP id qADG4jxw001806 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 13 Nov 2012 10:04:45 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Tue, 13 Nov 2012 10:04:44 -0600 From: "Bugenhagen, Michael K" To: "'Shane Amante'" , "STARK, BARBARA H" Thread-Topic: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have Thread-Index: AQHNwVbJJ4bqvu6efEW5f/2QOjlLSpfn4ocg Date: Tue, 13 Nov 2012 16:04:44 +0000 Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> <50A121EA.7080303@hs-augsburg.de> <2D09D61DDFA73D4C884805CC7865E6112F5CC293@GAALPA1MSGUSR9L.ITServices.sbc.com> <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> In-Reply-To: <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2012 16:05:19 -0000 Shane -=20 The only requirement I am fairly clear on is that some of the ISP's *(US= based at least) are required to produce performance results per speed tie= r for Customers. This was verbally mentioned in the BAR BOF, but I didn't= look for it in the draft. =20 So from a written requirements priority view I would think defining the w= hat those PM's are, and how to perform the PM's, along with how to report t= he PM (Validating / cleaning the data) are key. Regards, Mike =20 -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Sha= ne Amante Sent: Monday, November 12, 2012 10:24 PM To: STARK, BARBARA H Cc: Rolf Winter; lmap@ietf.org Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs.= Nice to Have Barbara, On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" wrote: >> I really wonder what exactly you mean with business requirements here=20 >> and what the IETF can or should do about those. Could you spell some=20 >> of these requirements out? >=20 > Yes. I've been trying to write down what I think I heard as being the bas= ic "business" requirements coming from Henning and James at the IETF. > I haven't managed to write it up into something formal, but here are some= high-level statements: >=20 > Regulators must be able to > 1) specify what tests get run; the set of tests needs to cover all acces= s technologies and allow "apples to apples" comparison of different access = network providers, the technologies they use, and among results of differen= t customers; the set of tests is expected to change over time (but hopefull= y not too quickly, since some tests may require considerable lead time to e= nsure that test infrastructure is in place, and to understand any performan= ce/security risks that the tests my incur); note that different regulators = may choose to require different sets of tests. > 2) ensure that there is sufficient sampling of customers (of access=20 > network providers) to ensure coverage of all access network providers,=20 > geographies, and access technologies, where the sample size is large=20 > enough to ensure anonymity of results and limit the effect of=20 > statistical variation > 3) be confident that data needed to provide meaningful test results on=20 > a per-customer level will exist, be collected, and collated > 4) have per-customer anonymous test results be available to the=20 > regulator and researchers >=20 > The first item implies to me that there is a need for a set of recommende= d tests. The set of tests will drive requirements not only for endpoints wi= thin the customer premises network, but also for test endpoints in the acce= ss network or beyond. > The second item, to me, looks less like an architectural element and more= like a recommendation for what constitutes an appropriate sample size.=20 > The third implies to me that access network providers will need to be abl= e to ensure that this is the case. > The fourth implies to me that access network providers will need to make = sure that they have some way of making results available, and that the resu= lts are per-customer but anonymous. >=20 > So when I look at items 1, 3 and 4, I see that these cause access network= providers to have requirements. > Not all access network providers are alike. > Some access network providers may want to have a 100% proprietary test co= ntrol and management and internal data collection architecture. > Some access network providers may want to re-use their existing TR-069/TR= -143/IPDR/OMA-DM/SNMP/etc. architectures. > Some access network providers may want to put a new box (like SamKnows) o= r an app (on a PC/smart phone/USB stick/etc.) in the customer premises netw= ork, and have the option of doing this themselves or contracting it out to = a 3rd party. > Some access network providers may want a new architecture that perhaps re= -uses some existing architectural elements, but maybe adds a new "control p= lane" protocol. Some may want to use a proprietary control plane protocol, = and some may want to get together with other access network providers and s= tandardize such a protocol. First, I think the correct term to use, above, is: "management protocol" or= OAM protocol, not "control plane protocol". Examples of the former in the= IETF are: SNMP, NETCONF. In the IETF, I think if/when people see "control= plane" protocol, they will instantly assume you mean: BGP, OSPF, LDP, etc.= Sorry to be the "word police" :-). Second, I think it's important to clarify your comments in light of "straw = man" architectural proposal in draft-schulzrinne-lmap-requirements-00. I o= nly say this because, IMO, I think that provides a decent starting point to= ensure we share the same terminology and points-of-reference in this dialo= gue. Specifically, it sounds like you're solely (?) focused on the "measur= ement client" in the above, correct? > If I have accurately represented the James/Henning "business=20 > requirements" that I heard, then I see nothing in them that would=20 > prohibit any of the access network provider options that I've listed.=20 > Which would make me happy. On the other hand, if a regulator is coming=20 > forward and saying "I have a requirement that the architecture for=20 > providing metrics must use a specific control plane protocol", then=20 > I'm concerned; because it would prohibit all architectures that do not=20 > use it. If an access network provider is coming forward and saying "I=20 > want a new control plane protocol to exist because I want to meet the=20 > regulatory requirements with an architecture that makes use of it",=20 > then I'm happy to support them (well, if they can get other providers=20 > to also want it -- I dislike creating "standards" that are only used=20 > by one provider). I'm also happy if the regulators express a desire to=20 > help the providers achieve this standardized control plane protocol. =20 > I just get worried when th e=20 > regulators state that a new standardized control plane protocol *must* ex= ist to meet the regulatory use case. I would point out a few things: 1) I don't think anyone has stated the "must" word, even lower-case :-) --= but, I would welcome being corrected if you believe that to be the case. = Furthermore, regulators can't state things in the IETF or in any IETF docum= ents, since the IETF is a global standards organization and regulations dif= fer (widely) all across the world. 2) Second, I believe the main thrust of the present straw man proposal is = to point out the need for a _unified_ methodology in which to conduct netwo= rk performance measurements on a much broader scale than exists, at the mom= ent. Personally, I read the straw man more as a "call to action" than any = specific, detailed proposal of "a solution", but maybe that's me. 3) Finally, without an objective set of requirements it will be difficult = to impossible to objectively evaluate existing protocols, which you mention= above, to see if all of them (or only some of them or none of them) will w= holly satisfy or only partially satisfy the needs outlined in draft-schulzr= inne. IMO, I think it's premature to assume that there can only exist a si= ngle OAM protocol, at this point. Obviously, from an Engineering PoV, I th= ink many would agree that there are benefits to a single OAM protocol (deve= lopment is focused on one thing); however, there are also downsides, as wel= l. Namely, that it *will* take quite a bit of time to specify in the IETF,= procure and longer still to get adopted. Thus, if it's /feasible/ to fit = most/all the requirements incrementally into an existing protocol(s), then = time to develop, procure and deploy is decreased (considerably). Regardles= s, I think those are questions to be answered when evaluating the solution = space against a widely agreed upon set of objective requirements, but we're not there yet= . In light of the above, it would be helpful to characterize your concerns in= light of draft-schulzrinne-lmap-requirements-00, since that's the only pro= posal on the table at the moment. If there is text you can cite from that = draft that concerns you, it would be helpful to get it clarified, ask to ge= t it *out* of that draft or whatever. :-) If you really want, focus just = on Section 6, "Requirements", which is a mere 3 pages long, at this point. = :-) -shane > Anyway, from "business requirement" #1, I see the need for a set of tests= to be identified. It's clear that some PHY/access technologies and other s= pecial interest groups and standards orgs are interested in providing input= to this set. It's also clear that the IETF has expertise in Layer 3 and ab= ove metrics. =20 >=20 > I've heard some access network providers say that they want a new control= plane protocol for the architecture they want to use to meet #3 and #4. Th= e access providers who want such a protocol should be free to have this pro= tocol worked in whatever organization they wish. >=20 > As it relates to other use cases (customer wants to run tests, applicatio= n service provider wants to run tests, service provider wants test environm= ent to meet a variety of non-regulatory needs), I need to leave it up to th= e proponents of those use cases to provide the business requirements. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap >=20 _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From rolf.winter@hs-augsburg.de Tue Nov 13 08:14:01 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 43AD221F8771 for ; Tue, 13 Nov 2012 08:14:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.057 X-Spam-Level: X-Spam-Status: No, score=-1.057 tagged_above=-999 required=5 tests=[AWL=0.745, BAYES_00=-2.599, HELO_EQ_DE=0.35, HELO_MISMATCH_DE=1.448, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7SvrbqBng6bv for ; Tue, 13 Nov 2012 08:14:00 -0800 (PST) Received: from bee.RZ.HS-Augsburg.DE (bee.RZ.HS-Augsburg.DE [141.82.25.26]) by ietfa.amsl.com (Postfix) with ESMTP id 6CB3021F8769 for ; Tue, 13 Nov 2012 08:13:59 -0800 (PST) Received: from Rolfs-MacBook-Pro.local ([141.82.51.130]) (authenticated bits=0) by bee.RZ.HS-Augsburg.DE (8.14.3/8.14.3) with ESMTP id qADGDqgn029515 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 13 Nov 2012 17:13:54 +0100 (MET) Message-ID: <50A271C0.3030700@hs-augsburg.de> Date: Tue, 13 Nov 2012 17:13:52 +0100 From: Rolf Winter User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: lmap@ietf.org References: <2D09D61DDFA73D4C884805CC7865E6112F5C8DDA@GAALPA1MSGUSR9L.ITServices.sbc.com>, <9510D26531EF184D9017DF24659BB87F33F336A0B2@EMV65-UKRD.domain1.systemhost.net> <2D09D61DDFA73D4C884805CC7865E6112F5C947A@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5CAE96@GAALPA1MSGUSR9L.ITServices.sbc.com> <50A121EA.7080303@hs-augsburg.de> <2D09D61DDFA73D4C884805CC7865E6112F5CC293@GAALPA1MSGUSR9L.ITServices.sbc.com> <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (bee.RZ.HS-Augsburg.DE [141.82.16.16]); Tue, 13 Nov 2012 17:13:54 +0100 (MET) X-HSA-RZ-MailScanner-Information: Please contact the ISP for more information X-HSA-RZ-MailScanner-ID: qADGDqgn029515 X-HSA-RZ-MailScanner: Found to be clean X-HSA-RZ-MailScanner-From: rolf.winter@hs-augsburg.de X-HSA-RZ-MailScanner-Watermark: 1353428034.19406@a4gvrrMwt62eZWWWERTCWg Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2012 16:14:01 -0000 Hi, if you have to report them today, then it should be trivial to define them as I assume they have already been defined. I would expect that at least some of the other questions should have an answer, too? Is there official information on those measurements available? I mean not the data itself but what is requested by the regulator (not just US but other countries as well)? Best, Rolf On 11/13/12 5:04 PM, Bugenhagen, Michael K wrote: > Shane - > > The only requirement I am fairly clear on is that some of the ISP's *(US based at least) are required to produce performance results per speed tier for Customers. This was verbally mentioned in the BAR BOF, but I didn't look for it in the draft. > > So from a written requirements priority view I would think defining the what those PM's are, and how to perform the PM's, along with how to report the PM (Validating / cleaning the data) are key. > > > Regards, > Mike > > > > > > -----Original Message----- > From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Shane Amante > Sent: Monday, November 12, 2012 10:24 PM > To: STARK, BARBARA H > Cc: Rolf Winter; lmap@ietf.org > Subject: Re: [lmap] Scope, Use Cases and Users, and Which are Mandatory vs. Nice to Have > > Barbara, > > On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" wrote: >>> I really wonder what exactly you mean with business requirements here >>> and what the IETF can or should do about those. Could you spell some >>> of these requirements out? >> >> Yes. I've been trying to write down what I think I heard as being the basic "business" requirements coming from Henning and James at the IETF. >> I haven't managed to write it up into something formal, but here are some high-level statements: >> >> Regulators must be able to >> 1) specify what tests get run; the set of tests needs to cover all access technologies and allow "apples to apples" comparison of different access network providers, the technologies they use, and among results of different customers; the set of tests is expected to change over time (but hopefully not too quickly, since some tests may require considerable lead time to ensure that test infrastructure is in place, and to understand any performance/security risks that the tests my incur); note that different regulators may choose to require different sets of tests. >> 2) ensure that there is sufficient sampling of customers (of access >> network providers) to ensure coverage of all access network providers, >> geographies, and access technologies, where the sample size is large >> enough to ensure anonymity of results and limit the effect of >> statistical variation >> 3) be confident that data needed to provide meaningful test results on >> a per-customer level will exist, be collected, and collated >> 4) have per-customer anonymous test results be available to the >> regulator and researchers >> >> The first item implies to me that there is a need for a set of recommended tests. The set of tests will drive requirements not only for endpoints within the customer premises network, but also for test endpoints in the access network or beyond. >> The second item, to me, looks less like an architectural element and more like a recommendation for what constitutes an appropriate sample size. >> The third implies to me that access network providers will need to be able to ensure that this is the case. >> The fourth implies to me that access network providers will need to make sure that they have some way of making results available, and that the results are per-customer but anonymous. >> >> So when I look at items 1, 3 and 4, I see that these cause access network providers to have requirements. >> Not all access network providers are alike. >> Some access network providers may want to have a 100% proprietary test control and management and internal data collection architecture. >> Some access network providers may want to re-use their existing TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures. >> Some access network providers may want to put a new box (like SamKnows) or an app (on a PC/smart phone/USB stick/etc.) in the customer premises network, and have the option of doing this themselves or contracting it out to a 3rd party. >> Some access network providers may want a new architecture that perhaps re-uses some existing architectural elements, but maybe adds a new "control plane" protocol. Some may want to use a proprietary control plane protocol, and some may want to get together with other access network providers and standardize such a protocol. > > First, I think the correct term to use, above, is: "management protocol" or OAM protocol, not "control plane protocol". Examples of the former in the IETF are: SNMP, NETCONF. In the IETF, I think if/when people see "control plane" protocol, they will instantly assume you mean: BGP, OSPF, LDP, etc. Sorry to be the "word police" :-). > > Second, I think it's important to clarify your comments in light of "straw man" architectural proposal in draft-schulzrinne-lmap-requirements-00. I only say this because, IMO, I think that provides a decent starting point to ensure we share the same terminology and points-of-reference in this dialogue. Specifically, it sounds like you're solely (?) focused on the "measurement client" in the above, correct? > > >> If I have accurately represented the James/Henning "business >> requirements" that I heard, then I see nothing in them that would >> prohibit any of the access network provider options that I've listed. >> Which would make me happy. On the other hand, if a regulator is coming >> forward and saying "I have a requirement that the architecture for >> providing metrics must use a specific control plane protocol", then >> I'm concerned; because it would prohibit all architectures that do not >> use it. If an access network provider is coming forward and saying "I >> want a new control plane protocol to exist because I want to meet the >> regulatory requirements with an architecture that makes use of it", >> then I'm happy to support them (well, if they can get other providers >> to also want it -- I dislike creating "standards" that are only used >> by one provider). I'm also happy if the regulators express a desire to >> help the providers achieve this standardized control plane protocol. >> I just get worried when th > e >> regulators state that a new standardized control plane protocol *must* exist to meet the regulatory use case. > > I would point out a few things: > 1) I don't think anyone has stated the "must" word, even lower-case :-) -- but, I would welcome being corrected if you believe that to be the case. Furthermore, regulators can't state things in the IETF or in any IETF documents, since the IETF is a global standards organization and regulations differ (widely) all across the world. > 2) Second, I believe the main thrust of the present straw man proposal is to point out the need for a _unified_ methodology in which to conduct network performance measurements on a much broader scale than exists, at the moment. Personally, I read the straw man more as a "call to action" than any specific, detailed proposal of "a solution", but maybe that's me. > 3) Finally, without an objective set of requirements it will be difficult to impossible to objectively evaluate existing protocols, which you mention above, to see if all of them (or only some of them or none of them) will wholly satisfy or only partially satisfy the needs outlined in draft-schulzrinne. IMO, I think it's premature to assume that there can only exist a single OAM protocol, at this point. Obviously, from an Engineering PoV, I think many would agree that there are benefits to a single OAM protocol (development is focused on one thing); however, there are also downsides, as well. Namely, that it *will* take quite a bit of time to specify in the IETF, procure and longer still to get adopted. Thus, if it's /feasible/ to fit most/all the requirements incrementally into an existing protocol(s), then time to develop, procure and deploy is decreased (considerably). Regardless, I think those are questions to be answered when evaluating the solution space agains! t a > widely agreed upon set of objective requirements, but we're not there yet. > > In light of the above, it would be helpful to characterize your concerns in light of draft-schulzrinne-lmap-requirements-00, since that's the only proposal on the table at the moment. If there is text you can cite from that draft that concerns you, it would be helpful to get it clarified, ask to get it *out* of that draft or whatever. :-) If you really want, focus just on Section 6, "Requirements", which is a mere 3 pages long, at this point. :-) > > -shane > > > > >> Anyway, from "business requirement" #1, I see the need for a set of tests to be identified. It's clear that some PHY/access technologies and other special interest groups and standards orgs are interested in providing input to this set. It's also clear that the IETF has expertise in Layer 3 and above metrics. >> >> I've heard some access network providers say that they want a new control plane protocol for the architecture they want to use to meet #3 and #4. The access providers who want such a protocol should be free to have this protocol worked in whatever organization they wish. >> >> As it relates to other use cases (customer wants to run tests, application service provider wants to run tests, service provider wants test environment to meet a variety of non-regulatory needs), I need to leave it up to the proponents of those use cases to provide the business requirements. >> Barbara >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >> > > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > From bs7652@att.com Tue Nov 13 09:07:07 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6264921F86FD for ; Tue, 13 Nov 2012 09:07:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6NMtPoNzpasI for ; Tue, 13 Nov 2012 09:07:03 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id 19F7C21F86B6 for ; Tue, 13 Nov 2012 09:07:01 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo08.seg.att.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 63e72a05.2aaad160e940.732104.00-534.2005924.nbfkord-smmo08.seg.att.com (envelope-from ); Tue, 13 Nov 2012 17:07:02 +0000 (UTC) X-MXL-Hash: 50a27e36056abef9-b58d04457bf8da46a52959a12bacb9072d62064e Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id a2e72a05.0.732059.00-416.2005738.nbfkord-smmo08.seg.att.com (envelope-from ); Tue, 13 Nov 2012 17:06:52 +0000 (UTC) X-MXL-Hash: 50a27e2c57bfc7f8-9cc5f29b772b6e1b964a0e49a60c338664e04bc3 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qADH6ma7013925; Tue, 13 Nov 2012 12:06:50 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qADH6hFo013620 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Nov 2012 12:06:44 -0500 Received: from GAALPA1MSGHUB9D.ITServices.sbc.com (gaalpa1msghub9d.itservices.sbc.com [130.8.36.90]) by sflint03.pst.cso.att.com (RSA Interceptor); Tue, 13 Nov 2012 12:06:26 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9D.ITServices.sbc.com ([130.8.36.90]) with mapi id 14.02.0318.001; Tue, 13 Nov 2012 12:06:25 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: Terminology Thread-Index: Ac3BwTW4OFGLEuUdQzKjb+p/4zsUbw== Date: Tue, 13 Nov 2012 17:06:24 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5CCE45@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [90.164.202.31] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=N+ee4RBB c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=LnSh8mvAXcUA:10 a=ofMgfj31e3cA:10 a=BLceEmwcHowA:10 a=kj9] X-AnalysisOut: [zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32RAAAA:8 a=bH6hqU01j] X-AnalysisOut: [EgA:10 a=-O5YPv7yAAAA:8 a=1aUtNOBKX3kOq-P1GuIA:9 a=CjuIK1q] X-AnalysisOut: [_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: [lmap] Terminology X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Nov 2012 17:07:07 -0000 I'm separating my response into a couple of topics, for better thread ident= ification. > > Some access network providers may want a new architecture that perhaps > > re-uses some existing architectural elements, but maybe adds a new "con= trol > > plane" protocol. Some may want to use a proprietary control plane proto= col, > > and some may want to get together with other access network providers a= nd > > standardize such a protocol. >=20 > First, I think the correct term to use, above, is: "management protocol" = or > OAM protocol, not "control plane protocol". Examples of the former in th= e > IETF are: SNMP, NETCONF. In the IETF, I think if/when people see "contro= l > plane" protocol, they will instantly assume you mean: BGP, OSPF, LDP, etc= . > Sorry to be the "word police" :-). Yes, I'm struggling with collision of various worlds here.=20 In the M2M world, there has been a lot of discussion around something that = has been referred to as a "control plane" protocol. See http://docbox.etsi.= org/workshop/2011/201110_m2mworkshop/03_m2mcooperation/broadbandforum_walls= .pdf slides 10 and 13-18, for an example. There have been suggestions elsew= here that something similar may be used to do near-real-time control of per= formance testing. An OAM protocol (Operation, Administration, and Management) is generally us= ed for reasonably static configuration. They aren't optimized for near-real= -time control. In the case of measurement architectures, I've seen suggesti= ons that an OAM protocol might be used for configuring aspects of measureme= nt testing (e.g., firmware or apps that include testing capabilities), whil= e a near-real-time protocol might be needed for controlling the tests. Howe= ver, I understand that people in IETF tend to use "control plane" in the co= ntext of network control, and not device/application control. So I agree it= 's best not to use that on an IETF list. I consider "measurement client to = measurement controller" to be a little bit long as the title for something = to discuss. Perhaps we can refer to it in discussion here as a Measurement = Control Protocol? Barbara From jason.weil@twcable.com Wed Nov 14 08:46:42 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B48321F8481 for ; Wed, 14 Nov 2012 08:46:42 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.679 X-Spam-Level: X-Spam-Status: No, score=-0.679 tagged_above=-999 required=5 tests=[AWL=0.784, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xgbG69f1WduV for ; Wed, 14 Nov 2012 08:46:41 -0800 (PST) Received: from cdpipgw01.twcable.com (cdpipgw01.twcable.com [165.237.59.22]) by ietfa.amsl.com (Postfix) with ESMTP id 27F1421F86A3 for ; Wed, 14 Nov 2012 08:46:41 -0800 (PST) X-SENDER-IP: 10.136.163.14 X-SENDER-REPUTATION: None X-IronPort-AV: E=Sophos;i="4.83,251,1352091600"; d="scan'208";a="470650632" Received: from unknown (HELO PRVPEXHUB05.corp.twcable.com) ([10.136.163.14]) by cdpipgw01.twcable.com with ESMTP/TLS/RC4-MD5; 14 Nov 2012 11:43:30 -0500 Received: from PRVPEXVS12.corp.twcable.com ([10.136.163.45]) by PRVPEXHUB05.corp.twcable.com ([10.136.163.14]) with mapi; Wed, 14 Nov 2012 11:43:37 -0500 From: "Weil, Jason" To: Shane Amante , "STARK, BARBARA H" Date: Wed, 14 Nov 2012 11:43:36 -0500 Thread-Topic: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have Thread-Index: Ac3ChzGsmEbAJkUdSsO82iB2ey5t9w== Message-ID: In-Reply-To: <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 16:46:42 -0000 I'll attempt to aggregate feedback on the draft as well as the Use Case discussion in a single email. Scope: Proposed Scope: Performance measurements for mass market broadband connectivity services. Suggested Tweak:Large-scale Performance Measurement for broadband data services Rationale - I don't think we should use the term mass market as it seems to me to imply only to large operators. Small operators (arguably not mass market) should be inclusive of the architecture as well. SECTION 2 -Use Case/s: I believe the 'Provider network measurements' Use Case in draft-schulzrinne-lmap-requirements-00 is itself generic enough to vet the requirements needed in the overall architecture. The other use cases presented seem to be a subset of this first imo. Whether the end user, the service provider or a 3rd party has access to the data stored in the collector feels like a system policy question. I believe we are putting too much emphasis on the regulatory requirement in the use case discussion. We should also capture the different network segments as this goes into the full quality of experience discussion included in this use case description. For example, the current measurement systems do a good good at measuring the access link capacity, but not a good job extending beyond that to the application source. A method for comparing application performance by region would be very useful for measuring the end-to-end customer experience and the measurement servers sitting in the path would be perfect for measuring the segment between application provider and network midpoint. Section 3 - Architecture In general I would refer to the items in this list as functional units as opposed to actual hardware "boxes" which is what it sounds like here. A physical device may have a number of these functional items included such as a measurement collector in response to a downstream clietn and a measurement client to perform upstream testing to an app servers. 3.5 - As others have mentioned I am not sure these needs to exist. A simple API to embedded systems could provide this information. SECTION 4 - Protocols: Measurement client to measurement controller - Need to add wording that the measurement controller may also return the actual test code/applet to run on the client as well as or as part of the firmware update periodically required on the measurement client. SECTION 6 - Requirements: What is alluded to but currently missing in the current version of the draft is a requirement for the active measurement testing to not negatively impact the Network under test. This is a key feature in the Measurement controller functional requirements that needs to be included from day one. Proposed C-6: The measurement controller MUST ensure that the test schedule it provides to measurement clients in its domain is generated in a manner that minimizes aggregate test load on the measured network. Another topic missing from the measurement controller discussion is the concept of topology awareness. In order to meet the requirement in C-6 above, a measurement controller requirement should include some form of network topology awareness in order to schedule tests in a controlled and deterministic manner. Proposed C-7: The measurement controller SHOULD have some awareness of the network topology for the set of measurement clients it is responsible for administering. And if we assume that there will be multiple measurement controllers that need to interoperate, then we also need some form of hierarchy such that one controller can be elected master for some subset of test nodes (possibly based on geography or administrative domain.) Other controllers could then request test scheduling from the elected master controller. Thanks, Jason On 11/12/12 11:24 PM, "Shane Amante" wrote: >Barbara, > >On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" wrote: >>> I really wonder what exactly you mean with business requirements here >>>and >>> what the IETF can or should do about those. Could you spell some of >>>these >>> requirements out? >> >> Yes. I've been trying to write down what I think I heard as being the >>basic "business" requirements coming from Henning and James at the IETF. >> I haven't managed to write it up into something formal, but here are >>some high-level statements: >> >> Regulators must be able to >> 1) specify what tests get run; the set of tests needs to cover all >>access technologies and allow "apples to apples" comparison of different >>access network providers, the technologies they use, and among results >>of different customers; the set of tests is expected to change over time >>(but hopefully not too quickly, since some tests may require >>considerable lead time to ensure that test infrastructure is in place, >>and to understand any performance/security risks that the tests my >>incur); note that different regulators may choose to require different >>sets of tests. >> 2) ensure that there is sufficient sampling of customers (of access >>network providers) to ensure coverage of all access network providers, >>geographies, and access technologies, where the sample size is large >>enough to ensure anonymity of results and limit the effect of >>statistical variation >> 3) be confident that data needed to provide meaningful test results on >>a per-customer level will exist, be collected, and collated >> 4) have per-customer anonymous test results be available to the >>regulator and researchers >> >> The first item implies to me that there is a need for a set of >>recommended tests. The set of tests will drive requirements not only for >>endpoints within the customer premises network, but also for test >>endpoints in the access network or beyond. >> The second item, to me, looks less like an architectural element and >>more like a recommendation for what constitutes an appropriate sample >>size. >> The third implies to me that access network providers will need to be >>able to ensure that this is the case. >> The fourth implies to me that access network providers will need to >>make sure that they have some way of making results available, and that >>the results are per-customer but anonymous. >> >> So when I look at items 1, 3 and 4, I see that these cause access >>network providers to have requirements. >> Not all access network providers are alike. >> Some access network providers may want to have a 100% proprietary test >>control and management and internal data collection architecture. >> Some access network providers may want to re-use their existing >>TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures. >> Some access network providers may want to put a new box (like SamKnows) >>or an app (on a PC/smart phone/USB stick/etc.) in the customer premises >>network, and have the option of doing this themselves or contracting it >>out to a 3rd party. >> Some access network providers may want a new architecture that perhaps >>re-uses some existing architectural elements, but maybe adds a new >>"control plane" protocol. Some may want to use a proprietary control >>plane protocol, and some may want to get together with other access >>network providers and standardize such a protocol. > >First, I think the correct term to use, above, is: "management protocol" >or OAM protocol, not "control plane protocol". Examples of the former in >the IETF are: SNMP, NETCONF. In the IETF, I think if/when people see >"control plane" protocol, they will instantly assume you mean: BGP, OSPF, >LDP, etc. Sorry to be the "word police" :-). > >Second, I think it's important to clarify your comments in light of >"straw man" architectural proposal in >draft-schulzrinne-lmap-requirements-00. I only say this because, IMO, I >think that provides a decent starting point to ensure we share the same >terminology and points-of-reference in this dialogue. Specifically, it >sounds like you're solely (?) focused on the "measurement client" in the >above, correct? > > >> If I have accurately represented the James/Henning "business >>requirements" that I heard, then I see nothing in them that would >>prohibit any of the access network provider options that I've listed. >>Which would make me happy. On the other hand, if a regulator is coming >>forward and saying "I have a requirement that the architecture for >>providing metrics must use a specific control plane protocol", then I'm >>concerned; because it would prohibit all architectures that do not use >>it. If an access network provider is coming forward and saying "I want a >>new control plane protocol to exist because I want to meet the >>regulatory requirements with an architecture that makes use of it", then >>I'm happy to support them (well, if they can get other providers to also >>want it -- I dislike creating "standards" that are only used by one >>provider). I'm also happy if the regulators express a desire to help the >>providers achieve this standardized control plane protocol. I just get >>worried when th > e >> regulators state that a new standardized control plane protocol *must* >>exist to meet the regulatory use case. > >I would point out a few things: >1) I don't think anyone has stated the "must" word, even lower-case :-) >-- but, I would welcome being corrected if you believe that to be the >case. Furthermore, regulators can't state things in the IETF or in any >IETF documents, since the IETF is a global standards organization and >regulations differ (widely) all across the world. >2) Second, I believe the main thrust of the present straw man proposal >is to point out the need for a _unified_ methodology in which to conduct >network performance measurements on a much broader scale than exists, at >the moment. Personally, I read the straw man more as a "call to action" >than any specific, detailed proposal of "a solution", but maybe that's me. >3) Finally, without an objective set of requirements it will be >difficult to impossible to objectively evaluate existing protocols, which >you mention above, to see if all of them (or only some of them or none of >them) will wholly satisfy or only partially satisfy the needs outlined in >draft-schulzrinne. IMO, I think it's premature to assume that there can >only exist a single OAM protocol, at this point. Obviously, from an >Engineering PoV, I think many would agree that there are benefits to a >single OAM protocol (development is focused on one thing); however, there >are also downsides, as well. Namely, that it *will* take quite a bit of >time to specify in the IETF, procure and longer still to get adopted. >Thus, if it's /feasible/ to fit most/all the requirements incrementally >into an existing protocol(s), then time to develop, procure and deploy is >decreased (considerably). Regardless, I think those are questions to be >answered when evaluating the solution space against a > widely agreed upon set of objective requirements, but we're not there >yet. > >In light of the above, it would be helpful to characterize your concerns >in light of draft-schulzrinne-lmap-requirements-00, since that's the only >proposal on the table at the moment. If there is text you can cite from >that draft that concerns you, it would be helpful to get it clarified, >ask to get it *out* of that draft or whatever. :-) If you really want, >focus just on Section 6, "Requirements", which is a mere 3 pages long, at >this point. :-) > >-shane > > > > >> Anyway, from "business requirement" #1, I see the need for a set of >>tests to be identified. It's clear that some PHY/access technologies and >>other special interest groups and standards orgs are interested in >>providing input to this set. It's also clear that the IETF has expertise >>in Layer 3 and above metrics. >> >> I've heard some access network providers say that they want a new >>control plane protocol for the architecture they want to use to meet #3 >>and #4. The access providers who want such a protocol should be free to >>have this protocol worked in whatever organization they wish. >> >> As it relates to other use cases (customer wants to run tests, >>application service provider wants to run tests, service provider wants >>test environment to meet a variety of non-regulatory needs), I need to >>leave it up to the proponents of those use cases to provide the business >>requirements. >> Barbara >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap >> > > >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap This E-mail and any of its attachments may contain Time Warner Cable propri= etary information, which is privileged, confidential, or subject to copyrig= ht belonging to Time Warner Cable. This E-mail is intended solely for the u= se of the individual or entity to which it is addressed. If you are not the= intended recipient of this E-mail, you are hereby notified that any dissem= ination, distribution, copying, or action taken in relation to the contents= of and attachments to this E-mail is strictly prohibited and may be unlawf= ul. If you have received this E-mail in error, please notify the sender imm= ediately and permanently delete the original and any copy of this E-mail an= d any printout. From rolf.winter@hs-augsburg.de Wed Nov 14 11:21:17 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 236C921F87A6 for ; Wed, 14 Nov 2012 11:21:17 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.801 X-Spam-Level: X-Spam-Status: No, score=-1.801 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_DE=0.35, HELO_MISMATCH_DE=1.448, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wLq5Bdgty7E7 for ; Wed, 14 Nov 2012 11:21:16 -0800 (PST) Received: from bee.RZ.HS-Augsburg.DE (bee.RZ.HS-Augsburg.DE [141.82.25.26]) by ietfa.amsl.com (Postfix) with ESMTP id B452D21F8799 for ; Wed, 14 Nov 2012 11:21:15 -0800 (PST) Received: from Rolfs-MacBook-Pro.local (ppp-188-174-32-147.dynamic.mnet-online.de [188.174.32.147]) (authenticated bits=0) by bee.RZ.HS-Augsburg.DE (8.14.3/8.14.3) with ESMTP id qAEJKxNn019543 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Nov 2012 20:20:59 +0100 (MET) Message-ID: <50A3EF1A.5030307@hs-augsburg.de> Date: Wed, 14 Nov 2012 20:20:58 +0100 From: Rolf Winter User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: "Weil, Jason" References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0 (bee.RZ.HS-Augsburg.DE [141.82.16.16]); Wed, 14 Nov 2012 20:21:01 +0100 (MET) X-HSA-RZ-MailScanner-Information: Please contact the ISP for more information X-HSA-RZ-MailScanner-ID: qAEJKxNn019543 X-HSA-RZ-MailScanner: Found to be clean X-HSA-RZ-MailScanner-From: rolf.winter@hs-augsburg.de X-HSA-RZ-MailScanner-Watermark: 1353525662.48407@VTkGfIu1jAMcdTl7gH51ww Cc: Shane Amante , "STARK, BARBARA H" , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 19:21:17 -0000 Hi, I would suggest to replace "mass market" with "residential" IFF we want to reduce scope. I would assume however that many things that will be developed for this will be broadly applicable and useful in other settings too. We might have to develop metrics of interest and maybe develop standard measurement techniques and parameters, however I'd urge to make the protocol mechanics as generic and extensible as possible. Remember RFC 5218 (of cause having the cautious considerations of RFC 6709 in mind). Best, Rolf On 11/14/12 5:43 PM, Weil, Jason wrote: > I'll attempt to aggregate feedback on the draft as well as the Use Case > discussion in a single email. > > Scope: > > Proposed Scope: Performance measurements for mass market broadband > connectivity services. > > Suggested Tweak:Large-scale Performance Measurement for broadband data > services > Rationale - I don't think we should use the term mass market as it seems > to me to imply only to large operators. Small operators (arguably not mass > market) should be inclusive of the architecture as well. > > SECTION 2 -Use Case/s: > > I believe the 'Provider network measurements' Use Case in > draft-schulzrinne-lmap-requirements-00 is itself generic enough to vet the > requirements needed in the overall architecture. The other use cases > presented seem to be a subset of this first imo. Whether the end user, the > service provider or a 3rd party has access to the data stored in the > collector feels like a system policy question. I believe we are putting > too much emphasis on the regulatory requirement in the use case > discussion. > > We should also capture the different network segments as this goes into > the full quality of experience discussion included in this use case > description. For example, the current measurement systems do a good good > at measuring the access link capacity, but not a good job extending beyond > that to the application source. A method for comparing application > performance by region would be very useful for measuring the end-to-end > customer experience and the measurement servers sitting in the path would > be perfect for measuring the segment between application provider and > network midpoint. > > Section 3 - Architecture > > In general I would refer to the items in this list as functional units as > opposed to actual hardware "boxes" which is what it sounds like here. A > physical device may have a number of these functional items included such > as a measurement collector in response to a downstream clietn and a > measurement client to perform upstream testing to an app servers. > > 3.5 - As others have mentioned I am not sure these needs to exist. A > simple API to embedded systems could provide this information. > > SECTION 4 - Protocols: > > Measurement client to measurement controller - Need to add wording that > the measurement controller may also return the actual test code/applet to > run on the client as well as or as part of the firmware update > periodically required on the measurement client. > > SECTION 6 - Requirements: > > What is alluded to but currently missing in the current version of the > draft is a requirement for the active measurement testing to not > negatively impact the Network under test. This is a key feature in the > Measurement controller functional requirements that needs to be included > from day one. > > Proposed C-6: The measurement controller MUST ensure that the test > schedule it provides to measurement clients in its domain is generated in > a manner that minimizes aggregate test load on the measured network. > > Another topic missing from the measurement controller discussion is the > concept of topology awareness. In order to meet the requirement in C-6 > above, a measurement controller requirement should include some form of > network topology awareness in order to schedule tests in a controlled and > deterministic manner. > > Proposed C-7: The measurement controller SHOULD have some awareness of the > network topology for the set of measurement clients it is responsible for > administering. > > And if we assume that there will be multiple measurement controllers that > need to interoperate, then we also need some form of hierarchy such that > one controller can be elected master for some subset of test nodes > (possibly based on geography or administrative domain.) Other controllers > could then request test scheduling from the elected master controller. > > > Thanks, > > Jason > > On 11/12/12 11:24 PM, "Shane Amante" wrote: > >> Barbara, >> >> On Nov 12, 2012, at 11:32 AM, "STARK, BARBARA H" wrote: >>>> I really wonder what exactly you mean with business requirements here >>>> and >>>> what the IETF can or should do about those. Could you spell some of >>>> these >>>> requirements out? >>> >>> Yes. I've been trying to write down what I think I heard as being the >>> basic "business" requirements coming from Henning and James at the IETF. >>> I haven't managed to write it up into something formal, but here are >>> some high-level statements: >>> >>> Regulators must be able to >>> 1) specify what tests get run; the set of tests needs to cover all >>> access technologies and allow "apples to apples" comparison of different >>> access network providers, the technologies they use, and among results >>> of different customers; the set of tests is expected to change over time >>> (but hopefully not too quickly, since some tests may require >>> considerable lead time to ensure that test infrastructure is in place, >>> and to understand any performance/security risks that the tests my >>> incur); note that different regulators may choose to require different >>> sets of tests. >>> 2) ensure that there is sufficient sampling of customers (of access >>> network providers) to ensure coverage of all access network providers, >>> geographies, and access technologies, where the sample size is large >>> enough to ensure anonymity of results and limit the effect of >>> statistical variation >>> 3) be confident that data needed to provide meaningful test results on >>> a per-customer level will exist, be collected, and collated >>> 4) have per-customer anonymous test results be available to the >>> regulator and researchers >>> >>> The first item implies to me that there is a need for a set of >>> recommended tests. The set of tests will drive requirements not only for >>> endpoints within the customer premises network, but also for test >>> endpoints in the access network or beyond. >>> The second item, to me, looks less like an architectural element and >>> more like a recommendation for what constitutes an appropriate sample >>> size. >>> The third implies to me that access network providers will need to be >>> able to ensure that this is the case. >>> The fourth implies to me that access network providers will need to >>> make sure that they have some way of making results available, and that >>> the results are per-customer but anonymous. >>> >>> So when I look at items 1, 3 and 4, I see that these cause access >>> network providers to have requirements. >>> Not all access network providers are alike. >>> Some access network providers may want to have a 100% proprietary test >>> control and management and internal data collection architecture. >>> Some access network providers may want to re-use their existing >>> TR-069/TR-143/IPDR/OMA-DM/SNMP/etc. architectures. >>> Some access network providers may want to put a new box (like SamKnows) >>> or an app (on a PC/smart phone/USB stick/etc.) in the customer premises >>> network, and have the option of doing this themselves or contracting it >>> out to a 3rd party. >>> Some access network providers may want a new architecture that perhaps >>> re-uses some existing architectural elements, but maybe adds a new >>> "control plane" protocol. Some may want to use a proprietary control >>> plane protocol, and some may want to get together with other access >>> network providers and standardize such a protocol. >> >> First, I think the correct term to use, above, is: "management protocol" >> or OAM protocol, not "control plane protocol". Examples of the former in >> the IETF are: SNMP, NETCONF. In the IETF, I think if/when people see >> "control plane" protocol, they will instantly assume you mean: BGP, OSPF, >> LDP, etc. Sorry to be the "word police" :-). >> >> Second, I think it's important to clarify your comments in light of >> "straw man" architectural proposal in >> draft-schulzrinne-lmap-requirements-00. I only say this because, IMO, I >> think that provides a decent starting point to ensure we share the same >> terminology and points-of-reference in this dialogue. Specifically, it >> sounds like you're solely (?) focused on the "measurement client" in the >> above, correct? >> >> >>> If I have accurately represented the James/Henning "business >>> requirements" that I heard, then I see nothing in them that would >>> prohibit any of the access network provider options that I've listed. >>> Which would make me happy. On the other hand, if a regulator is coming >>> forward and saying "I have a requirement that the architecture for >>> providing metrics must use a specific control plane protocol", then I'm >>> concerned; because it would prohibit all architectures that do not use >>> it. If an access network provider is coming forward and saying "I want a >>> new control plane protocol to exist because I want to meet the >>> regulatory requirements with an architecture that makes use of it", then >>> I'm happy to support them (well, if they can get other providers to also >>> want it -- I dislike creating "standards" that are only used by one >>> provider). I'm also happy if the regulators express a desire to help the >>> providers achieve this standardized control plane protocol. I just get >>> worried when th >> e >>> regulators state that a new standardized control plane protocol *must* >>> exist to meet the regulatory use case. >> >> I would point out a few things: >> 1) I don't think anyone has stated the "must" word, even lower-case :-) >> -- but, I would welcome being corrected if you believe that to be the >> case. Furthermore, regulators can't state things in the IETF or in any >> IETF documents, since the IETF is a global standards organization and >> regulations differ (widely) all across the world. >> 2) Second, I believe the main thrust of the present straw man proposal >> is to point out the need for a _unified_ methodology in which to conduct >> network performance measurements on a much broader scale than exists, at >> the moment. Personally, I read the straw man more as a "call to action" >> than any specific, detailed proposal of "a solution", but maybe that's me. >> 3) Finally, without an objective set of requirements it will be >> difficult to impossible to objectively evaluate existing protocols, which >> you mention above, to see if all of them (or only some of them or none of >> them) will wholly satisfy or only partially satisfy the needs outlined in >> draft-schulzrinne. IMO, I think it's premature to assume that there can >> only exist a single OAM protocol, at this point. Obviously, from an >> Engineering PoV, I think many would agree that there are benefits to a >> single OAM protocol (development is focused on one thing); however, there >> are also downsides, as well. Namely, that it *will* take quite a bit of >> time to specify in the IETF, procure and longer still to get adopted. >> Thus, if it's /feasible/ to fit most/all the requirements incrementally >> into an existing protocol(s), then time to develop, procure and deploy is >> decreased (considerably). Regardless, I think those are questions to be >> answered when evaluating the solution space against a >> widely agreed upon set of objective requirements, but we're not there >> yet. >> >> In light of the above, it would be helpful to characterize your concerns >> in light of draft-schulzrinne-lmap-requirements-00, since that's the only >> proposal on the table at the moment. If there is text you can cite from >> that draft that concerns you, it would be helpful to get it clarified, >> ask to get it *out* of that draft or whatever. :-) If you really want, >> focus just on Section 6, "Requirements", which is a mere 3 pages long, at >> this point. :-) >> >> -shane >> >> >> >> >>> Anyway, from "business requirement" #1, I see the need for a set of >>> tests to be identified. It's clear that some PHY/access technologies and >>> other special interest groups and standards orgs are interested in >>> providing input to this set. It's also clear that the IETF has expertise >>> in Layer 3 and above metrics. >>> >>> I've heard some access network providers say that they want a new >>> control plane protocol for the architecture they want to use to meet #3 >>> and #4. The access providers who want such a protocol should be free to >>> have this protocol worked in whatever organization they wish. >>> >>> As it relates to other use cases (customer wants to run tests, >>> application service provider wants to run tests, service provider wants >>> test environment to meet a variety of non-regulatory needs), I need to >>> leave it up to the proponents of those use cases to provide the business >>> requirements. >>> Barbara >>> _______________________________________________ >>> lmap mailing list >>> lmap@ietf.org >>> https://www.ietf.org/mailman/listinfo/lmap >>> >> >> >> _______________________________________________ >> lmap mailing list >> lmap@ietf.org >> https://www.ietf.org/mailman/listinfo/lmap > > > This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout. > From bs7652@att.com Wed Nov 14 14:23:25 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9C5DC21F84DA for ; Wed, 14 Nov 2012 14:23:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.566 X-Spam-Level: X-Spam-Status: No, score=-106.566 tagged_above=-999 required=5 tests=[AWL=0.033, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GeDWxM7-QElF for ; Wed, 14 Nov 2012 14:23:24 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id E83A821F84DF for ; Wed, 14 Nov 2012 14:23:23 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo05.seg.att.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id bd914a05.2aaace409940.1273249.00-528.3529356.nbfkord-smmo05.seg.att.com (envelope-from ); Wed, 14 Nov 2012 22:23:23 +0000 (UTC) X-MXL-Hash: 50a419db50fd85d7-666ed94344717181cc5164a2aeb7e2115eda2381 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 2d914a05.0.1273190.00-378.3529210.nbfkord-smmo05.seg.att.com (envelope-from ); Wed, 14 Nov 2012 22:23:15 +0000 (UTC) X-MXL-Hash: 50a419d316cedcde-4c43ea46a131bc3f19a2df2badd548a7ed5756a0 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEMNE7Z031063; Wed, 14 Nov 2012 17:23:14 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEMMa2U030014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Nov 2012 17:23:07 -0500 Received: from GAALPA1MSGHUB9F.ITServices.sbc.com (gaalpa1msghub9f.itservices.sbc.com [130.8.36.92]) by sflint03.pst.cso.att.com (RSA Interceptor); Wed, 14 Nov 2012 17:20:29 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9F.ITServices.sbc.com ([130.8.36.92]) with mapi id 14.02.0318.001; Wed, 14 Nov 2012 17:20:28 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: comments on draft-schulzrinne-lmap-requirements Thread-Index: Ac3Ctj0n/4I+kYPTTHCf+9h/yiH4YQ== Date: Wed, 14 Nov 2012 22:20:28 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.148.188] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=Cu4IhAED c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=SnuTjbt8LRMA:10 a=5CM9uOxGnn0A:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=Tx5D1hMhNisA:10 a=1XWaLZrsAAAA:8 a=75hyGj00_pRerG] X-AnalysisOut: [x3eyYA:9 a=CjuIK1q_8ugA:10 a=8mMUywqEAiQjmK-o:21 a=UjIkEqm] X-AnalysisOut: [q3UJ2WbsX:21] Cc: Rolf Winter , "lmap@ietf.org" Subject: [lmap] comments on draft-schulzrinne-lmap-requirements X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 22:23:25 -0000 Shane suggested that the draft is a good start and asked me to provide comm= ents against it. I've been avoiding doing that, because I'm not very fond o= f this draft, and am not sure if it can be "fixed", or if it should be comp= letely overhauled, or totally scrapped. But others apparently do like it, s= o let me explain some of my objections to the current draft. Section 2 Use Cases This section could probably be fixed. The last sentence of the Provider network measurements use case should be d= eleted, as it goes beyond use case, to describe how elements of this draft'= s proposed architecture would be used to meet the use case. I would recomme= nd changing the title and "user" of the use case to "Access Network Provide= rs", since the use case is about the needs of Access Network Providers for = measuring their own network. Note that it is possible for an ISP not to be = an access network provider. This is not as prevalent as it once was, but ma= y still be a useful distinction in some parts of the world. The measurement= needs of an ISP who does not own the access network may call for them to g= et measurement data from the access network provider. I believe that case i= s covered by the ASP case (below). User network diagnostics I would recommend renaming to just "End User" (sin= ce it seems to be concerned with the end user's own network, and the end us= er's subscribed broadband connection), and I would prefer to see running on= -demand tests as a distinct use case from accessing historical measurements= (relating to themselves). I see that the end user seems to have an interes= t in testing their own network, the access provider connection, and end-to-= end testing against an ASP. Trying to describe all of these within a single= architecture may be challenging, scope-wise. Multi-provider network measurements is a mixture of a variety of use cases = that need to be split out. There is a use case of Application Service Provi= der (ASP), who wants to tests against that ASP's customers. There is a sepa= rate use case of regulatory entities. I can't tell if the researcher use ca= se is distinct from the regulator use case or is covered by it. The ASP (wh= ich for me covers the access-reseller ISP) use case may also suggest the ne= ed for testing within their own networks, which could be very extensive (de= pending on their network). If ASPs have a need for on-demand testing, as we= ll as large-scale testing, then I would prefer to see these use cases separ= ated. Trying to describe the internal architecture for ASP network testing = in the same architecture as access provider (internal or external) testing = could be challenging, scope-wise. Including on-demand with large-scale use = cases could also be challenging. The statement that "it should be understood that a data collection platform= must serve multiple stakeholder interests" is problematic to me, because i= t suggests that an access network provider may not be permitted to deploy d= ata collection platforms that collect data intended only for use by that ac= cess network provider. This statement suggests that any platform that an ac= cess network provider deploys must also meet the needs of others. This sent= ence should either be removed or reworded (and moved to a paragraph where i= t would apply to all use cases) along the lines of "The data collection arc= hitecture should identify and define the (internal and external, or just ex= ternal/interworking?) capabilities needed to satisfy the interests of all (= some subset?) of these use cases." Defining an architecture that describes = functionality for meeting the needs of use cases is very different from ins= isting that a deployment of a measurement platform must meet the needs of a= ll stakeholders / use cases. Whether or not to deploy or include a function= should be a business decision. A well-defined functional architecture will= provide the ability to make this business decision, but will not attempt t= o mandate the business decision. Section 3 I see 2 very big problems with the architecture. 1) It is very much focused on the access provider's internal capabilities.= It seems to try to generalize some of the architectural elements to be not= necessarily an element of the access provider, but that just confuses thin= gs for me; because the requirements are very much geared towards assuming t= hat the architecture elements are under the control of the access provider.= For the most part, the draft defines one particular way in which the acces= s provider could (must) architect running tests and collecting data interna= l to their network, and provide others with access to the test capabilities= and resulting data. This is not the architecture I would recommend to the = company I work for. As such, I don't think it's all that interesting. Nor d= o I consider defining such an architecture to be a necessary step before id= entifying what the appropriate set of tests may be for doing access network= performance measurements to satisfy regulatory needs (or internal needs or= customer trouble-shooting needs). Quite frankly, I'm not particularly inte= rested in getting the IETF to help with defining an internal architecture f= or me to recommend. I'm not particularly interested in getting any org to d= o that; but if I were, there are probably other orgs where I would go first= , because they have greater expertise in the idiosyncrasies of my employer'= s access network technologies and existing management framework. With that = said -- if there are access providers who do want IETF to help them with th= is, I have no problem with that, and I might even be willing to help. But i= f there are no access providers (or their proxies) asking for IETF to defin= e this internal architecture for them, then I would recommend against the I= ETF insisting on defining such an architecture. I've been witness to way to= o many IETF discussions of the sort "I want services providers to do X and = they don't want to do X. Those bad, horrible, evil service providers. We ne= ed to find some way to force them to do X." I did read the disclaimer about= the draft expressing personal views, and I'm willing to trust that this is= truly the case. But the fact that the authors of the draft are employees o= f an agency that *does* have the ability to force access providers to do X,= together with the language of the draft, leaves me incredibly uneasy. I ho= pe that this work does not devolve into some attempt by non-access provider= s to force business models, business decisions, and specific architectures = on access providers, with subsequent access provider bashing when it become= s clear that not all access providers are willing to actually deploy (all a= spects of) the architecture.=20 2) The described architecture requires that many different functional comp= onents be present in single architectural elements. I prefer architectures = that define distinct functional components separately, and allow for the fu= nctional components to be physically placed where others feel it makes sens= e for them. It's ok to provide examples of how functions *could* exist in a= single physical box, but I generally find that *mandating* physical combin= ations within a functional architecture isn't a very good approach. Here are some specific comments on the architectural elements: Measurement client: I agree that if measurements (active or passive) are go= ing to be made, then one or more Measurement clients are a mandatory elemen= t. I agree with the first 3 sentences of the Measurement client definition.= But the idea that *all* Measurement clients must be manageable and "provid= e for communications within different network segments" is not something I = agree with. It seems to be implied that there is exactly one Measurement cl= ient for any given broadband customer. I disagree with that. A good perform= ance measurement architecture tends to consist of multiple Measurement clie= nts, each with different measurement capabilities and interfaces. The vario= us Measurement clients may also come from or be "owned" by various entities= . The requirements section requires that this element communicates in certa= in ways with other architectural elements. I disagree that these are requir= ements of all Measurement clients. IMO, the only other element that a gener= ic Measurement client is required to communicate with is the Measurement se= rver, and that only if the client does active tests. If the Measurement cli= ent is intended to be an "Access provider managed measurement client", then= this should be made explicit. Active tests involve a Measurement server. I agree with the existence of su= ch a Measurement server as an element of a performance measurement architec= ture that includes active tests. I have no problem with the definition of M= easurement server. This does assume that some of the as-yet-to-be-identifie= d tests will be active. But I'm ok with that assumption. The requirements s= ection assumes the Measurement server includes much more functionality than= this definition implies. I disagree that all Measurement servers must have= such functionality. As with Measurement clients, I would expect there to b= e multiple Measurement servers, and they may or may not be dedicated to spe= cific active tests, may or may not serve other function or have other inter= faces, and are very generic architecture elements that may be "owned" by va= rious entities. For example, I (personally) frequently use "www.google.com"= as a Measurement server that I conduct active (ping, traceroute) tests aga= inst, using my PC OS "Command Prompt" interface to provide me with the clie= nt. I also conduct tests against this Measurement server using my router as= the client, and interfacing to the router through a browser on my PC. Measurement controller: This is not a necessary architectural element. This= is an element specific to this proposed physical architecture. However, so= me of the external-facing functionality attributed to this element (in the = requirements section) may be desirable to define in the context of a functi= onal architecture, which attempts to understand the external-facing "interw= orking" functions that might be useful for accomplishing the various use ca= ses. An example of this are the requirements around a function that allows = 3rd parties to control and schedule tests to be run by elements under the c= ontrol of the access network provider. If there are no access providers ask= ing for assistance with design of their internal architectures, it might be= good to focus on understanding these external interfaces, more than the in= ternal ones. I do want to stress, though, that I consider it important to e= xpress functional requirements in such a way that it is clear "If this func= tion is desired/implemented, then the function must/should..." as opposed t= o "The (access provider's) measurement architecture must provide this funct= ion." Data collector: I do not see this as a necessary architectural element, eit= her. As with the Measurement controller, it has been assigned the role of p= roviding multiple, separable functions: collecting data, storing data, and = providing access to the data. Network parameter server: I do not see this as a necessary architectural el= ement. If there is a need for an externally-facing function that provides a= ccess to access-provider-collected data for the regulatory use case to incl= ude corresponding (anonymous) information related to subscribed line rate, = etc., then that is a requirement on that external-facing function. If an ac= cess provider wants help in figuring out how to get this information to the= external-facing function, then I would prefer for that access provider (or= their proxy) to ask for such help.=20 If there is a desire to define an interworking function that would allow a = 3rd party to get (not anonymous) customer-specific line information directl= y from the access provider (so the 3rd party could correlate it with data t= hey collect from their own Measurement client(s)), then that would be a dis= tinct function, as well. Personally, I would shy away from trying to define= this function. But if someone else wants to tackle it, I might be willing = to provide comments on it.=20 Before I move on, I'll briefly summarize the external-facing "interworking"= functions that I see the various use cases implying for an access network = provider (Again, in the context of: If the access network provider wants to= support a particular use case and implement the functions needed for it, t= hen the requirements for those functions are that the function must/should.= ..", and not "The access provider's architecture must..."). The Access Provider use case (measurements for internal access provider pur= poses) does not drive the need for any external interworking functions. The End User on-demand or historical performance use case for their home ne= twork does not need external interworking functions from the access network= provider. The End User on-demand use case for testing of the access network (with too= ls from the access network provider) would require a user interface that al= lows the end user to launch user-specified tests against their own connecti= on and view the results. The End User historical performance use case for data from the access netwo= rk provider would require a user interface that allows the end user to acce= ss historical performance metrics of their broadband connection. There seem= s to also be a requirement to allow a user to specify what tests to run (an= d parameters of those tests and schedule for running the tests). The Regulator use case would require access to anonymous line/connection sp= ecific measurement data. It appears there are also requirements for a funct= ion that allows tests to be specified, manipulated, and scheduled by the re= gulator or regulator-approved 3rd parties. The ASP use case for measurements without participation of an access provid= er would require no access provider interworking function. The ASP use case for on-demand customer-specific testing with help from an = access provider seem to require some function, but I'm not sure what it is.= The requirements section seems to imply a function that would allow the AS= P to request (from the access provider) on-demand and scheduled tests of th= e ASP's customers. And to be able to specify test parameters. The ASP use case for access to large-scale historical performance data for = their customers, with info from an access provider, does seem to require so= me function, but I'm not sure what it is. Again, there seems to be a requir= ement for the ASP to specify what tests get run, and test parameters. There also seem to be some requirements around the formatting and timestamp= ing of data provided by the access provider via the interworking functions. Having said all of this, I don't think there's much sense in my commenting = on specific requirements. I think the overall architecture is too flawed to= try to deal with the specific flaws of each and every requirement. And I d= on't even know if there's an access provider interested in having the inter= nal elements of this particular architecture properly specified. If there's an access provider who wants help in designing their internal ar= chitecture, I think it would be nice if they spoke up and described their n= eeds. Similarly, if someone else (e.g., ASPs, or those who speak on behalf = of end users) wants help in designing an internal architecture (for testing= of their private networks), it would be good to understand their needs. If= there's a desire to identify and better understand/characterize interworki= ng functions, then I'd be happy to participate in that. I'm not ready to ag= ree to specifying detailed requirements of interworking functions. But if w= e can understand and characterize them, then we can make an informed decisi= on regarding the need for detailed specification. Barbara From bs7652@att.com Wed Nov 14 14:30:49 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1E67221F87DD for ; Wed, 14 Nov 2012 14:30:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.569 X-Spam-Level: X-Spam-Status: No, score=-106.569 tagged_above=-999 required=5 tests=[AWL=0.030, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ym5-4BSuaZ7O for ; Wed, 14 Nov 2012 14:30:48 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 6F63821F85D4 for ; Wed, 14 Nov 2012 14:30:48 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 89b14a05.2aab0585b940.1281984.00-548.3544691.nbfkord-smmo06.seg.att.com (envelope-from ); Wed, 14 Nov 2012 22:30:48 +0000 (UTC) X-MXL-Hash: 50a41b986e4089a1-d4c5f3085ae70cae7b538392a09f28b6ed16032b Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id f8b14a05.0.1281944.00-479.3544572.nbfkord-smmo06.seg.att.com (envelope-from ); Wed, 14 Nov 2012 22:30:39 +0000 (UTC) X-MXL-Hash: 50a41b8f428002df-4929eca6a03c9f8a8f4012dc41b8f90df7e7fe4f Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEMUcgP003956; Wed, 14 Nov 2012 17:30:39 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEMUT6M003807 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Nov 2012 17:30:35 -0500 Received: from GAALPA1MSGHUB9F.ITServices.sbc.com (gaalpa1msghub9f.itservices.sbc.com [130.8.36.92]) by sflint04.pst.cso.att.com (RSA Interceptor); Wed, 14 Nov 2012 17:30:16 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9F.ITServices.sbc.com ([130.8.36.92]) with mapi id 14.02.0318.001; Wed, 14 Nov 2012 17:30:16 -0500 From: "STARK, BARBARA H" To: "Weil, Jason" , Shane Amante Thread-Topic: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have Thread-Index: AQHNwoelws7E4k/7WUCAAQRJGlWmbpfp54DA Date: Wed, 14 Nov 2012 22:30:14 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5CDB71@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <651F01B7-14EE-48C1-95A2-CDEDD5AA6CB4@castlepoint.net> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.148.188] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=VI9qaazX c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=SnuTjbt8LRMA:10 a=ylxSBkfk08sA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=bbd-PDihl1AA:10 a=TqjgTdAfyFY3HEJkQDcA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 22:30:49 -0000 > Proposed Scope: Performance measurements for mass market broadband > connectivity services. >=20 > Suggested Tweak:Large-scale Performance Measurement for broadband > data services Rationale - I don't think we should use the term mass marke= t as > it seems to me to imply only to large operators. Small operators (arguabl= y not > mass market) should be inclusive of the architecture as well. Just to explain -- I was trying to distinguish consumer and small business = "mass market" offerings from large business and enterprise offering that of= ten have SLAs associated with them. Many small operators also target what I= consider to be mass market customers, so I did see those as being included= . The testing done for SLA customers is pretty large-scale, so I'm worried = that "large-scale" might be considered to include them? But I'm happy with = alternate wordings, as long as we agree that we aren't trying to tackle tes= ting for SLA customers. Or, if we are, that should be specifically called o= ut, since it drives additional use cases and additional functionality. Barbara From acmorton@att.com Wed Nov 14 15:06:04 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 73C6021F8612 for ; Wed, 14 Nov 2012 15:06:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.395 X-Spam-Level: X-Spam-Status: No, score=-106.395 tagged_above=-999 required=5 tests=[AWL=0.204, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H7eXuC4yN9jF for ; Wed, 14 Nov 2012 15:06:04 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id BD71421F8551 for ; Wed, 14 Nov 2012 15:06:03 -0800 (PST) Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id bd324a05.0.527954.00-494.1474688.nbfkord-smmo08.seg.att.com (envelope-from ); Wed, 14 Nov 2012 23:06:03 +0000 (UTC) X-MXL-Hash: 50a423db5a4b070a-f6654f5c97d3d936a150f437e5623ea19619d064 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEN63lN008827 for ; Wed, 14 Nov 2012 18:06:03 -0500 Received: from sflint02.pst.cso.att.com (sflint02.pst.cso.att.com [144.154.234.229]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAEN60C7008802 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 14 Nov 2012 18:06:00 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by sflint02.pst.cso.att.com (RSA Interceptor) for ; Wed, 14 Nov 2012 18:05:50 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAEN5gFB003475 for ; Wed, 14 Nov 2012 18:05:42 -0500 Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAEN5ePa003455 for ; Wed, 14 Nov 2012 18:05:41 -0500 Received: from lt-hp1044652.att.com (vpn-135-70-155-219.vpn.mwst.att.com[135.70.155.219](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121114230553gw100632d2e>; Wed, 14 Nov 2012 23:05:55 +0000 X-Originating-IP: [135.70.155.219] Message-Id: <7.0.1.0.0.20121114171318.04ba1b08@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Wed, 14 Nov 2012 18:01:20 -0500 To: "Weil, Jason" , Shane Amante , "STARK, BARBARA H" From: Al Morton Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.145] X-AnalysisOut: [v=2.0 cv=INm1/HTG c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a] X-AnalysisOut: [=aoRfTMA3LyUA:10 a=Hz8c2qrut9wA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=bbd-PDih] X-AnalysisOut: [l1AA:10 a=e4nAUMgRFoQnWEIq6RMA:9 a=CjuIK1q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] Scope, Use Cases and Users and Which are Mandatory vs. Nice to Have X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 23:06:04 -0000 At 11:43 AM 11/14/2012, Weil, Jason wrote: >...SECTION 6 - Requirements: > >What is alluded to but currently missing in the current version of the >draft is a requirement for the active measurement testing to not >negatively impact the Network under test. This is a key feature in the >Measurement controller functional requirements that needs to be included >from day one. > >Proposed C-6: The measurement controller MUST ensure that the test >schedule it provides to measurement clients in its domain is generated in >a manner that minimizes aggregate test load on the measured network. +1, I'd even go further and include the metrics measured. It is an important tenet of the IPPM framework RFC 2330, sec6.2: ... Finally, some measurement methodologies may be 'conservative' in the sense that the act of measurement does not modify, or only slightly modifies, the value of the performance metric the methodology attempts to measure. {Comment: for example, in a wide-area high-speed network under modest load, a test using several small 'ping' packets to measure delay would likely not interfere (much) with the delay properties of that network as observed by others. The corresponding statement about tests using a large flow to measure flow capacity would likely fail.} There are many reasons to prefer a "conservative" methodology, especially if a reasonable projection or model based on fundamental metrics is possible/available. regards, Al From shane@castlepoint.net Thu Nov 15 13:54:50 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AA6D621F8618 for ; Thu, 15 Nov 2012 13:54:50 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.437 X-Spam-Level: X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0UoECC7Q-Mzb for ; Thu, 15 Nov 2012 13:54:49 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 746A821F8A59 for ; Thu, 15 Nov 2012 13:54:49 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 3DC1E2166 for ; Thu, 15 Nov 2012 21:54:48 +0000 (UTC) Received: from [10.9.0.10] (web.hollyman.com [64.78.239.73]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 0CA51211E; Thu, 15 Nov 2012 14:54:47 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 15 Nov 2012 14:54:44 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 15 14:54:48 2012 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 50a564a8199631495917699 X-DSPAM-Factors: 27, the+Network, 0.40000, the+Network, 0.40000, addresses+#+Measurement, 0.40000, force+#+#+#+X, 0.40000, shooting+#+#+frankly, 0.40000, using+#+IP, 0.40000, using+#+IP, 0.40000, functions+could, 0.40000, shane+#+#+en, 0.40000, that+#+this, 0.40000, I'm+#+particularly, 0.40000, I'm+#+particularly, 0.40000, I+#+#+#+work, 0.40000, to+#+#+#+or, 0.40000, to+#+#+#+or, 0.40000, But+#+#+#+here, 0.40000, function+#+hosted, 0.40000, to+#+#+presumably, 0.40000, they+#+#+to, 0.40000, to+#+#+#+an, 0.40000, to+#+#+#+an, 0.40000, be+#+#+where, 0.40000, to+#+#+#+to, 0.40000, to+#+#+#+to, 0.40000, 1+#+#+#+that, 0.40000, if+#+#+there, 0.40000, what+the, 0.40000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Nov 2012 21:54:50 -0000 Barbara, If you don't mind, I'm going to attempt to separate this very detailed, = but long :-), response out into separate threads to, hopefully, make the = dialogue somewhat more manageable. For now, I'm starting with your comments on Section 3. Other threads = will follow. On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: [---snip---] > Section 3 > I see 2 very big problems with the architecture. > 1) It is very much focused on the access provider's internal = capabilities. It seems to try to generalize some of the architectural = elements to be not necessarily an element of the access provider, but = that just confuses things for me; because the requirements are very much = geared towards assuming that the architecture elements are under the = control of the access provider. This seems like a good place to start, because I get the sense you may = be concerned about another party (e.g.: regulator) taking over control = of -- or dictating that Access Network Provider's install -- LMAP = architectural elements on the Access Network Provider's network. (Let = me know if I'm misunderstanding). My interpretation of the = draft-schulzrinne-lmap-requirements was that LMAP was aiming more to = build a cooperative 'federation' of these architectural elements. = However, I would agree that that interpretation, on my part, (or even = the word "federation") does not appear to be in the draft. Certainly = either interpretation, but hopefully the latter one :-), should be = clarified in a future revision of that draft, IMO. IOW, at a very high-level, let's say that the ANP has deployed their own = Measurement Controller. And, a Regulator, or other third-party, has = their own Measurement Controller. One potential solution I could see = emerging is that: a) Regulator's Measurement Controller asks ANP's Measurement Controller = for a "token" or set of "tokens" to authenticate itself for some period = of time to speak _directly_, using some IP-based protocol (NETCONF, who = knows?), with a group of the ANP's Measurement Clients. b) Assuming the Regulator's Measurement Controller receives those = tokens, then the Regulator's Measurement Controller speaks directly to = the ANP's Measurement Clients to ask them to perform tests, presumably = using some IP-based application protocol, (e.g.: SNMP, NETCONF, other?), = to make the request and send the results back to regulator's Data = Controller(s). I would note that the 'elegance' of the above model is that the ANP is = still "in control" insomuch as it needs to hand authentication tokens = (and, IP addresses) of Measurement Clients before the third-party = Measurement Controller can even ask the ANP's Measurement Clients to do = anything. No tokens =3D=3D no access to clients. One downside is that = Measurement Clients would clearly have to support whatever "new", or = _modified_, protocol it is that enables the Measurement Controller to = Measurement Client communication to take place. However, I'm personally = *not* worried about that at this stage, because we're still talking = about requirements. Finally, I would note that the above is just _one_ illustration of a = potential outcome in this space that could potentially satisfy the = current "straw man" requirements. I want to be clear that I'm not = stating that the above is the _only_ potential solution, nor is it even = be the best potential solution. But, that's why we're here talking = about this. :-) > For the most part, the draft defines one particular way in which the = access provider could (must) architect running tests and collecting data = internal to their network, and provide others with access to the test = capabilities and resulting data. This is not the architecture I would = recommend to the company I work for. As such, I don't think it's all = that interesting. Nor do I consider defining such an architecture to be = a necessary step before identifying what the appropriate set of tests = may be for doing access network performance measurements to satisfy = regulatory needs (or internal needs or customer troub > le-shooting needs). Quite frankly, I'm not particularly interested in = getting the IETF to help with defining an internal architecture for me = to recommend. I'm not particularly interested in getting any org to do = that; but if I were, there are probably other orgs where I would go = first, because they have greater expertise in the idiosyncrasies of my = employer's access network technologies and existing management = framework. With that said -- if there are access providers who do want = IETF to help them with this, I have no problem with that, and I might = even be willing to help. But if there are no access providers (or their = proxies) asking for IETF to define this internal architecture for them, = then I would recommend against the IETF insisting on defining such an = architecture. I've been witness to way too many IETF discussions of the = sort "I want services providers to do X and they don't want to do X. = Those bad, horrible, evil service providers. We need to find some way to = force=20 > them to do X." I did read the disclaimer about the draft expressing = personal views, and I'm willing to trust that this is truly the case. = But the fact that the authors of the draft are employees of an agency = that *does* have the ability to force access providers to do X, together = with the language of the draft, leaves me incredibly uneasy. I hope that = this work does not devolve into some attempt by non-access providers to = force business models, business decisions, and specific architectures on = access providers, with subsequent access provider bashing when it = becomes clear that not all access providers are willing to actually = deploy (all aspects of) the architecture.=20 > 2) The described architecture requires that many different functional = components be present in single architectural elements. I prefer = architectures that define distinct functional components separately, and = allow for the functional components to be physically placed where others = feel it makes sense for them. It's ok to provide examples of how = functions *could* exist in a single physical box, but I generally find = that *mandating* physical combinations within a functional architecture = isn't a very good approach. Re: comments in the last two sentences, I'm not clear that the present = draft was 'mandating' combinations of or placement of particular = functions. Rather, my read of the draft was that some functions or = placement seemed to make intuitive sense, i.e.: the Measurement Client = needs to be at the customer premise. I did not detect a mandate on placement of the Measurement Servers in = the draft. I can see how the "Network Parameter Server" may be hosted = on the ANP network, (because it happens to be an existing box/server = they already have). I can also see another potential answer, as well. = Namely, the "Network Parameter Server" may be a function/server hosted = at a third-party (e.g.: regulator) that periodically receives a 'push' = of updated data from the ANP's "Network Parameter Server"/system(s). I = can also see both functions co-existing at the same time with no harm, = as well, e.g.: one ANP may choose one method and a second ANP may choose = another ... both will work. I would make the same comment about the = "Data Collector". With respect to the "Measurement Controller" and the = case where the ANP owns/maintains the Measurement Client, its placement = may have to be at both the third-party and the ANP, due to (to be = defined?) needs of the ANP to authenticate requests, grant access to = speak to Measurement Clients, etc. This does not preclude the = third-party Measurement Controller from, ultimately, speaking directly = to a Measurement Client, but before the third-party Measurement = Controller does so it needs to first ask the ANP's Measurement = Controller "nicely". :-) At a high-level, this may be similar to how, = for example, OAuth works in the Application Layer (third-party HTTPS = authentication[1]). I'm not suggesting that OAuth is the only or even = best idea here, just that it's the closest analogy I can think of for = how this may ultimately be realized. -shane [1] https://en.wikipedia.org/wiki/OAuth= From shane@castlepoint.net Thu Nov 15 18:10:40 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C27D221F845A for ; Thu, 15 Nov 2012 18:10:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.271 X-Spam-Level: X-Spam-Status: No, score=-0.271 tagged_above=-999 required=5 tests=[AWL=0.166, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jjXZcNyqqxjf for ; Thu, 15 Nov 2012 18:10:40 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id ED2DB1F0C44 for ; Thu, 15 Nov 2012 18:10:39 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 9EFD62164 for ; Fri, 16 Nov 2012 02:10:38 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id E3A512109; Thu, 15 Nov 2012 19:10:36 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 15 Nov 2012 19:10:35 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 15 19:10:38 2012 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 50a5a09e199631022420521 X-DSPAM-Factors: 27, for+#+#+#+Packet, 0.40000, that+#+element, 0.40000, says+#+#+#+Client, 0.40000, are+resource, 0.40000, performance+#+#+#+the, 0.40000, contexts+#+#+#+to, 0.40000, multiple+Measurement, 0.40000, user's+#+#+#+snip, 0.40000, by+#+#+#+e, 0.40000, your+#+#+in, 0.40000, What+the, 0.40000, latency+loss, 0.40000, with+#+first, 0.40000, can+#+#+For, 0.40000, here+in, 0.40000, Points+#+#+Switches, 0.40000, can+#+#+with, 0.40000, smartphones+#+Some, 0.40000, sure+#+#+much, 0.40000, the+#+#+#+provided, 0.40000, is+#+something, 0.40000, above+text, 0.40000, are+#+constrained, 0.40000, test+#+receipt, 0.40000, and+#+#+#+the, 0.40000, owned+#+operated, 0.40000, owned+#+operated, 0.40000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Client X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 02:10:40 -0000 Barbara, Continuing the discussion specific to your comments on the Measurement = Client. On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: [--snip--] > Here are some specific comments on the architectural elements: > Measurement client: I agree that if measurements (active or passive) = are going to be made, then one or more Measurement clients are a = mandatory element. I agree with the first 3 sentences of the Measurement = client definition. Well 3 out of 4 (sentences) is a good start. :-) > But the idea that *all* Measurement clients must be manageable and = "provide for communications within different network segments" is not = something I agree with. I'm not sure I detect an implication that all measurement clients must = be manageable. What the fourth sentence says is the following: "Client measurement functionality must be implementable in a variety of = user contexts ..." = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... at this point, I believe the above sentence is stating that as we = think about developing solutions there may be various classes of = "measurement client" devices, e.g.: CPE modem, WiFi AP's?, Home routers? = (e.g.: D-Link, Netgear, Linksys, etc.), PC's, laptops, smartphones?, = other? Some of those devices are, naturally, going to be more = resource-constrained (CPU, flash, DRAM) than others. Thus, we need to = either consider: a) Designing a single solution for the lowest common denominator = measurement client possible; and/or, b) Designing "profiles" that various classes of measurement client can = conform to. For example, a "Basic profile" measurement client may only = be able to do basic throughput, latency & loss measurements, etc. An = "Advanced profile" measurement client may be able to do all of the = "Basic profile", plus fancier stuff like test for receipt of ICMPv6 = Packet Too Big packets to ensure that Path MTU is working ... or = something. I'm sure there are much better examples one can come up with = than that. :-) Perhaps we could ask the authors to modify the first part of that = sentence as follows: ---snip--- There is expected to be a variety of measurement clients in the = architecture. Some classes of measurement clients may be more resource = constrained than others in terms of CPU, DRAM and Flash memory. The = primary example are CPE modems. Secondary examples are: WiFi Access = Points, Home Routers/Switches, Personal Computers, Laptops, Tablets, = Smartphones, etc. Client measurement functionality must be = implementable across various classes of devices, in a variety of user = contexts ... ---snip--- > It seems to be implied that there is exactly one Measurement client = for any given broadband customer. I disagree with that. A good = performance measurement architecture tends to consist of multiple = Measurement clients, each with different measurement capabilities and = interfaces. Hopefully, the additional context I provided in my above text may = address the above? > The various Measurement clients may also come from or be "owned" by = various entities. That's a really good point that the current text does not capture. How = about we ask for the following text to be folded in: ---snip--- Measurement clients may be owned and operated by an ISP and other = measurement clients may be owned and operated by the ISPs customer, = (e.g.: home user's PC or tablet). ---snip--- > The requirements section requires that this element communicates in = certain ways with other architectural elements. I disagree that these = are requirements of all Measurement clients. IMO, the only other element = that a generic Measurement=20 > client is required to communicate with is the Measurement server, and = that only if the client does active tests. Hrm. I'm not sure I'd agree with the above. How do you get the = results, from a just completed performance measurement test, off the = measurement client? IOW, the measurement client will need to "push" its = test results to a "data collector" function (server in "the cloud"), = which presumably has lots of disk to store results? Presumably if the = CPE devices are resource constrained, then they won't be able to keep = perf. measurement results around that long in DRAM, or Flash. > If the Measurement client is intended to be an "Access provider = managed measurement client", then this should be made explicit. Perhaps we should not prematurely constrain the definition of the = "measurement client" here in the architecture section, since this is = supposed to be representative of all classes of measurement client. = Rather, we should look at (if necessary) incorporating your above = sentence in the requirements section? Or, perhaps, we end up with = requirements per Use Case where that belongs? Thanks, -shane= From shane@castlepoint.net Thu Nov 15 19:02:40 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 960091F0C7F for ; Thu, 15 Nov 2012 19:02:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.287 X-Spam-Level: X-Spam-Status: No, score=-0.287 tagged_above=-999 required=5 tests=[AWL=0.150, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SZ4JzrZin0yB for ; Thu, 15 Nov 2012 19:02:40 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id E10A41F0C6E for ; Thu, 15 Nov 2012 19:02:39 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id E72C12164 for ; Fri, 16 Nov 2012 03:02:38 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id DFF351A4B; Thu, 15 Nov 2012 20:02:37 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 15 Nov 2012 20:02:37 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <592CA143-D70B-4AA1-AC0C-A69DC8AD30C7@castlepoint.net> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 15 20:02:38 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50a5acce199632948953382 X-DSPAM-Factors: 27, Cc*Rolf+#+#+hs-augsburg.de, 0.01000, To*STARK+BARBARA, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, I+#+that, 0.01000, I+#+that, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, a+single, 0.01000, a+single, 0.01000, Cc*Winter+#+hs-augsburg.de, 0.01000, From*Amante+shane, 0.01000, are+#+to, 0.01000, are+#+to, 0.01000, the+#+#+the, 0.01000, 2012+at, 0.01000, Mime-Version*Mail+#+1499, 0.01000, To*bs7652+att.com, 0.01000, com+wrote, 0.01000, be+#+to, 0.01000, be+#+to, 0.01000, Mime-Version*6.2+1499, 0.01000, att+#+wrote, 0.01000, STARK+BARBARA, 0.01000, Mime-Version*OS+#+#+#+1499, 0.01000, To*BARBARA+#+#+att.com, 0.01000, To*BARBARA+#+bs7652, 0.01000, Cc*ietf.org+lmap, 0.01000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Server X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 03:02:40 -0000 Barbara, Continuing the discussion specific to your comments on the Measurement = Server. On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: [--snip--] > Active tests involve a Measurement server. I agree with the existence = of such a Measurement server as an element of a performance measurement = architecture that includes active tests. I have no problem with the = definition of Measurement server. This does assume that some of the = as-yet-to-be-identified tests will be active. But I'm ok with that = assumption. As am I. > The requirements section assumes the Measurement server includes much = more functionality than this definition implies. Actually, it's funny you should mention this, because Section 6, = Requirements, doesn't have a single requirement tagged with "S-*", which = would presumably apply _specifically_ to the measurement server = function. :-) But, there is this disclaimer, though:=20 In many cases, a single requirement governs more than one entity or protocol, so the labeling should be considered rough. Anyway, are the following requirements, the two that you find most = concerning wrt the Measurement Server? ---snip--- A-2: Measurement clients and servers MUST support an extensible set of performance metrics. D-3: The data exchange protocol between measurement server and data collector SHOULD allow the definition of common data elements, e.g., for network addresses and timestamps. ---snip--- > I disagree that all Measurement servers must have such functionality. = As with Measurement clients, I would expect there to be multiple = Measurement servers, and they may or may not be dedicated to specific = active tests, may or may not serve other function or have other = interfaces, and are very generic architecture elements that may be = "owned" by various entities. First, wrt Requirement A-2, I can see how a "MUST" may be concerning in = light of the physical capabilities of a particular class/type of = measurement server. However, I read that requirement more in light of = "The LMAP Architecture ...", not a specific implementation mandate. = Perhaps to avoid any doubt, maybe something like the following sentence = needs to get added to the introduction paragraph to section 6: ---snip--- The following requirements are applicable to the overall LMAP Reference = Architecture, as described in this document. How each of these = requirements are applied to specific implementation of a particular = type, category or class of measurement client, server, etc. device is = out-of-scope for this document. ---snip--- Is the above text good enough, for now? Re: D-3, I would note that says "SHOULD", which should provide some = flexibility in ensuring that future measurement servers need not adhere = to this specific requirement. But, going back to Section 3.2, Measurement Server, I can see how it may = be useful to add some of the points you raised above. How about = suggesting to add something like the following to Section 3.2? ---snip--- Measurement servers likely will be owned and operated by a variety of = administrative entities, some of which may be unrelated to the ISP = providing service to a measurement client. Therefore, measurement = servers may need a facility where they can describe their capability to = perform one, or more, suites of active performance measurement tests to = a measurement controller. In this way, the measurement controller may = facilitate coordination of measurement clients to measurement servers = that have matching active performance measurement capabilities. =20 ---snip--- Thanks, -shane= From shane@castlepoint.net Thu Nov 15 19:39:04 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8361421E803C for ; Thu, 15 Nov 2012 19:39:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.301 X-Spam-Level: X-Spam-Status: No, score=-0.301 tagged_above=-999 required=5 tests=[AWL=0.136, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0ZWe3jDR48YD for ; Thu, 15 Nov 2012 19:39:04 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id D649C1F0C6E for ; Thu, 15 Nov 2012 19:39:03 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id A9AB32168 for ; Fri, 16 Nov 2012 03:39:02 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 79D592109; Thu, 15 Nov 2012 20:39:01 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 15 Nov 2012 20:39:00 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 15 20:39:02 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50a5b556199631869412759 X-DSPAM-Factors: 27, to+#+the, 0.01000, to+#+the, 0.01000, PM+#+#+#+bs7652, 0.01000, Cc*Rolf+#+#+hs-augsburg.de, 0.01000, To*STARK+BARBARA, 0.01000, 2012+#+3, 0.01000, 3+20, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, a+single, 0.01000, a+single, 0.01000, 20+#+#+BARBARA, 0.01000, Cc*Winter+#+hs-augsburg.de, 0.01000, Subject*comments+on, 0.01000, From*Amante+shane, 0.01000, 14+2012, 0.01000, PM+#+BARBARA, 0.01000, Nov+#+#+#+3, 0.01000, 14+#+#+3, 0.01000, the+#+#+the, 0.01000, PM+#+#+H, 0.01000, 2012+at, 0.01000, Mime-Version*Mail+#+1499, 0.01000, at+#+#+PM, 0.01000, the+Measurement, 0.01000, To*bs7652+att.com, 0.01000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 03:39:04 -0000 Barbara, I'm still going ... on to the Measurement Controller. :-) On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: [--snip--] > Measurement controller: This is not a necessary architectural element. = This is an element specific to this proposed physical architecture. I'm unclear if your concern is related to: 1) having _strictly_ a single measurement controller be part of, or in = control of a single measurement "panel" (e.g.: group of measurement = clients & measurement server); or, 2) two, or more, measurement controllers (likely owned/operated by = different parties) acting in concert to coordinate the rendezvous = between an appropriate panel of measurement clients with measurement = servers. See my previous e-mail re: "Section 3" for an example of what = I'm trying to describe here. If neither of those are what concern you, can you point me in the right = direction? :-) > However, some of the external-facing functionality attributed to this = element (in the requirements section) may be desirable to define in the = context of a functional architecture, which attempts to understand the = external-facing "interworking" functions that might be useful for = accomplishing the various use cases. An example of this are the = requirements around a function that allows 3rd parties to control and = schedule tests to be run by elements under the control of the access = network provider. If there are no access providers asking for assistance = with design of their internal architectures, it might be good to focus = on understanding these external interfaces, more than the internal ones. Good points. That said, I would offer two points: A) We may also want to develop companion drafts/RFCs down the road that = describe the "internal architectures" if only in the context of an = Informational doc, so that (for example) developers and outside = operators, of third-party measurement controllers, can see the larger = picture and understand that when a third-party developed/owned/operated = measurement controller 'calls' to ISP-owned/operated measurement = controller they have a (high-level) picture of what's happening behind = the scenes. B) It is conceivable that there may be measurement platforms = (potentially encompassing everything from the measurement controller = through measurement client) that are not owned or operated by the ISP. = In this case, I believe, we definitely do want to clearly specify what = the requirements for standards-based interoperability of the measurement = controller to measurement client. > I do want to stress, though, that I consider it important to express = functional requirements in such a way that it is clear "If this function = is desired > /implemented, then the function must/should..." as opposed to "The = (access provider's) measurement architecture must provide this = function." Again, if we're talking about a broad set of requirements, applicable to = a larger set of potential constituents I think it's premature to = restrict the scope of applicability of the architectural elements, for = now.=20 -shane= From shane@castlepoint.net Thu Nov 15 20:03:42 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 98AA521E8040 for ; Thu, 15 Nov 2012 20:03:42 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.312 X-Spam-Level: X-Spam-Status: No, score=-0.312 tagged_above=-999 required=5 tests=[AWL=0.125, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mk3xmkMot6dz for ; Thu, 15 Nov 2012 20:03:42 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id ECF0C21E802E for ; Thu, 15 Nov 2012 20:03:41 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 0AC00216A for ; Fri, 16 Nov 2012 04:03:41 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 076172156; Thu, 15 Nov 2012 21:03:39 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 15 Nov 2012 21:03:39 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 15 21:03:40 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50a5bb1c199636484665750 X-DSPAM-Factors: 27, to+#+#+#+to, 0.01000, and+#+#+#+the, 0.01000, and+#+#+#+the, 0.01000, PM+#+#+#+bs7652, 0.01000, Cc*Rolf+#+#+hs-augsburg.de, 0.01000, To*STARK+BARBARA, 0.01000, 2012+#+3, 0.01000, 3+20, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, the+access, 0.01000, the+access, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, 20+#+#+BARBARA, 0.01000, the+#+#+#+to, 0.01000, the+#+#+#+to, 0.01000, Cc*Winter+#+hs-augsburg.de, 0.01000, Subject*comments+on, 0.01000, From*Amante+shane, 0.01000, 14+2012, 0.01000, comments+on, 0.01000, PM+#+BARBARA, 0.01000, Nov+#+#+#+3, 0.01000, 14+#+#+3, 0.01000, PM+#+#+H, 0.01000, 2012+at, 0.01000, Mime-Version*Mail+#+1499, 0.01000 Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Data Collector & Network Parameter Server X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 04:03:42 -0000 Barbara, OK, on to the Data Collector & Network Parameter Server. On Nov 14, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: [--snip--] > Data collector: I do not see this as a necessary architectural = element, either. As with the Measurement controller, it has been = assigned the role of providing multiple, separable functions: collecting = data, storing data, and providing access to the data. I'm not sure I understand your concern. Given the resource constraints = on some classes of measurement clients, doesn't it make sense for the = measurement client to send it's performance measurement results to "the = cloud" (a.k.a.: data collector) for longer-term archival, storage and = centralized access by the ISP and/or 3rd-parties? > Network parameter server: I do not see this as a necessary = architectural element. If there is a need for an externally-facing = function that provides access to access-provider-collected data for the = regulatory use case to include corresponding (anonymous) information = related to subscribed line rate, etc., then that is a requirement on = that external-facing function. If an access provider wants help in = figuring out how to get this information to the external-facing = function, then I would prefer for that access provider (or their proxy) = to ask for such help. I would think that if, say, an ISP already has a "network parameter = server" they already own & operate, then that's fine ... they can = continue to run it as they do today. However, the key aspect we = probably want to work on is defining a _standards-based_ "data model" = (e.g.: XML or JSON schema?) that can encapsulate appropriate/relevant = bits of data about subscribed lines in that (existing) Network Parameter = Server. This would allow for much more straightforward access to and = parsing of "network parameters" by, for example, 'authorized parties' as = the draft states (Section 3.4). > If there is a desire to define an interworking function that would = allow a 3rd party to get (not anonymous) customer-specific line = information directly from the access provider (so the 3rd party could = correlate it with data they collect from their own Measurement = client(s)), then that would be a distinct function, as well. Personally, = I would shy away from trying to define this function. But if someone = else wants to tackle it, I might be willing to provide comments on it.=20= Hrm. You raise a good point above about a 3rd-party acquiring = "non-anonymous" customer line information from the access provider so = that the 3rd-party could correlate it with data they collect from their = own (3rd-party) measurement clients. =20 1) I'm not sure what, if any, technical mechanisms the IETF could = develop to prevent such a thing from happening, other than to NOT = specify any data-models/schemas that encapsulate such sensitive, = non-anonymous information in a 'standards-based' data-model in the first = place. 2) That said, I would think that "privacy laws" in place in the U.S. = and other parts of the world (perhaps most significantly, the EU) would = prevent the ISPs from [ever] offering access to that data for fear of = violating both the law and, more importantly, their customer's trust as = well. OK, I think I'm going to stop replying to this e-mail chain for now to: = a) let you catch up; and, b) for me to get some sleep. :-) Thanks again, -shane= From trevor.burbridge@bt.com Fri Nov 16 01:26:49 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B82D121F857C for ; Fri, 16 Nov 2012 01:26:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.299 X-Spam-Level: X-Spam-Status: No, score=-3.299 tagged_above=-999 required=5 tests=[AWL=0.300, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 29yeWqGhRi0a for ; Fri, 16 Nov 2012 01:26:49 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp61.intersmtp.com [62.239.224.234]) by ietfa.amsl.com (Postfix) with ESMTP id CBB8C21F8577 for ; Fri, 16 Nov 2012 01:26:48 -0800 (PST) Received: from EVMHT63-UKRD.domain1.systemhost.net (10.36.3.100) by RDW083A005ED61.smtp-e1.hygiene.service (10.187.98.10) with Microsoft SMTP Server (TLS) id 8.3.279.1; Fri, 16 Nov 2012 09:26:56 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.92]) by EVMHT63-UKRD.domain1.systemhost.net ([10.36.3.100]) with mapi; Fri, 16 Nov 2012 09:26:47 +0000 From: To: , Date: Fri, 16 Nov 2012 09:26:45 +0000 Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 Thread-Index: Ac3De9gCzIGz5Tu0RaOOQpFUBJ1D3QAXU8aQ Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> In-Reply-To: <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: rolf.winter@hs-augsburg.de, lmap@ietf.org Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 09:26:50 -0000 Shane, As you say your scenario of multiple measurement controllers is not the onl= y way to achieve federation. Nor do I believe it is the most appropriate.=20 I'm much more inclined to believe that an organisation (regulator, ISP or o= ther) would deploy both their own clients and controller. Federation can be= achieved by sharing the data - e.g. an individual user or ISP can share da= ta for their line(s) with the regulator. For many purposes this data can be= pre-processed (errors removed, limits/filters applied, lines anonymised, a= ggregate values calculated) that will be appropriate for different relation= ships. The reasons I'm not convinced about multiple controllers: - the test controller will gain many benefits by being integrated with othe= r management systems - for example a home gateway management system. The de= vice security can be managed much more easily. In addition the device and l= ine characteristics gathered by such management systems can be used to tail= or the test schedule and configuration. For example higher speed lines may = run shorter throughput tests or older CPE may run lighter loads. Lines with= boost capabilities can be set to run longer throughput tests to produce bo= ost and off-boost values. - Two controllers means that there is potential for conflict between the te= sts. How do we know we are not overloading the device? How do we assign pri= ority between tests when they both want to run at the same time? Is the non= -ISP party able to force a test schedule that would impact on a user experi= ence? The ISP is responsible for maintaining the user experience, not the r= egulator. Trevor Burbridge >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >Shane Amante >Sent: 15 November 2012 21:55 >To: STARK, BARBARA H >Cc: Rolf Winter; lmap@ietf.org >Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: >Section 3 > >Barbara, [---snip---] >IOW, at a very high-level, let's say that the ANP has deployed their own >Measurement Controller. And, a Regulator, or other third-party, has their >own Measurement Controller. One potential solution I could see emerging i= s >that: >a) Regulator's Measurement Controller asks ANP's Measurement Controller >for a "token" or set of "tokens" to authenticate itself for some period of= time >to speak _directly_, using some IP-based protocol (NETCONF, who knows?), >with a group of the ANP's Measurement Clients. >b) Assuming the Regulator's Measurement Controller receives those tokens, >then the Regulator's Measurement Controller speaks directly to the ANP's >Measurement Clients to ask them to perform tests, presumably using some >IP-based application protocol, (e.g.: SNMP, NETCONF, other?), to make the >request and send the results back to regulator's Data Controller(s). > >I would note that the 'elegance' of the above model is that the ANP is sti= ll "in >control" insomuch as it needs to hand authentication tokens (and, IP >addresses) of Measurement Clients before the third-party Measurement >Controller can even ask the ANP's Measurement Clients to do anything. No >tokens =3D=3D no access to clients. One downside is that Measurement Clie= nts >would clearly have to support whatever "new", or _modified_, protocol it i= s >that enables the Measurement Controller to Measurement Client >communication to take place. However, I'm personally *not* worried about >that at this stage, because we're still talking about requirements. > >Finally, I would note that the above is just _one_ illustration of a poten= tial >outcome in this space that could potentially satisfy the current "straw ma= n" >requirements. I want to be clear that I'm not stating that the above is t= he >_only_ potential solution, nor is it even be the best potential solution. = But, >that's why we're here talking about this. :-) From alessandro.capello@telecomitalia.it Fri Nov 16 02:54:39 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A212B21F8607 for ; Fri, 16 Nov 2012 02:54:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.419 X-Spam-Level: X-Spam-Status: No, score=-0.419 tagged_above=-999 required=5 tests=[AWL=1.300, BAYES_00=-2.599, HELO_EQ_IT=0.635, HOST_EQ_IT=1.245, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1QojCh582Diz for ; Fri, 16 Nov 2012 02:54:39 -0800 (PST) Received: from GRFEDG702BA020.telecomitalia.it (grfedg702ba020.telecomitalia.it [156.54.233.201]) by ietfa.amsl.com (Postfix) with ESMTP id 64BF021F8604 for ; Fri, 16 Nov 2012 02:54:38 -0800 (PST) Received: from GRFHUB703BA020.griffon.local (10.188.101.113) by GRFEDG702BA020.telecomitalia.it (10.188.45.101) with Microsoft SMTP Server (TLS) id 8.3.279.5; Fri, 16 Nov 2012 11:54:29 +0100 Received: from GRFMBX702BA020.griffon.local ([10.188.101.11]) by GRFHUB703BA020.griffon.local ([10.188.101.113]) with mapi; Fri, 16 Nov 2012 11:54:28 +0100 From: Capello Alessandro To: "trevor.burbridge@bt.com" , "shane@castlepoint.net" , "bs7652@att.com" Date: Fri, 16 Nov 2012 11:54:26 +0100 Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 Thread-Index: Ac3De9gCzIGz5Tu0RaOOQpFUBJ1D3QAXU8aQAAO16xA= Message-ID: <36A93B31228D3B49B691AD31652BCAE9A5A71B35A6@GRFMBX702BA020.griffon.local> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> In-Reply-To: Accept-Language: en-US, it-IT Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, it-IT x-ti-disclaimer: Disclaimer1 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "rolf.winter@hs-augsburg.de" , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 10:54:39 -0000 Hi all, IMO IETF activities should focus only on identifying a set of abstract func= tions and on defining a flexible data model and extensible protocols (or ch= oosing/adapting existing ones) to enable communications between these funct= ions. Basic function could be, for instance: . ability to perform a measurement . ability to collect measurement results from other entities and to p= erform aggregation/pre-elaboration . ability to trigger a specific measurement on a different entity . . Some of these functions will be mandatory, other optional. Also an agreement on metrics is necessary but without constraining how they= are measured. Any other specification (e.g. where the functions are placed, who owns them= , etc.) is IMO out of scope for IETF and falls into the "over-specification= " category, something I consider a mistake in general (just a personal opin= ion). The draft actually contains some elements of over-specification, probably b= ecause it was written having a specific implementation in mind. But it also= catches some important technical issues: lack of a proper data model and l= ack of standard interfaces/protocols for handling measurements. These techn= ical issues should be solved without imposing unnecessary constraints in or= der to let anyone run its own business at best. Regards, Alessandro -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of tre= vor.burbridge@bt.com Sent: venerd=EC 16 novembre 2012 10.27 To: shane@castlepoint.net; bs7652@att.com Cc: rolf.winter@hs-augsburg.de; lmap@ietf.org Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Sectio= n 3 Shane, As you say your scenario of multiple measurement controllers is not the onl= y way to achieve federation. Nor do I believe it is the most appropriate. I'm much more inclined to believe that an organisation (regulator, ISP or o= ther) would deploy both their own clients and controller. Federation can be= achieved by sharing the data - e.g. an individual user or ISP can share da= ta for their line(s) with the regulator. For many purposes this data can be= pre-processed (errors removed, limits/filters applied, lines anonymised, a= ggregate values calculated) that will be appropriate for different relation= ships. The reasons I'm not convinced about multiple controllers: - the test controller will gain many benefits by being integrated with othe= r management systems - for example a home gateway management system. The de= vice security can be managed much more easily. In addition the device and l= ine characteristics gathered by such management systems can be used to tail= or the test schedule and configuration. For example higher speed lines may = run shorter throughput tests or older CPE may run lighter loads. Lines with= boost capabilities can be set to run longer throughput tests to produce bo= ost and off-boost values. - Two controllers means that there is potential for conflict between the te= sts. How do we know we are not overloading the device? How do we assign pri= ority between tests when they both want to run at the same time? Is the non= -ISP party able to force a test schedule that would impact on a user experi= ence? The ISP is responsible for maintaining the user experience, not the r= egulator. Trevor Burbridge >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >Shane Amante >Sent: 15 November 2012 21:55 >To: STARK, BARBARA H >Cc: Rolf Winter; lmap@ietf.org >Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: >Section 3 > >Barbara, [---snip---] >IOW, at a very high-level, let's say that the ANP has deployed their own >Measurement Controller. And, a Regulator, or other third-party, has their >own Measurement Controller. One potential solution I could see emerging i= s >that: >a) Regulator's Measurement Controller asks ANP's Measurement Controller >for a "token" or set of "tokens" to authenticate itself for some period of= time >to speak _directly_, using some IP-based protocol (NETCONF, who knows?), >with a group of the ANP's Measurement Clients. >b) Assuming the Regulator's Measurement Controller receives those tokens, >then the Regulator's Measurement Controller speaks directly to the ANP's >Measurement Clients to ask them to perform tests, presumably using some >IP-based application protocol, (e.g.: SNMP, NETCONF, other?), to make the >request and send the results back to regulator's Data Controller(s). > >I would note that the 'elegance' of the above model is that the ANP is sti= ll "in >control" insomuch as it needs to hand authentication tokens (and, IP >addresses) of Measurement Clients before the third-party Measurement >Controller can even ask the ANP's Measurement Clients to do anything. No >tokens =3D=3D no access to clients. One downside is that Measurement Clie= nts >would clearly have to support whatever "new", or _modified_, protocol it i= s >that enables the Measurement Controller to Measurement Client >communication to take place. However, I'm personally *not* worried about >that at this stage, because we're still talking about requirements. > >Finally, I would note that the above is just _one_ illustration of a poten= tial >outcome in this space that could potentially satisfy the current "straw ma= n" >requirements. I want to be clear that I'm not stating that the above is t= he >_only_ potential solution, nor is it even be the best potential solution. = But, >that's why we're here talking about this. :-) _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle per= sone indicate. La diffusione, copia o qualsiasi altra azione derivante dall= a conoscenza di queste informazioni sono rigorosamente vietate. Qualora abb= iate ricevuto questo documento per errore siete cortesemente pregati di dar= ne immediata comunicazione al mittente e di provvedere alla sua distruzione= , Grazie. This e-mail and any attachments is confidential and may contain privileged = information intended for the addressee(s) only. Dissemination, copying, pri= nting or use by anybody else is unauthorised. If you are not the intended r= ecipient, please delete this message and any attachments and advise the sen= der by return e-mail, Thanks. From trevor.burbridge@bt.com Fri Nov 16 02:59:14 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C2ECE21F85FC for ; Fri, 16 Nov 2012 02:59:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.149 X-Spam-Level: X-Spam-Status: No, score=-3.149 tagged_above=-999 required=5 tests=[AWL=-0.150, BAYES_00=-2.599, J_CHICKENPOX_91=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p9ye7DMJJCSp for ; Fri, 16 Nov 2012 02:59:13 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp61.intersmtp.com [62.239.224.234]) by ietfa.amsl.com (Postfix) with ESMTP id 543B221F857C for ; Fri, 16 Nov 2012 02:59:13 -0800 (PST) Received: from EVMHT65-UKRD.domain1.systemhost.net (10.36.3.102) by RDW083A005ED61.smtp-e1.hygiene.service (10.187.98.10) with Microsoft SMTP Server (TLS) id 8.3.279.1; Fri, 16 Nov 2012 10:59:21 +0000 Received: from EMV64-UKRD.domain1.systemhost.net ([169.254.2.92]) by EVMHT65-UKRD.domain1.systemhost.net ([10.36.3.102]) with mapi; Fri, 16 Nov 2012 10:59:12 +0000 From: To: , , Date: Fri, 16 Nov 2012 10:59:11 +0000 Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 Thread-Index: Ac3De9gCzIGz5Tu0RaOOQpFUBJ1D3QAXU8aQAAO16xAAAExgoA== Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> <36A93B31228D3B49B691AD31652BCAE9A5A71B35A6@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A5A71B35A6@GRFMBX702BA020.griffon.local> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: rolf.winter@hs-augsburg.de, lmap@ietf.org Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 10:59:14 -0000 I agree. However, that doesn't mean that we don't have to decide (a) if federation i= s required; and (b) how we would achieve it through a technical solution. Trevor Burbridge >-----Original Message----- >From: Capello Alessandro [mailto:alessandro.capello@telecomitalia.it] >Sent: 16 November 2012 10:54 >To: Burbridge,T,Trevor,DUB8 R; shane@castlepoint.net; bs7652@att.com >Cc: rolf.winter@hs-augsburg.de; lmap@ietf.org >Subject: RE: [lmap] comments on draft-schulzrinne-lmap-requirements: >Section 3 > >Hi all, > >IMO IETF activities should focus only on identifying a set of abstract >functions and on defining a flexible data model and extensible protocols (= or >choosing/adapting existing ones) to enable communications between these >functions. > >Basic function could be, for instance: >. ability to perform a measurement >. ability to collect measurement results from other entities and to >perform aggregation/pre-elaboration >. ability to trigger a specific measurement on a different entity >. . > >Some of these functions will be mandatory, other optional. >Also an agreement on metrics is necessary but without constraining how >they are measured. > >Any other specification (e.g. where the functions are placed, who owns the= m, >etc.) is IMO out of scope for IETF and falls into the "over-specification" >category, something I consider a mistake in general (just a personal opini= on). > >The draft actually contains some elements of over-specification, probably >because it was written having a specific implementation in mind. But it al= so >catches some important technical issues: lack of a proper data model and >lack of standard interfaces/protocols for handling measurements. These >technical issues should be solved without imposing unnecessary constraints >in order to let anyone run its own business at best. > >Regards, >Alessandro > > >-----Original Message----- >From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >trevor.burbridge@bt.com >Sent: venerd=EC 16 novembre 2012 10.27 >To: shane@castlepoint.net; bs7652@att.com >Cc: rolf.winter@hs-augsburg.de; lmap@ietf.org >Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: >Section 3 > >Shane, > >As you say your scenario of multiple measurement controllers is not the on= ly >way to achieve federation. Nor do I believe it is the most appropriate. > >I'm much more inclined to believe that an organisation (regulator, ISP or >other) would deploy both their own clients and controller. Federation can = be >achieved by sharing the data - e.g. an individual user or ISP can share da= ta for >their line(s) with the regulator. For many purposes this data can be pre- >processed (errors removed, limits/filters applied, lines anonymised, >aggregate values calculated) that will be appropriate for different >relationships. > >The reasons I'm not convinced about multiple controllers: >- the test controller will gain many benefits by being integrated with oth= er >management systems - for example a home gateway management system. >The device security can be managed much more easily. In addition the devic= e >and line characteristics gathered by such management systems can be used >to tailor the test schedule and configuration. For example higher speed li= nes >may run shorter throughput tests or older CPE may run lighter loads. Lines >with boost capabilities can be set to run longer throughput tests to produ= ce >boost and off-boost values. >- Two controllers means that there is potential for conflict between the t= ests. >How do we know we are not overloading the device? How do we assign >priority between tests when they both want to run at the same time? Is the >non-ISP party able to force a test schedule that would impact on a user >experience? The ISP is responsible for maintaining the user experience, no= t >the regulator. > >Trevor Burbridge > >>-----Original Message----- >>From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of >>Shane Amante >>Sent: 15 November 2012 21:55 >>To: STARK, BARBARA H >>Cc: Rolf Winter; lmap@ietf.org >>Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: >>Section 3 >> >>Barbara, >[---snip---] >>IOW, at a very high-level, let's say that the ANP has deployed their >>own Measurement Controller. And, a Regulator, or other third-party, >>has their own Measurement Controller. One potential solution I could >>see emerging is >>that: >>a) Regulator's Measurement Controller asks ANP's Measurement >>Controller for a "token" or set of "tokens" to authenticate itself for >>some period of time to speak _directly_, using some IP-based protocol >>(NETCONF, who knows?), with a group of the ANP's Measurement Clients. >>b) Assuming the Regulator's Measurement Controller receives those >>tokens, then the Regulator's Measurement Controller speaks directly to >>the ANP's Measurement Clients to ask them to perform tests, presumably >>using some IP-based application protocol, (e.g.: SNMP, NETCONF, >>other?), to make the request and send the results back to regulator's Dat= a >Controller(s). >> >>I would note that the 'elegance' of the above model is that the ANP is >>still "in control" insomuch as it needs to hand authentication tokens >>(and, IP >>addresses) of Measurement Clients before the third-party Measurement >>Controller can even ask the ANP's Measurement Clients to do anything. >>No tokens =3D=3D no access to clients. One downside is that Measurement >>Clients would clearly have to support whatever "new", or _modified_, >>protocol it is that enables the Measurement Controller to Measurement >>Client communication to take place. However, I'm personally *not* >>worried about that at this stage, because we're still talking about >requirements. >> >>Finally, I would note that the above is just _one_ illustration of a >>potential outcome in this space that could potentially satisfy the curren= t >"straw man" >>requirements. I want to be clear that I'm not stating that the above >>is the _only_ potential solution, nor is it even be the best potential >>solution. But, that's why we're here talking about this. :-) > >_______________________________________________ >lmap mailing list >lmap@ietf.org >https://www.ietf.org/mailman/listinfo/lmap > >Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle >persone indicate. La diffusione, copia o qualsiasi altra azione derivante = dalla >conoscenza di queste informazioni sono rigorosamente vietate. Qualora >abbiate ricevuto questo documento per errore siete cortesemente pregati di >darne immediata comunicazione al mittente e di provvedere alla sua >distruzione, Grazie. > >This e-mail and any attachments is confidential and may contain privileged >information intended for the addressee(s) only. Dissemination, copying, >printing or use by anybody else is unauthorised. If you are not the intend= ed >recipient, please delete this message and any attachments and advise the >sender by return e-mail, Thanks. From bs7652@att.com Fri Nov 16 11:07:55 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B1DE221F8669 for ; Fri, 16 Nov 2012 11:07:55 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.646 X-Spam-Level: X-Spam-Status: No, score=-106.646 tagged_above=-999 required=5 tests=[AWL=-0.047, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kTB4RuidFa8W for ; Fri, 16 Nov 2012 11:07:55 -0800 (PST) Received: from nbfkord-smmo04.seg.att.com (nbfkord-smmo04.seg.att.com [209.65.160.86]) by ietfa.amsl.com (Postfix) with ESMTP id 13C0B21F850E for ; Fri, 16 Nov 2012 11:07:55 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo04.seg.att.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id b0f86a05.2aaada21c940.1983823.00-575.5536018.nbfkord-smmo04.seg.att.com (envelope-from ); Fri, 16 Nov 2012 19:07:55 +0000 (UTC) X-MXL-Hash: 50a68f0b7c8995e2-ab4bdb7ed558908ab067da4839c6b9f261ef7820 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 7fe86a05.0.1983710.00-434.5535696.nbfkord-smmo04.seg.att.com (envelope-from ); Fri, 16 Nov 2012 19:07:38 +0000 (UTC) X-MXL-Hash: 50a68efa4bcea05c-f0622721cf6fe1e9d3c06e6d5eeb33642ac6782f Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAGJ7ZEP025317; Fri, 16 Nov 2012 14:07:35 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAGJ76AL023910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 16 Nov 2012 14:07:32 -0500 Received: from GAALPA1MSGHUB9A.ITServices.sbc.com (gaalpa1msghub9a.itservices.sbc.com [130.8.36.87]) by sflint03.pst.cso.att.com (RSA Interceptor); Fri, 16 Nov 2012 14:06:44 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9A.ITServices.sbc.com ([130.8.36.87]) with mapi id 14.02.0318.001; Fri, 16 Nov 2012 14:06:44 -0500 From: "STARK, BARBARA H" To: Capello Alessandro , "trevor.burbridge@bt.com" , "shane@castlepoint.net" Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 Thread-Index: AQHNw+jGcg84XSJXhUGbNCJtOI/eI5fsyWsg Date: Fri, 16 Nov 2012 19:06:44 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DC40B@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> <36A93B31228D3B49B691AD31652BCAE9A5A71B35A6@GRFMBX702BA020.griffon.local> In-Reply-To: <36A93B31228D3B49B691AD31652BCAE9A5A71B35A6@GRFMBX702BA020.griffon.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.160.196] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=M5v63VMs c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=irxWeyX9JDoA:10 a=JREhNs1m1kEA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=HtSEp9yQD4UA:10 a=1QMgPyk96JeNcc9RwmMA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: "rolf.winter@hs-augsburg.de" , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 19:07:55 -0000 > Hi all, >=20 > IMO IETF activities should focus only on identifying a set of abstract fu= nctions > and on defining a flexible data model and extensible protocols (or > choosing/adapting existing ones) to enable communications between these > functions. I think that identifying abstract functions is reasonable. I believe = that whether or not data models or protocols are created to enable the func= tions is dependent on whether there exist entities who are willing and want= ing to implement the functions (and not just entities who want to try to re= quire others to implement). Protocols developed to enable functions that ha= ve no willing deployers aren't a very good idea. > Basic function could be, for instance: > . ability to perform a measurement > . ability to collect measurement results from other entities and to= perform > aggregation/pre-elaboration > . ability to trigger a specific measurement on a different entity > . . >=20 > Some of these functions will be mandatory, other optional. > Also an agreement on metrics is necessary but without constraining how > they are measured. Some functions may be *conditionally* mandatory -- that is, if a part= icular use case is to be supported, then a particular function is necessary= . I hope that there is no attempt within the IETF to impose use case suppor= t on any entity. And, just because a function is needed to support a use ca= se, doesn't mean it has to be fully architected and defined. If no potentia= l implementers of a use case see a compelling reason to support that use ca= se, then proceeding with detailed definition of the functions needed for th= e use case is not something I would recommend. Similarly, if no potential i= mplementers of a function that has no external interfaces are asking for as= sistance with defining the function, I would recommend against detailed wor= k on that function.=20 Barbara From bs7652@att.com Mon Nov 19 09:27:05 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 819C821F86DA for ; Mon, 19 Nov 2012 09:27:05 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.896 X-Spam-Level: X-Spam-Status: No, score=-105.896 tagged_above=-999 required=5 tests=[AWL=-0.786, BAYES_05=-1.11, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QYqOfsW6b1P2 for ; Mon, 19 Nov 2012 09:27:04 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id 8F48421F86C7 for ; Mon, 19 Nov 2012 09:27:04 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo08.seg.att.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 8eb6aa05.2aaae1a28940.320687.00-570.877077.nbfkord-smmo08.seg.att.com (envelope-from ); Mon, 19 Nov 2012 17:27:04 +0000 (UTC) X-MXL-Hash: 50aa6be863ead927-29061f6d130226160156db8c69d250f6c901f47b Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id ddb6aa05.0.320532.00-439.876624.nbfkord-smmo08.seg.att.com (envelope-from ); Mon, 19 Nov 2012 17:26:55 +0000 (UTC) X-MXL-Hash: 50aa6bdf7d3cd230-a84d4fe665b5198b2e75c5ed30a732de17bb3ffe Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJHQpUu010378; Mon, 19 Nov 2012 12:26:53 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJHQ7Fj009314 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Nov 2012 12:26:49 -0500 Received: from GAALPA1MSGHUB9C.ITServices.sbc.com (gaalpa1msghub9c.itservices.sbc.com [130.8.36.89]) by sflint03.pst.cso.att.com (RSA Interceptor); Mon, 19 Nov 2012 12:25:37 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9C.ITServices.sbc.com ([130.8.36.89]) with mapi id 14.02.0318.001; Mon, 19 Nov 2012 12:25:37 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 Thread-Index: AQHNw3vYzbTV5bmGG0SdRCQDhON38JfxXzXA Date: Mon, 19 Nov 2012 17:25:36 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD00B@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> In-Reply-To: <33BD35E8-2B98-45E8-8639-A17C3EA63F18@castlepoint.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.93.19] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=OqD4PVDt c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=GD_NjPN4rjcA:10 a=Mu8DY_ZSATEA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=HtSEp9yQD4UA:10 a=hA6WZjjBopajybZN9wIA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Section 3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 17:27:05 -0000 > [---snip---] > My interpretation of the draft-schulzrinne-lmap- > requirements was that LMAP was aiming more to build a cooperative > 'federation' of these architectural elements. However, I would agree tha= t > that interpretation, on my part, (or even the word "federation") does not > appear to be in the draft. Certainly either interpretation, but hopefull= y the > latter one :-), should be clarified in a future revision of that draft, I= MO. But is such "federation" really needed? IMO, no. The complexity it cr= eates around control and management and security (management of credentials= , permissions, ensuring that all necessary safeguard are in place to preven= t 3rd parties from negatively impacting a customer's service), with a need = for execution environments in often simple client devices is considerable. = I believe that insisting on the existence of such "federation" is overkill = for the problem of providing metrics, measurements and tools sufficient for= any of the base use cases (around enabling assessment of access networks a= nd allow for troubleshooting and diagnosis of problems). I believe that it is possible for a well-defined set of metrics to allow fo= r identification of a reasonably static set of measurements and tests, that= doesn't require constant updating of and changing of the Measurement Clien= ts, or external management of the testing. While it is generally good to al= low for evolution of the tests over time, I believe this can be accomplishe= d in many cases through occasional firmware upgrades, handled directly by t= he device owner/manager, rather than a dynamic changing of test application= s controlled by 3rd parties. I believe that good data can be provided to re= gulators and researchers without allowing them to control, schedule, or oth= erwise manage the tests. I believe consumers can be provided local control = of their on-demand and scheduled (active) testing, that doesn't require all= sorts of tokens and permissions and massive security, threat mitigation, a= nd prevention of undesired impacts. I believe that 3rd parties who want to = test with a specific customer, can work directly with that customer (and th= rough the customer's local tools), and do not need direct access to Access = Network Provider (ANP) elements. I believe that focusing on identification of metrics would be much more ben= eficial to providing a solution for the basic use cases, rather than focusi= ng on an architecture that requires massive security, new management and co= ntrol interfaces, and client execution environments. [---snip---] > Rather, my read of the draft was that some functions or placement seemed > to make intuitive sense, i.e.: the Measurement Client needs to be at the > customer premise. In DSL environments, there is a single physical point-to-point connec= tion between the customer premises and the DSLAM. For passive measurements,= the DSLAM is a possible place to do passive measurement of DSL throughput,= as well as some simple DSL/ATM/Ethernet-specific active tests. Even some s= imple active tests at IP (like a ping). In PON cases, the ONT at the side o= f the house or at the curb (for fiber to the curb) is not generally conside= red a part of the customer premises, but is an excellent place for doing pa= ssive (and some simple active-towards-the-access-network) tests. I disagree= that the Measurement Client needs to be at or in the customer premises. Es= pecially if ANP testing of their own network is part of the goal, or if reg= ulator assessment of the end-to-end access network (which may have choke po= ints further in) is part of the goal. Providing 3rd party control of testin= g within these network elements would be a very hard sell for the operators= of these elements. From an ANP self-measuring and self-testing perspective= , these are definitely Measurement Clients. Barbara From bs7652@att.com Mon Nov 19 10:59:27 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 71A3621F8635 for ; Mon, 19 Nov 2012 10:59:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.597 X-Spam-Level: X-Spam-Status: No, score=-106.597 tagged_above=-999 required=5 tests=[AWL=0.003, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BWmEK+kfFY9g for ; Mon, 19 Nov 2012 10:59:26 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id DB19821F84ED for ; Mon, 19 Nov 2012 10:59:25 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo05.seg.att.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id d818aa05.2aaac421d940.356502.00-580.986391.nbfkord-smmo05.seg.att.com (envelope-from ); Mon, 19 Nov 2012 18:59:25 +0000 (UTC) X-MXL-Hash: 50aa818d2d4d656b-31d62e58c26e241d420ba76c6619ea46a91db2da Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 1818aa05.0.356413.00-223.986124.nbfkord-smmo05.seg.att.com (envelope-from ); Mon, 19 Nov 2012 18:59:15 +0000 (UTC) X-MXL-Hash: 50aa81832876812b-e9fb0f0f5ea8a0ba80afaa01e3ee39dce3957631 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJIxDGL012859; Mon, 19 Nov 2012 13:59:13 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJIwxSF012498 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Nov 2012 13:59:12 -0500 Received: from GAALPA1MSGHUB9F.ITServices.sbc.com (gaalpa1msghub9f.itservices.sbc.com [130.8.36.92]) by sflint03.pst.cso.att.com (RSA Interceptor); Mon, 19 Nov 2012 13:58:41 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9F.ITServices.sbc.com ([130.8.36.92]) with mapi id 14.02.0318.001; Mon, 19 Nov 2012 13:58:41 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Client Thread-Index: AQHNw5+VbNRtG4V7U0+efHOR2EaKFJfxbpXQ Date: Mon, 19 Nov 2012 18:58:40 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD0DF@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.93.19] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=VcpAyiV9 c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=GD_NjPN4rjcA:10 a=Xs6Miaog7CUA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=AQzkKcj-swcA:10 a=3WerFUC7PCsPfpoAjC4A:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Client X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 18:59:27 -0000 [--snip--] > > Here are some specific comments on the architectural elements: > > Measurement client: I agree that if measurements (active or passive) ar= e > > going to be made, then one or more Measurement clients are a mandatory > > element. I agree with the first 3 sentences of the Measurement client > > definition. >=20 > Well 3 out of 4 (sentences) is a good start. :-) >=20 >=20 > > But the idea that *all* Measurement clients must be manageable and > > "provide for communications within different network segments" is not > > something I agree with. >=20 > I'm not sure I detect an implication that all measurement clients must be > manageable. What the fourth sentence says is the following: > "Client measurement functionality must be implementable in a variety of > user contexts ..." As I mentioned, I thought it was possible to agree on the existence o= f a Measurement Client (as defined by the first 3 sentences) as an architec= tural element. The problem really arises with the assumptions as to the nat= ure of this element that are implied by certain of the requirements: A-2: Measurement clients and servers MUST support an extensible set of performance metrics. A-4: Measurement clients MUST be able perform both active and passive measurements. M-3: A measurement client MUST be able to register with the data collection platform automatically, announcing its availability and relevant system parameters. (For example, a cable or DSL modem may indicate its make and model number.) M-4: A measurement client MUST be able to declare what kind of measurements it can perform, e.g., by enumerating a set of measurement identifiers. C-4: A measurement client MUST be able to authenticate and authorize the measurement controller. As I mentioned in my last email, I believe Measurement Clients may exist in= side ONTs, DSLAMs, and even very, very simple DSL modems (some of which are= nothing more than a USB dongle that connects to a PC, and many of which ar= e completely unmanaged, except locally by the end user). The idea that thes= e devices / network elements must support these requirements in order to pa= rticipate in a holistic scheme/architecture to collect meaningful performan= ce data is something I do not agree with. I also believe that it is possibl= e for a Management Client to be directly under the control of the end user,= and potentially not be upgradable or capable of an extensible set of tests= . I believe that the Windows Command Prompt is a very effective (for me) en= d-user-controlled Measurement Client for performing a variety of tests. But= it does not meet the above requirements. Nor should it be expected to. I u= se it often, though, for my on-demand connectivity and performance testing. To me, this architecture draft assumes that a "federated" architecture is n= ecessary for useful collection of performance measurements and conducting t= ests. I disagree with that premise. There also seems to be an assumption th= at a "federated" architecture must be defined prior to defining appropriate= metrics. I disagree. I believe that a federated architecture (that allows = for all sorts of management and control and updating of tests/schedules/etc= .) is unnecessary. I believe it is possible to re-use a lot of what exists,= with minimal additions, to accomplish large scale performance testing and = on-demand diagnostics, that meet the basic "needs" (not to be confused with= "wants") of various parties. I believe it is possible for 3rd parties and = ANPs to develop one-to-one relationships with an end user, to allow for on-= demand and scheduled testing and collection of test data. The end user can = even allow for 3rd parties to access tools under end-user control, which in= teract with ANP Measurement Servers (taking advantage of the end user to AN= P relationship). For large scale performance data -- I believe it is possib= le to collect such data and provide access to the resulting anonymized data= without enabling 3rd party management and control of the elements involved= in data collection. My recommendation would be to focus on defining appropriate metrics. Once w= e understand the metrics, we can work on understanding the appropriate meas= urements and tests, and the set of Measurement Clients and Measurements Ser= vers that could be used to perform those measurements and tests. If somethi= ng else is needed from a standards organization, that should become evident= once the metrics are identified. Barbara From roger@consensii.com Mon Nov 19 13:01:25 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4E0DB21F8709 for ; Mon, 19 Nov 2012 13:01:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.436 X-Spam-Level: X-Spam-Status: No, score=-0.436 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_NET=0.611, HTML_MESSAGE=0.001, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q8W-Blz7-rpi for ; Mon, 19 Nov 2012 13:01:24 -0800 (PST) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [IPv6:2001:558:fe2d:43:76:96:30:16]) by ietfa.amsl.com (Postfix) with ESMTP id AC86D21F8697 for ; Mon, 19 Nov 2012 13:01:24 -0800 (PST) Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74]) by qmta01.emeryville.ca.mail.comcast.net with comcast id RR6U1k0041bwxycA1Z1Ql8; Mon, 19 Nov 2012 21:01:24 +0000 Received: from [192.168.1.32] ([24.9.17.249]) by omta18.emeryville.ca.mail.comcast.net with comcast id RZ1N1k00D5NRaDk8eZ1PNW; Mon, 19 Nov 2012 21:01:24 +0000 From: Roger B. Marks Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: multipart/alternative; boundary="Apple-Mail=_D57E86DB-15C5-4776-B519-3E085CE52309" Date: Mon, 19 Nov 2012 14:01:22 -0700 In-Reply-To: To: lmap@ietf.org References: Message-Id: X-Mailer: Apple Mail (2.1283) Subject: [lmap] Report to LMAP list regarding IEEE Project P802.16.3 X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 21:01:25 -0000 --Apple-Mail=_D57E86DB-15C5-4776-B519-3E085CE52309 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Dear LMAP List, The IEEE 802.16 Working Group on Broadband Wireless Access has prepared = a report to external organizations (including the LMAP list) regarding = IEEE Project P802.16.3. The statement, available at = , provides an update on the = project, which is addressing "Mobile Broadband Network Performance = Measurements." Comments on the project are welcome, particularly regarding the draft = IEEE 802.16.3 Architecture and Requirements. For details, please refer = to the statement and the referenced Call for Contributions. Regards, Roger Marks Roger B. Marks Chair, IEEE 802.16 Working Group on Broadband Wireless Access = --Apple-Mail=_D57E86DB-15C5-4776-B519-3E085CE52309 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Dear LMAP List,

The IEEE 802.16 Working Group on Broadband = Wireless Access has prepared a report to external organizations = (including the LMAP list) regarding IEEE Project = P802.16.3. The statement, available at <http://doc.wirelessman.org/= 16-12-0680>, provides an update on the project, which is = addressing "Mobile Broadband Network Performance = Measurements."

Comments on the project are welcome, particularly = regarding the draft IEEE 802.16.3 Architecture = and Requirements. For details, please refer to the = statement and the referenced Call for = Contributions.

Regards,

Roger = Marks

Roger B. Marks <r.b.marks at = ieee.org>
Chair, IEEE 802.16 Working Group on Broadband Wireless = Access <http://WirelessMAN.org>

= --Apple-Mail=_D57E86DB-15C5-4776-B519-3E085CE52309-- From bs7652@att.com Mon Nov 19 13:53:05 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 26E8D21F8886 for ; Mon, 19 Nov 2012 13:53:05 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.597 X-Spam-Level: X-Spam-Status: No, score=-106.597 tagged_above=-999 required=5 tests=[AWL=0.002, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qXf72mJ-8P7T for ; Mon, 19 Nov 2012 13:53:04 -0800 (PST) Received: from nbfkord-smmo04.seg.att.com (nbfkord-smmo04.seg.att.com [209.65.160.86]) by ietfa.amsl.com (Postfix) with ESMTP id 0F8AF21F8796 for ; Mon, 19 Nov 2012 13:53:04 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo04.seg.att.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 04aaaa05.2aaadfc25940.428935.00-585.1195935.nbfkord-smmo04.seg.att.com (envelope-from ); Mon, 19 Nov 2012 21:53:04 +0000 (UTC) X-MXL-Hash: 50aaaa4015f58cff-8c56092cecf08da8fc96a0f639074c3dedc85888 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo04.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 33aaaa05.0.428820.00-385.1195603.nbfkord-smmo04.seg.att.com (envelope-from ); Mon, 19 Nov 2012 21:52:53 +0000 (UTC) X-MXL-Hash: 50aaaa353cdc37b7-e9eaa7e5600c59dfb6f56d925a77973ac43e2462 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJLqpso007076; Mon, 19 Nov 2012 16:52:51 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJLqbAg006796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Nov 2012 16:52:43 -0500 Received: from GAALPA1MSGHUB9B.ITServices.sbc.com (gaalpa1msghub9b.itservices.sbc.com [130.8.36.88]) by sflint04.pst.cso.att.com (RSA Interceptor); Mon, 19 Nov 2012 16:52:05 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9B.ITServices.sbc.com ([130.8.36.88]) with mapi id 14.02.0318.001; Mon, 19 Nov 2012 16:52:04 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Server Thread-Index: AQHNw6bZBi20s/Wyz0OlsLx+EikGwpfxrjKw Date: Mon, 19 Nov 2012 21:52:04 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD29A@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> <592CA143-D70B-4AA1-AC0C-A69DC8AD30C7@castlepoint.net> In-Reply-To: <592CA143-D70B-4AA1-AC0C-A69DC8AD30C7@castlepoint.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.93.19] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=O7GOSmBW c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=GD_NjPN4rjcA:10 a=Xs6Miaog7CUA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=AQzkKcj-swcA:10 a=4GZKhabjOFyaG-NXdAMA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Server X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 21:53:05 -0000 > > The requirements section assumes the Measurement server includes > > much more functionality than this definition implies. >=20 > Actually, it's funny you should mention this, because Section 6, > Requirements, doesn't have a single requirement tagged with "S-*", which > would presumably apply _specifically_ to the measurement server function. > :-) But, there is this disclaimer, though: > In many cases, > a single requirement governs more than one entity or protocol, so the > labeling should be considered rough. >=20 > Anyway, are the following requirements, the two that you find most > concerning wrt the Measurement Server? > ---snip--- > A-2: Measurement clients and servers MUST support an extensible set > of performance metrics. >=20 > D-3: The data exchange protocol between measurement server and data > collector SHOULD allow the definition of common data elements, > e.g., for network addresses and timestamps. Those and=20 A-6: All entities MUST be able to authenticate the entities they communicate with. D-3: The data exchange protocol between measurement server and data collector SHOULD allow the definition of common data elements, e.g., for network addresses and timestamps. This architecture seems to imply that it's relevant to the end user on-dema= nd test case. My favorite test is "ping". I do my ping testing against a va= riety of Internet websites, but also my access/Internet provider's DNS serv= ers and IP gateways. What I ping depends on what my problem is. Anything th= at responds to my ping is a Measurement Server in my book. As an end user, I have no need for the set of tests that such Measurement S= ervers support. I also have no need for my client to authenticate with them. I don't think = they've ever required authentication of my client. I have no idea if they c= ommunicate with a Data Collector, and I really don't care. Similarly, I often use "bandwidth test" websites for on-demand bandwidth te= sts. I really don't care whether they're extensible, authentication has nev= er been an issue, and I have no requirement for them to communicate with a = Data Collector > ---snip--- > The following requirements are applicable to the overall LMAP Reference > Architecture, as described in this document. How each of these > requirements are applied to specific implementation of a particular type, > category or class of measurement client, server, etc. device is out-of-sc= ope > for this document. > ---snip--- > Is the above text good enough, for now? No. Unless the described architectural elements are changed to be "LM= AP Measurement Client" and "LMAP Measurement Server", with definition of th= ese being that an LMAP Measurement Client/Server is one that meets the requ= irements enumerated in this document. I don't think it's possible to go fro= m the generic architecture elements of Measurement Client/Server to the enu= merated requirements that seem to have something very specific in mind that= is not at all consistent with the generic definitions provided in the arch= itecture section. =20 > But, going back to Section 3.2, Measurement Server, I can see how it may = be > useful to add some of the points you raised above. How about suggesting = to > add something like the following to Section 3.2? > ---snip--- > Measurement servers likely will be owned and operated by a variety of > administrative entities, some of which may be unrelated to the ISP provid= ing > service to a measurement client. Therefore, measurement servers may > need a facility where they can describe their capability to perform one, = or > more, suites of active performance measurement tests to a measurement > controller. In this way, the measurement controller may facilitate > coordination of measurement clients to measurement servers that have > matching active performance measurement capabilities. > ---snip--- That still doesn't allow for a website that responds to a ping to be consid= ered a Measurement Server. Or a website that provides bandwidth measurement= s. But they are Measurement Servers. They just aren't Measurement Servers t= hat meet these "LMAP" requirements. Do any of the use cases create a need f= or all Measurement Servers (used by that use case) to meet these requiremen= ts? I don't think so. Even the 3rd party testing use case should allow for = ping tests without insisting that the server supporting the ping tests meet= these requirements. =20 This LMAP architecture is assuming that a requirement of the architecture i= s to support all use cases, within a single architecture. I disagree with t= hat premise. And I disagree that 3rd party management and control of "LMAP" architecture= elements is a compelling use case. The protocols may be simple, but it's i= ncredibly complex to manage and administer, and to put in all the controls = needed to prevent end users from being negatively impacted by 3rd parties. = A simple requirement of "make sure 3rd parties can't break anything" is ins= ufficient and ignores the fact that ensuring this is incredibly complex and= difficult. Barbara From bs7652@att.com Mon Nov 19 14:13:51 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8E16221F84BB for ; Mon, 19 Nov 2012 14:13:51 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.597 X-Spam-Level: X-Spam-Status: No, score=-106.597 tagged_above=-999 required=5 tests=[AWL=0.002, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bxkQnlwEdclg for ; Mon, 19 Nov 2012 14:13:51 -0800 (PST) Received: from nbfkord-smmo07.seg.att.com (nbfkord-smmo07.seg.att.com [209.65.160.93]) by ietfa.amsl.com (Postfix) with ESMTP id DF36221F84C0 for ; Mon, 19 Nov 2012 14:13:50 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo07.seg.att.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id e1faaa05.5e76b940.446410.00-590.1235084.nbfkord-smmo07.seg.att.com (envelope-from ); Mon, 19 Nov 2012 22:13:50 +0000 (UTC) X-MXL-Hash: 50aaaf1e711443bc-937de51351169ab483cdc032be833b75df604bf6 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo07.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 11faaa05.0.446259.00-392.1234627.nbfkord-smmo07.seg.att.com (envelope-from ); Mon, 19 Nov 2012 22:13:39 +0000 (UTC) X-MXL-Hash: 50aaaf13324422a7-d8754f4eb73fe9040f60d2ad4166c9f1c3a962f8 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJMDZkh027618; Mon, 19 Nov 2012 17:13:37 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAJMDRC4027244 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Nov 2012 17:13:31 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint04.pst.cso.att.com (RSA Interceptor); Mon, 19 Nov 2012 17:12:48 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Mon, 19 Nov 2012 17:12:48 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller Thread-Index: AQHNw6vvfqUxbdgsw0yqTxJeRkNy0pfxug4Q Date: Mon, 19 Nov 2012 22:12:47 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.93.19] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=If4wrxWa c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=GD_NjPN4rjcA:10 a=Xs6Miaog7CUA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=AQzkKcj-swcA:10 a=AVtNoHceVaW6URvP9YEA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2012 22:13:51 -0000 > > Measurement controller: This is not a necessary architectural element. = This > > is an element specific to this proposed physical architecture. >=20 > I'm unclear if your concern is related to: > 1) having _strictly_ a single measurement controller be part of, or in co= ntrol > of a single measurement "panel" (e.g.: group of measurement clients & > measurement server); or, > 2) two, or more, measurement controllers (likely owned/operated by > different parties) acting in concert to coordinate the rendezvous between= an > appropriate panel of measurement clients with measurement servers. See > my previous e-mail re: "Section 3" for an example of what I'm trying to > describe here. >=20 > If neither of those are what concern you, can you point me in the right > direction? :-) Measurement Controllers are not required by all use cases (e.g., end = user on-demand testing that is controlled and managed by the end user, usin= g devices under the direct control and management of the end user, doesn't = call for the existence of a Measurement Controller). I do not find the use case of allowing 3rd party management and control of = test infrastructure to be a compelling use case. IMO, this is not a necessa= ry function of any of the more general use cases for support of end-user sp= ecific on-demand testing and large scale performance measurements. Outside of this 3rd party use case, none of the other use cases really care= about how control is achieved -- it's a function internal to an architectu= re of a single entity. And as I mentioned in my other emails, not all Measu= rement Clients or Measurement Servers need to allow for measurement-specifi= c remote control. We have no clue as to what the desired metrics are, or the recommended meas= urements and tests to achieve those metrics; so assuming a Measurement Cont= roller is needed to coordinate any of these tests is premature. Therefore, = I don't consider it to be a necessary element of an architecture. Barbara From bs7652@att.com Tue Nov 20 06:18:06 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E317821F84C8 for ; Tue, 20 Nov 2012 06:18:06 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ReiB7IF85aly for ; Tue, 20 Nov 2012 06:18:06 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id 0795F21F84BB for ; Tue, 20 Nov 2012 06:18:05 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo08.seg.att.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id e119ba05.5e214940.664887.00-561.1816807.nbfkord-smmo08.seg.att.com (envelope-from ); Tue, 20 Nov 2012 14:18:06 +0000 (UTC) X-MXL-Hash: 50ab911e6a74884a-8434d6c5a3ba30d7271fcb4d0807f4e98bbdcbcb Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 2119ba05.0.664824.00-432.1816614.nbfkord-smmo08.seg.att.com (envelope-from ); Tue, 20 Nov 2012 14:17:57 +0000 (UTC) X-MXL-Hash: 50ab91156eaf6168-6890083048bfded6752f4dc9ef585ae6a662e9b6 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKEHqFX022275; Tue, 20 Nov 2012 09:17:54 -0500 Received: from sflint03.pst.cso.att.com (sflint03.pst.cso.att.com [144.154.234.230]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKEGjJt020722 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 20 Nov 2012 09:17:48 -0500 Received: from GAALPA1MSGHUB9D.ITServices.sbc.com (gaalpa1msghub9d.itservices.sbc.com [130.8.36.90]) by sflint03.pst.cso.att.com (RSA Interceptor); Tue, 20 Nov 2012 09:16:04 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9D.ITServices.sbc.com ([130.8.36.90]) with mapi id 14.02.0318.001; Tue, 20 Nov 2012 09:16:03 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Data Collector & Network Parameter Server Thread-Index: AQHNw69j/TB2TBZlwU28ZwRgpZKV7pfyumeg Date: Tue, 20 Nov 2012 14:16:03 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD6E3@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5CDB4F@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.199.78.119] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=OqD4PVDt c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=XF2aQeIDtRMA:10 a=fMZXka5Qa6EA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=H1VJ9q1ywZkA:10 a=na07ZvPuBV9duIOPAQIA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10 a=Lgov3D3Hji859wIW:21 a=4MTaFE_Na5nKIJ94:21] Cc: Rolf Winter , "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Data Collector & Network Parameter Server X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 14:18:07 -0000 > [--snip--] > > Data collector: I do not see this as a necessary architectural element,= either. > > As with the Measurement controller, it has been assigned the role of > > providing multiple, separable functions: collecting data, storing data,= and > > providing access to the data. >=20 > I'm not sure I understand your concern. Given the resource constraints o= n > some classes of measurement clients, doesn't it make sense for the > measurement client to send it's performance measurement results to "the > cloud" (a.k.a.: data collector) for longer-term archival, storage and cen= tralized > access by the ISP and/or 3rd-parties? For the end user on-demand tests, I don't see that a distinct data co= llector necessarily needs to exist. The Measurement Client could present co= llected data to the end user directly through a local user interface. The e= nd user may or may not want to have the data saved for future use. The end = user may not have a requirement for on-demand test data to be shared with o= thers. Also, since the LMAP Data Collector definition insists that many fun= ctions be combined together (data collection function, data storage functio= n, and providing access to the data function), I do not agree that this com= bined-function Data Collector architectural element, as defined in the draf= t, is necessary to providing functionality that some of the use cases do ca= ll for. I agree that a data collection function, a data storage function, a= nd a data access function are needed by some of the use cases. I believe it= is possible for data collection and storage functions to be internal to a = single entity's management architecture, and am not convinced that deployer= s and implementers of these functions are asking for IETF help in defining = them. =20 > > Network parameter server: I do not see this as a necessary architectura= l > element. If there is a need for an externally-facing function that provid= es > access to access-provider-collected data for the regulatory use case to > include corresponding (anonymous) information related to subscribed line > rate, etc., then that is a requirement on that external-facing function. = If an > access provider wants help in figuring out how to get this information to= the > external-facing function, then I would prefer for that access provider (o= r > their proxy) to ask for such help. >=20 > I would think that if, say, an ISP already has a "network parameter serve= r" > they already own & operate, then that's fine ... they can continue to run= it as > they do today. However, the key aspect we probably want to work on is > defining a _standards-based_ "data model" (e.g.: XML or JSON schema?) > that can encapsulate appropriate/relevant bits of data about subscribed l= ines > in that (existing) Network Parameter Server. This would allow for much > more straightforward access to and parsing of "network parameters" by, fo= r > example, 'authorized parties' as the draft states (Section 3.4). I don't object to data model definition. I do think it may be prematu= re, in the absence of a good understanding of the metrics. Without understa= nding the metrics, I believe the data model would need to be very abstract.= Metrics, to me, could include an expression of "subscribed line rate" that= has consistent meaning across all access technologies. This information is= something I would consider a "network parameter". I would prefer to see IE= TF focus on metrics, rather than a measurement management and control archi= tecture. Metrics can drive a data model that is independent of the architec= ture used to deliver the data. > > If there is a desire to define an interworking function that would allo= w a 3rd > > party to get (not anonymous) customer-specific line information directl= y > > from the access provider (so the 3rd party could correlate it with data= they > > collect from their own Measurement client(s)), then that would be a dis= tinct > > function, as well. Personally, I would shy away from trying to define t= his > > function. But if someone else wants to tackle it, I might be willing to= provide > > comments on it. >=20 > Hrm. You raise a good point above about a 3rd-party acquiring "non- > anonymous" customer line information from the access provider so that the > 3rd-party could correlate it with data they collect from their own (3rd-p= arty) > measurement clients. > 1) I'm not sure what, if any, technical mechanisms the IETF could develo= p to > prevent such a thing from happening, other than to NOT specify any data- > models/schemas that encapsulate such sensitive, non-anonymous > information in a 'standards-based' data-model in the first place. > 2) That said, I would think that "privacy laws" in place in the U.S. and= other > parts of the world (perhaps most significantly, the EU) would prevent the > ISPs from [ever] offering access to that data for fear of violating both = the law > and, more importantly, their customer's trust as well. I'm not aware of any access provider who *wants* to do this. I couldn= 't quite tell if some of the requirements were trying to provide support fo= r this use case -- I could read some of the statements either way. I'm glad= we agree that this is not a use case that we need to support. Barbara From mlinsner@cisco.com Tue Nov 20 06:22:22 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C8AC321F8691 for ; Tue, 20 Nov 2012 06:22:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.25 X-Spam-Level: X-Spam-Status: No, score=-10.25 tagged_above=-999 required=5 tests=[AWL=0.349, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ySj2ADwA0zv6 for ; Tue, 20 Nov 2012 06:22:22 -0800 (PST) Received: from rcdn-iport-6.cisco.com (rcdn-iport-6.cisco.com [173.37.86.77]) by ietfa.amsl.com (Postfix) with ESMTP id 3E12821F86AD for ; Tue, 20 Nov 2012 06:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=2511; q=dns/txt; s=iport; t=1353421341; x=1354630941; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version:content-transfer-encoding; bh=WfVyo+CINvYrZwqc3UQfrSulxkQ3NcdAT5AEVvP0P74=; b=Wyh1yNRHEWR2bmtf/Q7dJieu69ajhiGHENKTJ0L21Vj7PUGodDlglWB6 0NAEJ9sIk9lxtQEvtJykrV8tvZG4Vj0MO4S1yvKYA12QEP4MYmB2WogxX qb933Zeh6iGEfpnkKrrQu+1VR5udis1muIi+w3xnFt4CRACflyWcomZaT g=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av4EAB+Rq1CtJXHB/2dsb2JhbABFwlkWc4IlHR0CATwTCBJ9DgYOiBKwGZBTkSoDiFyNIoVrilaDDQ X-IronPort-AV: E=McAfee;i="5400,1158,6901"; a="144375658" Received: from rcdn-core2-6.cisco.com ([173.37.113.193]) by rcdn-iport-6.cisco.com with ESMTP; 20 Nov 2012 14:22:17 +0000 Received: from [10.116.195.120] (rtp-mlinsner-8717.cisco.com [10.116.195.120]) by rcdn-core2-6.cisco.com (8.14.5/8.14.5) with ESMTP id qAKEMFbq031823; Tue, 20 Nov 2012 14:22:16 GMT User-Agent: Microsoft-MacOutlook/14.2.3.120616 Date: Tue, 20 Nov 2012 09:22:15 -0500 From: Marc Linsner To: "STARK, BARBARA H" Message-ID: Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 14:22:22 -0000 Barbara, > > Measurement Controllers are not required by all use cases (e.g., >end user on-demand testing that is controlled and managed by the end >user, using devices under the direct control and management of the end >user, doesn't call for the existence of a Measurement Controller). >I do not find the use case of allowing 3rd party management and control >of test infrastructure to be a compelling use case. IMO, this is not a >necessary function of any of the more general use cases for support of >end-user specific on-demand testing and large scale performance >measurements. >Outside of this 3rd party use case, none of the other use cases really >care about how control is achieved -- it's a function internal to an >architecture of a single entity. And as I mentioned in my other emails, >not all Measurement Clients or Measurement Servers need to allow for >measurement-specific remote control. >We have no clue as to what the desired metrics are, or the recommended >measurements and tests to achieve those metrics; so assuming a >Measurement Controller is needed to coordinate any of these tests is >premature. Therefore, I don't consider it to be a necessary element of an >architecture. >Barbara It's probably just me, but I'm getting confused about your usage of '3rd party management and control of test infrastructure'. Are you referring to controlling tests run on a client not 'owned' by the entity operating the controller? I'm taking a more simplistic view of the architecture. A measurement client can reside 'anywhere', it's a software entity communicating with the controller via layer 3/4/7 (not layer 2 control plane). IMO, for security reasons, the client will be configured to communicate with a measurement controller (no open ports soliciting a controller). If it is the desire of the entity configuring the client to allow a 3rd party to control the client, that is a perfectly valid use case/scenario. As far as metrics, my assumption is any metric defined by IPPM or related SDO body, with additions as desired. From the plenary presentation I would expect at least the following metrics: Sustained Download, Sustained Upload, Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload, UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under Load, Total Bytes Uploaded. Obviously, extensibility is a key feature. -Marc- From bs7652@att.com Tue Nov 20 07:33:24 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6C7AE21F868D for ; Tue, 20 Nov 2012 07:33:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id boZfW4e3r-AU for ; Tue, 20 Nov 2012 07:33:23 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id 5F13A21F8685 for ; Tue, 20 Nov 2012 07:33:23 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo05.seg.att.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 3c2aba05.2aaae8657940.678281.00-586.1880825.nbfkord-smmo05.seg.att.com (envelope-from ); Tue, 20 Nov 2012 15:33:23 +0000 (UTC) X-MXL-Hash: 50aba2c3054645ab-7d443baff3141624531093a5c3998e24412a1a0f Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 1c2aba05.0.678263.00-235.1880764.nbfkord-smmo05.seg.att.com (envelope-from ); Tue, 20 Nov 2012 15:33:21 +0000 (UTC) X-MXL-Hash: 50aba2c13d9e72fc-8579f9db73b490c48ea580f0fe78e922d1283ec4 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKFXKMv008048; Tue, 20 Nov 2012 10:33:21 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKFXFKm007874 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 20 Nov 2012 10:33:16 -0500 Received: from GAALPA1MSGHUB9D.ITServices.sbc.com (gaalpa1msghub9d.itservices.sbc.com [130.8.36.90]) by sflint04.pst.cso.att.com (RSA Interceptor); Tue, 20 Nov 2012 10:31:50 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9D.ITServices.sbc.com ([130.8.36.90]) with mapi id 14.02.0318.001; Tue, 20 Nov 2012 10:31:37 -0500 From: "STARK, BARBARA H" To: Marc Linsner Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller Thread-Index: AQHNw6vvfqUxbdgsw0yqTxJeRkNy0pfxug4QgAFnTID//7I0EA== Date: Tue, 20 Nov 2012 15:31:36 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.199.78.119] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=VcpAyiV9 c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=XF2aQeIDtRMA:10 a=Xs6Miaog7CUA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=AQzkKcj-swcA:10 a=AUd_NHdVAAAA:8 a=48vgC7mUAAAA:8] X-AnalysisOut: [ a=rkAcL4g0dfB-rfBVgpAA:9 a=CjuIK1q_8ugA:10 a=JfD0Fch1gWkA] X-AnalysisOut: [:10 a=lZB815dzVvQA:10] Cc: "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 15:33:24 -0000 > -----Original Message----- > From: Marc Linsner [mailto:mlinsner@cisco.com] > Sent: Tuesday, November 20, 2012 9:22 AM > To: STARK, BARBARA H > Cc: lmap@ietf.org > Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: > Measurement Controller >=20 > Barbara, >=20 > > > > Measurement Controllers are not required by all use cases (e.g., > >end user on-demand testing that is controlled and managed by the end > >user, using devices under the direct control and management of the end > >user, doesn't call for the existence of a Measurement Controller). > >I do not find the use case of allowing 3rd party management and control > >of test infrastructure to be a compelling use case. IMO, this is not a > >necessary function of any of the more general use cases for support of > >end-user specific on-demand testing and large scale performance > >measurements. > >Outside of this 3rd party use case, none of the other use cases really > >care about how control is achieved -- it's a function internal to an > >architecture of a single entity. And as I mentioned in my other emails, > >not all Measurement Clients or Measurement Servers need to allow for > >measurement-specific remote control. > >We have no clue as to what the desired metrics are, or the recommended > >measurements and tests to achieve those metrics; so assuming a > >Measurement Controller is needed to coordinate any of these tests is > >premature. Therefore, I don't consider it to be a necessary element of > >an architecture. > >Barbara >=20 > It's probably just me, but I'm getting confused about your usage of '3rd = party > management and control of test infrastructure'. Are you referring to > controlling tests run on a client not 'owned' by the entity operating the > controller? Yes. That use case is called out in this draft. =20 > I'm taking a more simplistic view of the architecture. A measurement cli= ent > can reside 'anywhere', it's a software entity communicating with the > controller via layer 3/4/7 (not layer 2 control plane). IMO, for securit= y > reasons, the client will be configured to communicate with a measurement > controller (no open ports soliciting a controller). If it is the desire = of the > entity configuring the client to allow a 3rd party to control the client,= that is a > perfectly valid use case/scenario. I agree that Measurement Clients can reside anywhere. I disagree that= all Measurement Clients must be controlled. Measurement Clients for use by= the end user only do not necessarily require a Measurement Controller. I a= gree with your statement on control configuration with the caveat that it o= nly applies *if* a client is controlled. I agree that 3rd party control is = a valid use case. What I believe I said (in another email) is that I consid= er it to be an incredibly complex use case and not very compelling. By that= I mean that I don't want to spend my time working on it. If others do, tha= t's fine, as long as they aren't surprised and angered by the fact that not= everyone will want to implement and deploy it, and that there may be activ= e resistance to any attempt to mandate it.=20 > As far as metrics, my assumption is any metric defined by IPPM or related > SDO body, with additions as desired. From the plenary presentation I woul= d > expect at least the following metrics: Sustained Download, Sustained Uplo= ad, > Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, > ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload, > UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under > Load, Total Bytes Uploaded. Obviously, extensibility is a key feature. How key is extensibility, how dynamic does extensibility have to be, = and are various approaches to achieving extensibility allowed? Can extensib= ility be achieved by firmware upgrades that happen maybe once a year? Do al= l clients and servers have to be extensible, or is it ok to achieve extensi= bility through the addition of new clients and servers, potentially in new/= different/separate physical devices/elements? Does extensibility imply near= infinite extensibility or is it reasonable for extensibility to be constra= ined by lack of resources available on the device that hosts a client? Barbara From nweaver@icsi.berkeley.edu Tue Nov 20 08:01:35 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B190721F8528 for ; Tue, 20 Nov 2012 08:01:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 160mGcmlYGcc for ; Tue, 20 Nov 2012 08:01:35 -0800 (PST) Received: from rock.ICSI.Berkeley.EDU (rock.ICSI.Berkeley.EDU [192.150.186.19]) by ietfa.amsl.com (Postfix) with ESMTP id E603721F84E7 for ; Tue, 20 Nov 2012 08:01:34 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id 488802C4018; Tue, 20 Nov 2012 08:01:34 -0800 (PST) X-Virus-Scanned: amavisd-new at ICSI.Berkeley.EDU Received: from rock.ICSI.Berkeley.EDU ([127.0.0.1]) by localhost (maihub.ICSI.Berkeley.EDU [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Xy0xRjWRs-YI; Tue, 20 Nov 2012 08:01:33 -0800 (PST) Received: from gala.icir.org (gala [192.150.187.49]) (Authenticated sender: nweaver) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id D1CAB2C4015; Tue, 20 Nov 2012 08:01:33 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Nicholas Weaver In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Tue, 20 Nov 2012 08:01:33 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1283) Cc: Marc Linsner , Nicholas Weaver , "lmap@ietf.org" Subject: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:01:35 -0000 On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote: >> -----Original Message----- >> From: Marc Linsner [mailto:mlinsner@cisco.com] >> As far as metrics, my assumption is any metric defined by IPPM or = related >> SDO body, with additions as desired. =46rom the plenary presentation = I would >> expect at least the following metrics: Sustained Download, Sustained = Upload, >> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, >> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst = Upload, >> UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under >> Load, Total Bytes Uploaded. Obviously, extensibility is a key = feature. >=20 > How key is extensibility, how dynamic does extensibility have to = be, and are various approaches to achieving extensibility allowed? Can = extensibility be achieved by firmware upgrades that happen maybe once a = year? Do all clients and servers have to be extensible, or is it ok to = achieve extensibility through the addition of new clients and servers, = potentially in new/different/separate physical devices/elements? Does = extensibility imply near infinite extensibility or is it reasonable for = extensibility to be constrained by lack of resources available on the = device that hosts a client? > Barbara I've said it once, I'll say it one final time: This focus on architecture is deeply misguided. As someone who's built = one such architecture (Netalyzr), working on two others (Fathom, a = browser extension, and techniques using GuruPlugs/Raspberry Pis), the = architecture is the easy part. In all this work, having some "standard" way to communicate between my = measurement clients and servers is useless. I just use HTTP, HTTPS, or = SSH depending on whether quick & easy or server side or mutual = authentication is required, with whatever load-balancing is needed = handled by random assignment of remote test node. =20 The internal data store is whatever is appropriate for the project (eg. = for Netalyzr its python pickled objects for storage on our web server, = data in a Postgress database for analysis). =20 Updates are handled appropriately (Netalyzr is updateless, its an applet = so its always up-to-date. Our android version will just use the app = store update mechanism. Fathom we use Firefox's update mechanism. The = plugs are a hack using SSH, but its a fair amount of custom goop, and = they update nightly). =20 And all three end up having rather different architectural requirements. The hard part is developing the test metrics and measurements, both in = the abstract and within the API constraints of the target platform. = E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, = so therefore I can just "Run Netalyzr" for all Netalyzr tests. But the = same isn't the case for the Ripe Atlas or Bismark platform. And Marc's list is actually just a start: you also have generic proxy = presence & location detection, application specific proxy detection and = analysis, packet injection detection, differential application behavior = detection, application agnostic traffic shaping detection, DNSSEC = support, DNS manipulation detection, path MTU, IPv6 for all of the = previous, and a ton of others. Netalyzr is currently composed of >100 = distinct tests. And this amount just continues to grow. If the IETF actually wants to have an impact in this area, don't bother = focusing on protocols for communication, focus on tests with given = attributes. Because people like me who actually build these things are = going to ignore an architectural focus because that has no value. But = having ACCEPTED STANDARD tests would be very valuable. =20 E.g. a good, standard latency-under-load test that can be written in = Flash (so TCP/UDP connections only to cooperating servers) would be = great. Our current one in Netalyzr is only OK: we believe that one = could do a lot better. If you could then figure out WHERE the bottleneck is (using a server = that can also do simultaneous traceroutes), this would be even more = valuable. From mlinsner@cisco.com Tue Nov 20 08:09:11 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 49B5E21F861E for ; Tue, 20 Nov 2012 08:09:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.32 X-Spam-Level: X-Spam-Status: No, score=-10.32 tagged_above=-999 required=5 tests=[AWL=0.279, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UhzRcnKeCFwe for ; Tue, 20 Nov 2012 08:09:10 -0800 (PST) Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) by ietfa.amsl.com (Postfix) with ESMTP id 53C7C21F85CB for ; Tue, 20 Nov 2012 08:09:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5129; q=dns/txt; s=iport; t=1353427750; x=1354637350; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version:content-transfer-encoding; bh=qT8WxQQrBacOCNhnTDhN/k9iU6AJefpINneH5G950f4=; b=I/dDsZwZlE+bBSDScynQkiFIi4o4w3r2qrpak7Pl0Stt4kCQuH5u5Z0N xwGv0gJ3r1WOrobxbCFV5vctiSoqGVcfBYGWnxxfobpfKFoBNtyPna2la 3nJsS5om9S4LQdu8NCvoSQ1anmipgJHTOiEaZt/tBjQVVcn2U6bft3r/v o=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av4EAPmpq1CtJV2c/2dsb2JhbABFwlkWc4IlHR0CATwTCBJ9DgYOiBKwFpBMjFyETgOIXI0ihWuKVoMNgT8HFw X-IronPort-AV: E=McAfee;i="5400,1158,6901"; a="144394284" Received: from rcdn-core-5.cisco.com ([173.37.93.156]) by rcdn-iport-2.cisco.com with ESMTP; 20 Nov 2012 16:09:09 +0000 Received: from [10.116.195.120] (rtp-mlinsner-8717.cisco.com [10.116.195.120]) by rcdn-core-5.cisco.com (8.14.5/8.14.5) with ESMTP id qAKG974h001427; Tue, 20 Nov 2012 16:09:08 GMT User-Agent: Microsoft-MacOutlook/14.2.3.120616 Date: Tue, 20 Nov 2012 11:09:06 -0500 From: Marc Linsner To: "STARK, BARBARA H" Message-ID: Thread-Topic: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: "lmap@ietf.org" Subject: Re: [lmap] comments on draft-schulzrinne-lmap-requirements: Measurement Controller X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:09:11 -0000 >> >> > >> > Measurement Controllers are not required by all use cases (e.g., >> >end user on-demand testing that is controlled and managed by the end >> >user, using devices under the direct control and management of the end >> >user, doesn't call for the existence of a Measurement Controller). >> >I do not find the use case of allowing 3rd party management and control >> >of test infrastructure to be a compelling use case. IMO, this is not a >> >necessary function of any of the more general use cases for support of >> >end-user specific on-demand testing and large scale performance >> >measurements. >> >Outside of this 3rd party use case, none of the other use cases really >> >care about how control is achieved -- it's a function internal to an >> >architecture of a single entity. And as I mentioned in my other emails, >> >not all Measurement Clients or Measurement Servers need to allow for >> >measurement-specific remote control. >> >We have no clue as to what the desired metrics are, or the recommended >> >measurements and tests to achieve those metrics; so assuming a >> >Measurement Controller is needed to coordinate any of these tests is >> >premature. Therefore, I don't consider it to be a necessary element of >> >an architecture. >> >Barbara >> >> It's probably just me, but I'm getting confused about your usage of >>'3rd party >> management and control of test infrastructure'. Are you referring to >> controlling tests run on a client not 'owned' by the entity operating >>the >> controller? > > Yes. That use case is called out in this draft. > >> I'm taking a more simplistic view of the architecture. A measurement >>client >> can reside 'anywhere', it's a software entity communicating with the >> controller via layer 3/4/7 (not layer 2 control plane). IMO, for >>security >> reasons, the client will be configured to communicate with a measurement >> controller (no open ports soliciting a controller). If it is the >>desire of the >> entity configuring the client to allow a 3rd party to control the >>client, that is a >> perfectly valid use case/scenario. > > I agree that Measurement Clients can reside anywhere. I disagree >that all Measurement Clients must be controlled. Measurement Clients for >use by the end user only do not necessarily require a Measurement >Controller. I agree with your statement on control configuration with the >caveat that it only applies *if* a client is controlled. I agree that 3rd >party control is a valid use case. What I believe I said (in another >email) is that I consider it to be an incredibly complex use case and not >very compelling. By that I mean that I don't want to spend my time >working on it. If others do, that's fine, as long as they aren't >surprised and angered by the fact that not everyone will want to >implement and deploy it, and that there may be active resistance to any >attempt to mandate it. IMO, remote management of a measurement client is a feature, but in the context of LMAP, it is required. Nothing has been stated that client local UI will not be able to invoke any/all tests, I believe that is a client implementor decision. I am curious why you believe 3rd party control "to be an incredibly complex use case and not very compelling". Outside of the obvious security considerations, then from a protocol perspective, what makes 3rd party control any different from 1st party control? > > >> As far as metrics, my assumption is any metric defined by IPPM or >>related >> SDO body, with additions as desired. From the plenary presentation I >>would >> expect at least the following metrics: Sustained Download, Sustained >>Upload, >> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, >> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload, >> UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under >> Load, Total Bytes Uploaded. Obviously, extensibility is a key feature. > > How key is extensibility, how dynamic does extensibility have to >be, and are various approaches to achieving extensibility allowed? Can >extensibility be achieved by firmware upgrades that happen maybe once a >year? Do all clients and servers have to be extensible, or is it ok to >achieve extensibility through the addition of new clients and servers, >potentially in new/different/separate physical devices/elements? Does >extensibility imply near infinite extensibility or is it reasonable for >extensibility to be constrained by lack of resources available on the >device that hosts a client? >Barbara I'm not sure your concerns wrt extensibility are a LMAP issue. If LMAP does their job correctly, the controller-client protocol will be extensible. Whether/when a v1 client/controller is upgradable to v2 can't be dictated/demanded by LMAP/IETF. With that said, it would behove LMAP to include any/all known/reasonable use cases in v1, and at a future time, v2 taking into consideration v1 capabilities/backward compatibility. -Marc- From mlinsner@cisco.com Tue Nov 20 08:18:24 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BB6DA21F876F for ; Tue, 20 Nov 2012 08:18:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.366 X-Spam-Level: X-Spam-Status: No, score=-10.366 tagged_above=-999 required=5 tests=[AWL=0.233, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4js7zQE4s1la for ; Tue, 20 Nov 2012 08:18:24 -0800 (PST) Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id EBB1B21F8753 for ; Tue, 20 Nov 2012 08:18:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=4849; q=dns/txt; s=iport; t=1353428304; x=1354637904; h=date:subject:from:to:cc:message-id:in-reply-to: mime-version:content-transfer-encoding; bh=u+CAekNjXTgZmfRAyJh/YOCTOEVRJsvnS9kKcZxeVJk=; b=BG2SnB2+KHNFHND3hmYfjSAqRiJ6jYz2uZWl3TyL0OJ3zR/VzFC5I5Wb 1MQvqu5rPC03ffMvPX5CJF1EZ+4R+JpNcVae22xi68ECO0oRXSqTJ7za6 EbyhAxL2xBgRQTyp2J/cM/udn/8KvWAx55+WDj8J5WyoRtLGjvlt1dE/u w=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av8EANGsq1CtJV2b/2dsb2JhbABFDsJLFnOCHgEBAQQ6AgE8DAcIDgMBAgECASdNAwYIBg4FiA2wHJBLjDUgB4ROA4hcjSKFa4pWgjJbgT4BHg X-IronPort-AV: E=McAfee;i="5400,1158,6901"; a="144417953" Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-7.cisco.com with ESMTP; 20 Nov 2012 16:18:23 +0000 Received: from [10.116.195.120] (rtp-mlinsner-8717.cisco.com [10.116.195.120]) by rcdn-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id qAKGIL4N023382; Tue, 20 Nov 2012 16:18:22 GMT User-Agent: Microsoft-MacOutlook/14.2.3.120616 Date: Tue, 20 Nov 2012 11:18:21 -0500 From: Marc Linsner To: Nicholas Weaver Message-ID: Thread-Topic: Focus on Tests, Not Architectures... In-Reply-To: <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: "lmap@ietf.org" Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:18:24 -0000 Nicholas, Thanks for jumping in. I believe most in the IETF rely on the IPPM wg to develop the test metrics. LMAP is focused on client/controller management issues. I'm curious how you propose vendor A's controller to work with vendor B's client if the communication protocol/architecture isn't written down for all to see? -Marc- -----Original Message----- From: Nicholas Weaver Date: Tuesday, November 20, 2012 11:01 AM To: "STARK, BARBARA H" Cc: Nicholas Weaver , Marc Linsner , "lmap@ietf.org" Subject: Focus on Tests, Not Architectures... > >On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote: >>> -----Original Message----- >>> From: Marc Linsner [mailto:mlinsner@cisco.com] >>> As far as metrics, my assumption is any metric defined by IPPM or >>>related >>> SDO body, with additions as desired. From the plenary presentation I >>>would >>> expect at least the following metrics: Sustained Download, Sustained >>>Upload, >>> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, >>> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload, >>> UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under >>> Load, Total Bytes Uploaded. Obviously, extensibility is a key feature. >> >> How key is extensibility, how dynamic does extensibility have to >>be, and are various approaches to achieving extensibility allowed? Can >>extensibility be achieved by firmware upgrades that happen maybe once a >>year? Do all clients and servers have to be extensible, or is it ok to >>achieve extensibility through the addition of new clients and servers, >>potentially in new/different/separate physical devices/elements? Does >>extensibility imply near infinite extensibility or is it reasonable for >>extensibility to be constrained by lack of resources available on the >>device that hosts a client? >> Barbara > >I've said it once, I'll say it one final time: > >This focus on architecture is deeply misguided. As someone who's built >one such architecture (Netalyzr), working on two others (Fathom, a >browser extension, and techniques using GuruPlugs/Raspberry Pis), the >architecture is the easy part. > >In all this work, having some "standard" way to communicate between my >measurement clients and servers is useless. I just use HTTP, HTTPS, or >SSH depending on whether quick & easy or server side or mutual >authentication is required, with whatever load-balancing is needed >handled by random assignment of remote test node. > >The internal data store is whatever is appropriate for the project (eg. >for Netalyzr its python pickled objects for storage on our web server, >data in a Postgress database for analysis). > >Updates are handled appropriately (Netalyzr is updateless, its an applet >so its always up-to-date. Our android version will just use the app >store update mechanism. Fathom we use Firefox's update mechanism. The >plugs are a hack using SSH, but its a fair amount of custom goop, and >they update nightly). > >And all three end up having rather different architectural requirements. > > > >The hard part is developing the test metrics and measurements, both in >the abstract and within the API constraints of the target platform. E.g. >why I like the GuruPlugs and Raspberry Pi is that I can run Java, so >therefore I can just "Run Netalyzr" for all Netalyzr tests. But the same >isn't the case for the Ripe Atlas or Bismark platform. > >And Marc's list is actually just a start: you also have generic proxy >presence & location detection, application specific proxy detection and >analysis, packet injection detection, differential application behavior >detection, application agnostic traffic shaping detection, DNSSEC >support, DNS manipulation detection, path MTU, IPv6 for all of the >previous, and a ton of others. Netalyzr is currently composed of >100 >distinct tests. And this amount just continues to grow. > > >If the IETF actually wants to have an impact in this area, don't bother >focusing on protocols for communication, focus on tests with given >attributes. Because people like me who actually build these things are >going to ignore an architectural focus because that has no value. But >having ACCEPTED STANDARD tests would be very valuable. > >E.g. a good, standard latency-under-load test that can be written in >Flash (so TCP/UDP connections only to cooperating servers) would be >great. Our current one in Netalyzr is only OK: we believe that one could >do a lot better. > >If you could then figure out WHERE the bottleneck is (using a server that >can also do simultaneous traceroutes), this would be even more valuable. > From nweaver@ICSI.Berkeley.EDU Tue Nov 20 08:47:41 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EE2B221F87AF for ; Tue, 20 Nov 2012 08:47:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zMAvD6uQupg1 for ; Tue, 20 Nov 2012 08:47:40 -0800 (PST) Received: from rock.ICSI.Berkeley.EDU (rock.ICSI.Berkeley.EDU [192.150.186.19]) by ietfa.amsl.com (Postfix) with ESMTP id 6166D21F87AC for ; Tue, 20 Nov 2012 08:47:40 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id 20D682C4019; Tue, 20 Nov 2012 08:47:40 -0800 (PST) X-Virus-Scanned: amavisd-new at ICSI.Berkeley.EDU Received: from rock.ICSI.Berkeley.EDU ([127.0.0.1]) by localhost (maihub.ICSI.Berkeley.EDU [127.0.0.1]) (amavisd-new, port 10024) with LMTP id ceg-ADjqB3wr; Tue, 20 Nov 2012 08:47:39 -0800 (PST) Received: from gala.icir.org (gala [192.150.187.49]) (Authenticated sender: nweaver) by rock.ICSI.Berkeley.EDU (Postfix) with ESMTP id BE0972C4015; Tue, 20 Nov 2012 08:47:39 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Nicholas Weaver In-Reply-To: Date: Tue, 20 Nov 2012 08:47:39 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <2DB384AB-8A39-4FFD-9B0B-11E0CE5562C5@ICSI.Berkeley.EDU> References: To: Marc Linsner X-Mailer: Apple Mail (2.1283) Cc: Nicholas Weaver , "lmap@ietf.org" Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:47:41 -0000 On Nov 20, 2012, at 8:18 AM, Marc Linsner wrote: > Nicholas, >=20 > Thanks for jumping in. I believe most in the IETF rely on the IPPM wg = to > develop the test metrics. LMAP is focused on client/controller = management > issues. >=20 > I'm curious how you propose vendor A's controller to work with vendor = B's > client if the communication protocol/architecture isn't written down = for > all to see? >=20 > -Marc- Its not a problem: Vendor A's controller with Vendor B's client can = only talk to each other anyway if they are running the same test. Which = is often insanely test specific what they need to talk about: you aren't = going to be able to standardize that away in any generic fashion. Then for the protocol to communicate results & control, just use = HTTP/HTTPS/SSH (pick your transit protocol) and JSON/XML/HTML (pick your = data framing) which meets your requirements for convenience and = authentication. E.g. we already have this for Netalyzr. =20 If you run the Netalyzr command-line client, you can get the results in = a nice JSON blob through HTTP from our server when everything is said = and done, but that the JSON blob's content and structure is Netalyzr = specific is not a surprise, it needs to be Netalyzr specific. = http://n1.netalyzr.icsi.berkeley.edu/json/id=3D43ca253f-20573-9eafe28f-27f= a-4bb9-b262 As you can see, we chose HTTP (no need for authentication) and JSON = (easier to parse & unparse), and we don't have all the server side = rendering in the JSON representation (yet) but all client-side data is = included. If someone wants a JSON rendering for a server side result we = don't yet do, tell us and we will add it in. This creates a nice standard way for someone else to communicate and = fetch results from our service, and they can use our binary client as a = black box (since actually implementing all of the Netalyzr test suite = would be a big endeavor, and pointless if your client can run Java). Similarly, we have a visiting student building a really cool = proxy-traceroute service (where does a proxy exist between you and the = Internet, measured from the Internet side, on a specific port?). =20 Getting the results will be just an HTTP fetch, with a specific HTML = framing. In this case we chose HTML so its both human readable and = likely to be (relatively) unmangled at least in semantics. From trammell@tik.ee.ethz.ch Tue Nov 20 08:59:28 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3589721F878C; Tue, 20 Nov 2012 08:59:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.716 X-Spam-Level: X-Spam-Status: No, score=-6.716 tagged_above=-999 required=5 tests=[AWL=-0.117, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7zTIN0IRuHeO; Tue, 20 Nov 2012 08:59:27 -0800 (PST) Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) by ietfa.amsl.com (Postfix) with ESMTP id 2FBBA21F877B; Tue, 20 Nov 2012 08:59:27 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id E96FAD9309; Tue, 20 Nov 2012 17:59:25 +0100 (MET) X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 7FcRR0nuFUNV; Tue, 20 Nov 2012 17:59:25 +0100 (MET) Received: from pb-10243.ethz.ch (pb-10243.ethz.ch [82.130.102.152]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: briant) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id B6AAED9307; Tue, 20 Nov 2012 17:59:25 +0100 (MET) Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Brian Trammell In-Reply-To: <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> Date: Tue, 20 Nov 2012 17:59:25 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <887F51A4-05F2-44F1-8F29-5F49D35DF447@tik.ee.ethz.ch> References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> To: Nicholas Weaver X-Mailer: Apple Mail (2.1283) Cc: ippm@ietf.org, lmap@ietf.org Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:59:28 -0000 Hi, Nick, (Copying this over to IPPM as well, as the focus-on-metrics discussion = will be interesting there too). On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote: > I've said it once, I'll say it one final time: >=20 > This focus on architecture is deeply misguided. As someone who's = built one such architecture (Netalyzr), working on two others (Fathom, a = browser extension, and techniques using GuruPlugs/Raspberry Pis), the = architecture is the easy part. >=20 > In all this work, having some "standard" way to communicate between my = measurement clients and servers is useless. I just use HTTP, HTTPS, or = SSH depending on whether quick & easy or server side or mutual = authentication is required, with whatever load-balancing is needed = handled by random assignment of remote test node. =20 >=20 > The internal data store is whatever is appropriate for the project = (eg. for Netalyzr its python pickled objects for storage on our web = server, data in a Postgress database for analysis). =20 >=20 > Updates are handled appropriately (Netalyzr is updateless, its an = applet so its always up-to-date. Our android version will just use the = app store update mechanism. Fathom we use Firefox's update mechanism. = The plugs are a hack using SSH, but its a fair amount of custom goop, = and they update nightly). =20 >=20 > And all three end up having rather different architectural = requirements. Architectures, I would submit (and as I've said here before), are useful = to hang terminology off of: you need boxes and lines so everyone = understands (roughly) the same thing when you say "client" and "server" = and "controller" with reference to a set of measurements you define. But yes, specifying a way to specify a measurement is largely arbitrary. = I'd even advocate separating the representation from the transport, = especially as the general problem of measurement interoperability has = data sources operating and wildly different scales, but as you say = deciding this is the easy part. > The hard part is developing the test metrics and measurements, both in = the abstract and within the API constraints of the target platform. = E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, = so therefore I can just "Run Netalyzr" for all Netalyzr tests. But the = same isn't the case for the Ripe Atlas or Bismark platform. >=20 > And Marc's list is actually just a start: you also have generic proxy = presence & location detection, application specific proxy detection and = analysis, packet injection detection, differential application behavior = detection, application agnostic traffic shaping detection, DNSSEC = support, DNS manipulation detection, path MTU, IPv6 for all of the = previous, and a ton of others. Netalyzr is currently composed of >100 = distinct tests. And this amount just continues to grow. >=20 >=20 > If the IETF actually wants to have an impact in this area, don't = bother focusing on protocols for communication, focus on tests with = given attributes. Because people like me who actually build these = things are going to ignore an architectural focus because that has no = value. But having ACCEPTED STANDARD tests would be very valuable. =20 + 1e. Stated another way, the protocol gives you the = grammar. What you need to make this at all interoperable is the = vocabulary.=20 IPPM has defined a several of these at a low level, with a focus on = active measurement methods that produce comparable results. There = appears to be interest in IPPM both in adding passive methods for the = existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding = definitions of more complex metrics based in part on the simple ones = (see draft-mathis-ippm-model-based-metrics). > E.g. a good, standard latency-under-load test that can be written in = Flash (so TCP/UDP connections only to cooperating servers) would be = great. Our current one in Netalyzr is only OK: we believe that one = could do a lot better. >=20 > If you could then figure out WHERE the bottleneck is (using a server = that can also do simultaneous traceroutes), this would be even more = valuable. (This seems to make the assumption that traceroute-measurable latency is = strongly correlated with application latency at a given hop, but I get = your point...) The efforts of IPPM to date have been focused on simply describable, = easily repeatable metrics... there's not a lot of specification of = arbitrarily complex or unknown preconditions (as would be useful in the = latency-under-load example you give), because these tend to be counter = to the goals of maximizing repeatability and minimizing uncertainty. = There's also a focus on active measurement for the same reason. On the = other hand, a lot of work in distributed measurement systems is focused = on measuring the network as it is, with implicit or explicit passive = components (again, returning to "latency under load", the load traffic = will be made up of generated as well as existing background load).=20 So, what might be useful would be to open another line of work within = IPPM on interoperable descriptions of some of these existing tests as = empirically specified metrics, perhaps with relaxed compliance with the = IPPM framework as appropriate. This could be used to provide a more = complex vocabulary to LMAP, but could also be useful independently -- if = LMAP is eventually defined to apply only to a restricted set of use = cases (i.e. specifically mandated tests for regulatory compliance, but = not research or troubleshooting), comparable metrics with incompatible = architecture and protocols could still lead to wider interoperability in = the measurement space, the need for protocol shims (which are after all = easy to write if the number inside has the same meaning) aside. Best regards, Brian= From bs7652@att.com Tue Nov 20 09:07:53 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DDA9121F87BA for ; Tue, 20 Nov 2012 09:07:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1hUNGl75UKQH for ; Tue, 20 Nov 2012 09:07:53 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 4590221F87EF for ; Tue, 20 Nov 2012 09:07:53 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 9e8bba05.79645940.731848.00-561.2018605.nbfkord-smmo06.seg.att.com (envelope-from ); Tue, 20 Nov 2012 17:07:53 +0000 (UTC) X-MXL-Hash: 50abb8e922a9c2c0-8237aee0550ca956b6dc0f4ec718ec2f0ed20cd1 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id dd8bba05.0.731685.00-453.2018138.nbfkord-smmo06.seg.att.com (envelope-from ); Tue, 20 Nov 2012 17:07:42 +0000 (UTC) X-MXL-Hash: 50abb8de613fbf18-f10e3566226818d49df72fd6feeb51c97d7713ed Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKH7f9T028660; Tue, 20 Nov 2012 12:07:41 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAKH7Rup028359 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 20 Nov 2012 12:07:37 -0500 Received: from GAALPA1MSGHUB9A.ITServices.sbc.com (gaalpa1msghub9a.itservices.sbc.com [130.8.36.87]) by sflint04.pst.cso.att.com (RSA Interceptor); Tue, 20 Nov 2012 12:02:55 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9A.ITServices.sbc.com ([130.8.36.87]) with mapi id 14.02.0318.001; Tue, 20 Nov 2012 12:02:45 -0500 From: "STARK, BARBARA H" To: Nicholas Weaver Thread-Topic: Focus on Tests, Not Architectures... Thread-Index: AQHNxzhY10HdXQT4IUaXrZyP29lPp5fy8O9A Date: Tue, 20 Nov 2012 17:02:44 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5DD96D@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> In-Reply-To: <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.199.78.119] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=MZfbTeDf c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=XF2aQeIDtRMA:10 a=XV3CA8akORcA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=h5VdO6nhOuIA:10 a=j6sCAVXa16JJeV30_t8A:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: Marc Linsner , "lmap@ietf.org" Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 17:07:54 -0000 > If the IETF actually wants to have an impact in this area, don't bother f= ocusing > on protocols for communication, focus on tests with given attributes. > Because people like me who actually build these things are going to ignor= e an > architectural focus because that has no value. But having ACCEPTED > STANDARD tests would be very valuable. +1 There are clients that will support control via OMA-DM, clients that will s= upport control via TR-069, clients that will support control by SNMP, clien= ts that will support control by some yet-to-be-defined protocol, clients wh= ere client and controller vendor are the same and the protocol is proprieta= ry, etc. It's generally a good idea to know how to control a controllable c= lient before deploying it. Since client and controller are deployed by the same entity, it should be u= p to that entity to figure out their control architecture, and that entity = should have full say in how to do their own control architecture. Is there any entity asking for help from IETF in figuring out what they wan= t to internally deploy? Barbara From philip.eardley@bt.com Tue Nov 20 09:09:14 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 30F8C21F880B for ; Tue, 20 Nov 2012 09:09:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -103.599 X-Spam-Level: X-Spam-Status: No, score=-103.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R1Oua2QU0wbc for ; Tue, 20 Nov 2012 09:09:13 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp64.intersmtp.com [62.239.224.237]) by ietfa.amsl.com (Postfix) with ESMTP id 1BA9521F8803 for ; Tue, 20 Nov 2012 09:09:11 -0800 (PST) Received: from EVMHT65-UKRD.domain1.systemhost.net (10.36.3.102) by RDW083A008ED64.smtp-e4.hygiene.service (10.187.98.13) with Microsoft SMTP Server (TLS) id 8.3.279.1; Tue, 20 Nov 2012 17:09:07 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.2.94]) by EVMHT65-UKRD.domain1.systemhost.net ([10.36.3.102]) with mapi; Tue, 20 Nov 2012 17:09:07 +0000 From: To: , Date: Tue, 20 Nov 2012 17:09:06 +0000 Thread-Topic: [lmap] Focus on Tests, Not Architectures... Thread-Index: Ac3HOFRoT3fQrYMZQze8d4V0AXsctwACF+7w Message-ID: <9510D26531EF184D9017DF24659BB87F33F3644337@EMV65-UKRD.domain1.systemhost.net> References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> In-Reply-To: <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: mlinsner@cisco.com, lmap@ietf.org Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 17:09:14 -0000 agree with initial focus on tests. even if you think architecture(s) should be in scope, I think it's better t= o start by defining the work that everyone agrees would be valuable to do. = Otherwise we'll end up not starting the metrics/tests work until we've done= a bunch of architecture work! Perhaps once the tests/metrics work is defin= ed, we can work out whether there is any new protocol needed (for reporting= results or initiating tests say) phil -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of Nic= holas Weaver Sent: 20 November 2012 16:02 To: STARK, BARBARA H Cc: Marc Linsner; Nicholas Weaver; lmap@ietf.org Subject: [lmap] Focus on Tests, Not Architectures... On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote: >> -----Original Message----- >> From: Marc Linsner [mailto:mlinsner@cisco.com] As far as metrics, my=20 >> assumption is any metric defined by IPPM or related SDO body, with=20 >> additions as desired. From the plenary presentation I would expect at=20 >> least the following metrics: Sustained Download, Sustained Upload,=20 >> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures,=20 >> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst=20 >> Upload, UDP Latency, Video Streaming Measure, DNS Resolution, Latency=20 >> Under Load, Total Bytes Uploaded. Obviously, extensibility is a key fea= ture. >=20 > How key is extensibility, how dynamic does extensibility have to be= , and are various approaches to achieving extensibility allowed? Can extens= ibility be achieved by firmware upgrades that happen maybe once a year? Do = all clients and servers have to be extensible, or is it ok to achieve exten= sibility through the addition of new clients and servers, potentially in ne= w/different/separate physical devices/elements? Does extensibility imply ne= ar infinite extensibility or is it reasonable for extensibility to be const= rained by lack of resources available on the device that hosts a client? > Barbara I've said it once, I'll say it one final time: This focus on architecture is deeply misguided. As someone who's built one= such architecture (Netalyzr), working on two others (Fathom, a browser ext= ension, and techniques using GuruPlugs/Raspberry Pis), the architecture is = the easy part. In all this work, having some "standard" way to communicate between my meas= urement clients and servers is useless. I just use HTTP, HTTPS, or SSH dep= ending on whether quick & easy or server side or mutual authentication is r= equired, with whatever load-balancing is needed handled by random assignmen= t of remote test node. =20 The internal data store is whatever is appropriate for the project (eg. for= Netalyzr its python pickled objects for storage on our web server, data in= a Postgress database for analysis). =20 Updates are handled appropriately (Netalyzr is updateless, its an applet so= its always up-to-date. Our android version will just use the app store up= date mechanism. Fathom we use Firefox's update mechanism. The plugs are a= hack using SSH, but its a fair amount of custom goop, and they update nigh= tly). =20 And all three end up having rather different architectural requirements. The hard part is developing the test metrics and measurements, both in the = abstract and within the API constraints of the target platform. E.g. why I= like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I= can just "Run Netalyzr" for all Netalyzr tests. But the same isn't the ca= se for the Ripe Atlas or Bismark platform. And Marc's list is actually just a start: you also have generic proxy prese= nce & location detection, application specific proxy detection and analysis= , packet injection detection, differential application behavior detection, = application agnostic traffic shaping detection, DNSSEC support, DNS manipul= ation detection, path MTU, IPv6 for all of the previous, and a ton of other= s. Netalyzr is currently composed of >100 distinct tests. And this amount= just continues to grow. If the IETF actually wants to have an impact in this area, don't bother foc= using on protocols for communication, focus on tests with given attributes.= Because people like me who actually build these things are going to ignor= e an architectural focus because that has no value. But having ACCEPTED ST= ANDARD tests would be very valuable. =20 E.g. a good, standard latency-under-load test that can be written in Flash = (so TCP/UDP connections only to cooperating servers) would be great. Our c= urrent one in Netalyzr is only OK: we believe that one could do a lot bette= r. If you could then figure out WHERE the bottleneck is (using a server that c= an also do simultaneous traceroutes), this would be even more valuable. _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From ken.ko@adtran.com Tue Nov 20 09:53:57 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AB65B21F86D2; Tue, 20 Nov 2012 09:53:57 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lGTkU847y4u2; Tue, 20 Nov 2012 09:53:55 -0800 (PST) Received: from p02c12o147.mxlogic.net (p02c12o147.mxlogic.net [208.65.145.80]) by ietfa.amsl.com (Postfix) with ESMTP id 3F8FF21F8685; Tue, 20 Nov 2012 09:53:50 -0800 (PST) Received: from unknown [76.164.174.81] (EHLO p02c12o147.mxlogic.net) by p02c12o147.mxlogic.net(mxl_mta-6.16.0-0) with ESMTP id fa3cba05.7b256940.130268.00-2464.322329.p02c12o147.mxlogic.net (envelope-from ); Tue, 20 Nov 2012 10:53:51 -0700 (MST) X-MXL-Hash: 50abc3af450e353e-3b629b84a2528c555fae729e0516e192692cd925 Received: from unknown [76.164.174.81] by p02c12o147.mxlogic.net(mxl_mta-6.16.0-0) with SMTP id ba3cba05.0.129991.00-2297.322245.p02c12o147.mxlogic.net (envelope-from ); Tue, 20 Nov 2012 10:53:48 -0700 (MST) X-MXL-Hash: 50abc3ac178a042e-c92e3fed749881c402abc50985266b19d8200731 Received: from ex-mb1.corp.adtran.com ([fe80::51a3:972d:5f16:9952]) by ex-hc1.corp.adtran.com ([fe80::a43f:7ea6:7688:37b%14]) with mapi id 14.01.0339.001; Tue, 20 Nov 2012 11:53:19 -0600 From: KEN KO To: "STARK, BARBARA H" , Nicholas Weaver Thread-Topic: Focus on Tests, Not Architectures... Thread-Index: AQHNx0GXMS9obShxv0GLr9dkQIvHk5fy/ykA Date: Tue, 20 Nov 2012 17:53:19 +0000 Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> <2D09D61DDFA73D4C884805CC7865E6112F5DD96D@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5DD96D@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.30.5.205] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-AnalysisOut: [v=2.0 cv=YoS4sfkX c=1 sm=1 a=0XgpNN2/4a34ymu16SVwsQ==:17 a] X-AnalysisOut: [=CB9cjBeVwSIA:10 a=XV3CA8akORcA:10 a=qZHQZMT3apYA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=xqWC_Br6kY4A:10 a=eJNrpio] X-AnalysisOut: [GAAAA:8 a=h5VdO6nhOuIA:10 a=O0DK09EvJxzNftSsNEoA:9 a=CjuIK] X-AnalysisOut: [1q_8ugA:10] X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [76.164.174.81] Cc: Marc Linsner , "ippm@ietf.org" , "lmap@ietf.org" Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 17:53:58 -0000 >Since client and controller are deployed by the same entity, it should be = up to that entity to figure out their control architecture, and that entity= should have full say in how to do their own control architecture. I don't think it's a universally held assumption that the client and contro= ller are deployed by the same entity. For instance, a regulator use case sp= anning multiple operators could use a controller function (and, a data coll= ection function) that communicates with clients deployed by each of those o= perators. So I don't think we should ignore architecture. That said, I do believe that focusing on tests is an excellent idea and tha= t that work could progress in ippm while other lmap issues are still being = discussed. From jamesmilleresquire@gmail.com Tue Nov 20 10:01:18 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7036C21F873D for ; Tue, 20 Nov 2012 10:01:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.598 X-Spam-Level: X-Spam-Status: No, score=-3.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id an6AfWxXuJ-0 for ; Tue, 20 Nov 2012 10:01:14 -0800 (PST) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) by ietfa.amsl.com (Postfix) with ESMTP id 0176C21F86E5 for ; Tue, 20 Nov 2012 10:01:13 -0800 (PST) Received: by mail-qc0-f172.google.com with SMTP id b25so4477871qca.31 for ; Tue, 20 Nov 2012 10:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:cc:content-type; bh=/B3AM8yzw9RWQ0qsBzt75QViAT2teOgvnRjzVskXvTU=; b=mTxs64+DlFz3tQF4hSb2KG++ZPsN9dd7pvmeOCspLw2r9w5DguZCn2DPc2A28sj1Lp 2BqRxhCwOhO0ydyl4Gbggxk9iZ32z0TvvIDOkTvlZI6RSFJ8yshTXgut7H/w8JHzJGxM cgc0qbyRRvonoVj01s50HIhZ5pltGkZInuI/tiKjQX9943z8nzM1QX5c7uxRTU+seLju 7Uo0QOP5J1iO0GoqdGi9b3/FfOAksNZgrLcPZry0o51crysFZ16bsPO7qk4RHoEt4d0F FKLzl5h211MJHWbmXxGijf7AVwW1UH//BlIgCPuKex2Vs7slHyg6DaDxGM/Q8gmipueq ndnw== Received: by 10.224.188.13 with SMTP id cy13mr15526292qab.42.1353434473366; Tue, 20 Nov 2012 10:01:13 -0800 (PST) MIME-Version: 1.0 Received: by 10.49.121.134 with HTTP; Tue, 20 Nov 2012 10:00:43 -0800 (PST) From: James Miller Date: Tue, 20 Nov 2012 13:00:43 -0500 Message-ID: To: "STARK, BARBARA H" , "lmap@ietf.org" , Nicholas Weaver Content-Type: multipart/alternative; boundary=485b397dd3b503063e04cef105ae Cc: Shane Amante , Rolf Winter , Henning Schulzrinne Subject: [lmap] On Management, Privacy and Use Case Points X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 18:01:18 -0000 --485b397dd3b503063e04cef105ae Content-Type: text/plain; charset=ISO-8859-1 DISCLAIMER: The opinions expressed are those of the author and do not necessarily represent the views of the Federal Communications Commission or the United States Government. or necessarily co-authors or others on the list! Barbara / Nick, Thanks for everyone's discussions. I wanted to chime in on some related points that seem to have congealed. IMHO, the first three points relate more to operational concerns for implementing a LMAP program and go to scope of IPPM and LMAP. The point on metrics would be a good topic of an IPPM-directed draft. If I misunderstand any of the points quoted in these chains below please clarify but I've picked up the following topics: 1) No large-scale management is necessary to collect metrics 2) No carrier or other's business case requires large scale management standard 3) Privacy concerns do not permit large scale management 4) Metrics are largely sufficient or within scope of IPPM. 1) Management is Relevant to Evaluating the Quality of Data Collected. Barbara references some sources for "speed data" available to consumers and Nick Weaver also chimed on his great work. The FCC view, perhaps shared by others, is that data collected from a managed platform provides a basis for making comparisons of broadband providers performance and supports an evaluation of offerings. Where the technical conditions and methods of the tests (maybe defined as metrics) are executed by parties committed (and audited) to act in good faith in methodologically sounds ways, the data produced is useful for consumers, academics, and broadband providers for many purposes, including for the FCC's review of the state of consumer broadband performance. Thus understanding the architecture and how it supports the collection of metrics I think is something people care deeply about, and a focus solely on "metrics" without understanding the platform is flawed. The distinctions being asserted between metrics and architecture dissolve when we scrutinize the conditions defining accuracy and precision for performance measurements. You can't expect to use a kindergarten 12" ruler for all purposes or expect what they taught you in kindergarten as process would be sufficient for every measurement case, and the case holds for broadband measurement. Data that isn't managed and subjected to scrutiny presents undefined conditions, and may not be suitable for many analytic purposes. Collections that I am aware of that are widely referenced employ some management structure and this effort is to explore how the architectures could be standardized to reduce the costs and burdens of scaling programs. For FCC purposes we subject collected results to significant review in preparing our reports that have documented consumer fixed broadband performance over the last two years. In the LMAP draft we've put down some of the major conditions that were relevant in the FCC studies, control of the test schedule, protection of customer privacy, etc. Implementation as a "federated" or other architecture is an issue of who schedules tests and collects the data, and as the draft states, is only one of several scenarios but in the least *requires* authentication and other management functions in the platform. Nick makes the point that protocols may be an easier problem than metrics, but wanted to pull out he implements three architectural management structures "depending on whether quick & easy or server side or mutual authentication is require[d]." So maybe extending the point, the LMAP structure would at minimum support three classes of client server authentication: Light and Easy (L&E), Server Side (SS), and Mutual Authentication (MA). I don't agree that you can ignore the problem with a focus on metrics as the same problems with then pop up with how the tests are implemented, and the same problems will occur across tests, demonstrating that there's a modularity in design that needs to be leveraged. :: Standards around LMAP offer benefits such as reducing costs of components and management, and may otherwise be essential to scaling large-scale measurement to larger numbers. As a baseline, do we agree on the cost and burden reducing benefit of standards generally, and that a standard around LMAP would reduce costs and make scalablity of large-scale measurement more feasible for the categories of users defined in the document? 2) Performance Measurement is a Developing Business Requirement. Prior to the FCC program that began in 2010, we are aware that several broadband providers of consumer broadband services were contracting for technical solutions substantially similar or identical to the ones we contracted for in the FCC program but I can't speculate what carriers had some sort of similar measurement program in place. On the questions about how relevant large-scale measurement is for the industry, I would suggest that there's plenty of evidence of adoption of "LMAP" style programs in the US prior to the FCC program, and interest has only grown both in the US and Internationally. For the FCC program, 13 national ISPs providing residential fixed service to consumers comprising 85% of the US market are tested and participate in the program. These companies were signatories to the Code of Conduct and regularly participate in meetings and other aspects of the program. For reference, signatories to the 2012 Code of Conduct were Adtran, AT&T, Cablevision, CenturyLink, Charter, Comcast, Corning, Cox, Fiber to the Home Council, Frontier, Georgia Tech, Genband, Insight (by TWC), Internet Society, Mediacom, Motorola, Qwest, Time Warner Cable, US Telecom, Verizon, Viasat and Windstream. I cannot speak to the specifics of carrier's independent broadband performance efforts but there are references in the press and national advertising that indicate that carriers do employ programs like those described in the LMAP draft. LMAP should support many use cases, including the cases described in section 2 of the draft at a minimum, but the three discussed have been industry, regulator and researcher lead efforts. That there is established, strong interest in broadband metrics from industry, academia and researchers, and regulators seems well founded, in any event. http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf http://transition.fcc.gov/cgb/measuringbroadbandreport/technical_appendix/Technical_Appendix_Full.pdf 3) Privacy is Crucially Important and Can Be Protected. Without straying from the technical focus of the list, let me first say that privacy issues are a serious topic that everyone in the field takes very seriously, but that they need not preclude a well conceived and *managed* program. While a best-practice list might be helpful in some fora, I think it's sufficient here to discuss how to ensure that the technical pieces are capable of enforcing whatever data integrity policy an implementor may adopt. This is not the correct forum to define a monolithic privacy policy, but is a good place to discuss the resiliency of approaches to protecting a variety of policies, IMHO. The draft observes that existing authentication protocols and approaches could be adopted for LMAP. Barbara referenced US and EU law specifically, but I'd point out that in the FCC program, governed by US law, the terms and conditions that volunteers agree to were crafted after much review that incorporated the input of many stakeholders, and worked together with other mechanisms to protect privacy. The FCC is conducting a review of the existing privacy components for the mobile program and I would believe that any large-scale program would likely tailor the specifics of a privacy strategy to the specifics of the particular program and nature of the data to be collected. 4) Broadband Performance Metric Needs are Diverse and Growing. While Ping tests are valuable, there are limits to the value of ICMP ping for measurement purposes, but it's clear that hundreds of thousands of consumers expressed interest in volunteering to be a part of the FCC efforts and other parties discussing consumer interest in more detailed broadband statistics support a clear demand for data in this area. The FCC's panel of U.S. broadband subscribers is drawn from a pool of over 145,000 volunteers following an ongoing recruitment campaign that ran from May 2010 through February 2012. As we discussed in the BOF, it was the FCC's view, largely confirmed by the participation and results of the study and echoed by other regulators globally, researchers like those supporting M-LAB, and other stakeholders, that more information about broadband performance is valuable, in many cases necessary, and has been insufficient. [combing threads referenced above] On Wed, Nov 14, 2012 at 5:20 PM, STARK, BARBARA H wrote: > Shane suggested that the draft is a good start and asked me to provide > comments against it. I've been avoiding doing that, because I'm not very > fond of this draft, and am not sure if it can be "fixed", or if it should > be completely overhauled, or totally scrapped. But others apparently do > like it, so let me explain some of my objections to the current draft. > [Barbara's email was the start for much of the following discussion and included specifics spurred discussion of the topics above] On Tue, Nov 20, 2012 at 9:16 AM, STARK, BARBARA H wrote: > > > [--snip--] > > > Data collector: I do not see this as a necessary architectural > element, either. > > > As with the Measurement controller, it has been assigned the role of > > > providing multiple, separable functions: collecting data, storing > data, and > > > providing access to the data. > > > > I'm not sure I understand your concern. Given the resource constraints > on > > some classes of measurement clients, doesn't it make sense for the > > measurement client to send it's performance measurement results to "the > > cloud" (a.k.a.: data collector) for longer-term archival, storage and > centralized > > access by the ISP and/or 3rd-parties? > > For the end user on-demand tests, I don't see that a distinct data > collector necessarily needs to exist. The Measurement Client could present > collected data to the end user directly through a local user interface. The > end user may or may not want to have the data saved for future use. The end > user may not have a requirement for on-demand test data to be shared with > others. Also, since the LMAP Data Collector definition insists that many > functions be combined together (data collection function, data storage > function, and providing access to the data function), I do not agree that > this combined-function Data Collector architectural element, as defined in > the draft, is necessary to providing functionality that some of the use > cases do call for. I agree that a data collection function, a data storage > function, and a data access function are needed by some of the use cases. I > believe it is possible for data collection and storage functions to be > internal to a single entity's > management architecture, and am not convinced that deployers and > implementers of these functions are asking for IETF help in defining them. > > > > Network parameter server: I do not see this as a necessary > architectural > > element. If there is a need for an externally-facing function that > provides > > access to access-provider-collected data for the regulatory use case to > > include corresponding (anonymous) information related to subscribed line > > rate, etc., then that is a requirement on that external-facing function. > If an > > access provider wants help in figuring out how to get this information > to the > > external-facing function, then I would prefer for that access provider > (or > > their proxy) to ask for such help. > > > > I would think that if, say, an ISP already has a "network parameter > server" > > they already own & operate, then that's fine ... they can continue to > run it as > > they do today. However, the key aspect we probably want to work on is > > defining a _standards-based_ "data model" (e.g.: XML or JSON schema?) > > that can encapsulate appropriate/relevant bits of data about subscribed > lines > > in that (existing) Network Parameter Server. This would allow for much > > more straightforward access to and parsing of "network parameters" by, > for > > example, 'authorized parties' as the draft states (Section 3.4). > > I don't object to data model definition. I do think it may be > premature, in the absence of a good understanding of the metrics. Without > understanding the metrics, I believe the data model would need to be very > abstract. Metrics, to me, could include an expression of "subscribed line > rate" that has consistent meaning across all access technologies. This > information is something I would consider a "network parameter". I would > prefer to see IETF focus on metrics, rather than a measurement management > and control architecture. Metrics can drive a data model that is > independent of the architecture used to deliver the data. > > > > If there is a desire to define an interworking function that would > allow a 3rd > > > party to get (not anonymous) customer-specific line information > directly > > > from the access provider (so the 3rd party could correlate it with > data they > > > collect from their own Measurement client(s)), then that would be a > distinct > > > function, as well. Personally, I would shy away from trying to define > this > > > function. But if someone else wants to tackle it, I might be willing > to provide > > > comments on it. > > > > Hrm. You raise a good point above about a 3rd-party acquiring "non- > > anonymous" customer line information from the access provider so that the > > 3rd-party could correlate it with data they collect from their own > (3rd-party) > > measurement clients. > > 1) I'm not sure what, if any, technical mechanisms the IETF could > develop to > > prevent such a thing from happening, other than to NOT specify any data- > > models/schemas that encapsulate such sensitive, non-anonymous > > information in a 'standards-based' data-model in the first place. > > 2) That said, I would think that "privacy laws" in place in the U.S. > and other > > parts of the world (perhaps most significantly, the EU) would prevent the > > ISPs from [ever] offering access to that data for fear of violating both > the law > > and, more importantly, their customer's trust as well. > > I'm not aware of any access provider who *wants* to do this. I > couldn't quite tell if some of the requirements were trying to provide > support for this use case -- I could read some of the statements either > way. I'm glad we agree that this is not a use case that we need to support. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > On Mon, Nov 19, 2012 at 5:12 PM, STARK, BARBARA H wrote: > > > Measurement controller: This is not a necessary architectural element. > This > > > is an element specific to this proposed physical architecture. > > > > I'm unclear if your concern is related to: > > 1) having _strictly_ a single measurement controller be part of, or in > control > > of a single measurement "panel" (e.g.: group of measurement clients & > > measurement server); or, > > 2) two, or more, measurement controllers (likely owned/operated by > > different parties) acting in concert to coordinate the rendezvous > between an > > appropriate panel of measurement clients with measurement servers. See > > my previous e-mail re: "Section 3" for an example of what I'm trying to > > describe here. > > > > If neither of those are what concern you, can you point me in the right > > direction? :-) > > Measurement Controllers are not required by all use cases (e.g., end > user on-demand testing that is controlled and managed by the end user, > using devices under the direct control and management of the end user, > doesn't call for the existence of a Measurement Controller). > I do not find the use case of allowing 3rd party management and control of > test infrastructure to be a compelling use case. IMO, this is not a > necessary function of any of the more general use cases for support of > end-user specific on-demand testing and large scale performance > measurements. > Outside of this 3rd party use case, none of the other use cases really > care about how control is achieved -- it's a function internal to an > architecture of a single entity. And as I mentioned in my other emails, not > all Measurement Clients or Measurement Servers need to allow for > measurement-specific remote control. > We have no clue as to what the desired metrics are, or the recommended > measurements and tests to achieve those metrics; so assuming a Measurement > Controller is needed to coordinate any of these tests is premature. > Therefore, I don't consider it to be a necessary element of an architecture. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap On Tue, Nov 20, 2012 at 11:01 AM, Nicholas Weaver wrote: > > On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote: > >> -----Original Message----- > >> From: Marc Linsner [mailto:mlinsner@cisco.com] > >> As far as metrics, my assumption is any metric defined by IPPM or > related > >> SDO body, with additions as desired. From the plenary presentation I > would > >> expect at least the following metrics: Sustained Download, Sustained > Upload, > >> Web Browsing Download, UDP Packet Loss, VoIP Measure, DNS Failures, > >> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Upload, > >> UDP Latency, Video Streaming Measure, DNS Resolution, Latency Under > >> Load, Total Bytes Uploaded. Obviously, extensibility is a key feature. > > > > How key is extensibility, how dynamic does extensibility have to > be, and are various approaches to achieving extensibility allowed? Can > extensibility be achieved by firmware upgrades that happen maybe once a > year? Do all clients and servers have to be extensible, or is it ok to > achieve extensibility through the addition of new clients and servers, > potentially in new/different/separate physical devices/elements? Does > extensibility imply near infinite extensibility or is it reasonable for > extensibility to be constrained by lack of resources available on the > device that hosts a client? > > Barbara > > I've said it once, I'll say it one final time: > > This focus on architecture is deeply misguided. As someone who's built > one such architecture (Netalyzr), working on two others (Fathom, a browser > extension, and techniques using GuruPlugs/Raspberry Pis), the architecture > is the easy part. > > In all this work, having some "standard" way to communicate between my > measurement clients and servers is useless. I just use HTTP, HTTPS, or SSH > depending on whether quick & easy or server side or mutual authentication > is required, with whatever load-balancing is needed handled by random > assignment of remote test node. > > The internal data store is whatever is appropriate for the project (eg. > for Netalyzr its python pickled objects for storage on our web server, data > in a Postgress database for analysis). > > Updates are handled appropriately (Netalyzr is updateless, its an applet > so its always up-to-date. Our android version will just use the app store > update mechanism. Fathom we use Firefox's update mechanism. The plugs are > a hack using SSH, but its a fair amount of custom goop, and they update > nightly). > > And all three end up having rather different architectural requirements. > > > > The hard part is developing the test metrics and measurements, both in the > abstract and within the API constraints of the target platform. E.g. why I > like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I > can just "Run Netalyzr" for all Netalyzr tests. But the same isn't the > case for the Ripe Atlas or Bismark platform. > > And Marc's list is actually just a start: you also have generic proxy > presence & location detection, application specific proxy detection and > analysis, packet injection detection, differential application behavior > detection, application agnostic traffic shaping detection, DNSSEC support, > DNS manipulation detection, path MTU, IPv6 for all of the previous, and a > ton of others. Netalyzr is currently composed of >100 distinct tests. And > this amount just continues to grow. > > > If the IETF actually wants to have an impact in this area, don't bother > focusing on protocols for communication, focus on tests with given > attributes. Because people like me who actually build these things are > going to ignore an architectural focus because that has no value. But > having ACCEPTED STANDARD tests would be very valuable. > > E.g. a good, standard latency-under-load test that can be written in Flash > (so TCP/UDP connections only to cooperating servers) would be great. Our > current one in Netalyzr is only OK: we believe that one could do a lot > better. > > If you could then figure out WHERE the bottleneck is (using a server that > can also do simultaneous traceroutes), this would be even more valuable. > > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap On Mon, Nov 19, 2012 at 1:58 PM, STARK, BARBARA H wrote: > [--snip--] > > > Here are some specific comments on the architectural elements: > > > Measurement client: I agree that if measurements (active or passive) > are > > > going to be made, then one or more Measurement clients are a mandatory > > > element. I agree with the first 3 sentences of the Measurement client > > > definition. > > > > Well 3 out of 4 (sentences) is a good start. :-) > > > > > > > But the idea that *all* Measurement clients must be manageable and > > > "provide for communications within different network segments" is not > > > something I agree with. > > > > I'm not sure I detect an implication that all measurement clients must be > > manageable. What the fourth sentence says is the following: > > "Client measurement functionality must be implementable in a variety of > > user contexts ..." > > As I mentioned, I thought it was possible to agree on the existence > of a Measurement Client (as defined by the first 3 sentences) as an > architectural element. The problem really arises with the assumptions as to > the nature of this element that are implied by certain of the requirements: > > A-2: Measurement clients and servers MUST support an extensible set > of performance metrics. > A-4: Measurement clients MUST be able perform both active and > passive measurements. > M-3: A measurement client MUST be able to register with the data > collection platform automatically, announcing its availability and > relevant system parameters. (For example, a cable or DSL modem > may indicate its make and model number.) > M-4: A measurement client MUST be able to declare what kind of > measurements it can perform, e.g., by enumerating a set of > measurement identifiers. > C-4: A measurement client MUST be able to authenticate and authorize > the measurement controller. > > As I mentioned in my last email, I believe Measurement Clients may exist > inside ONTs, DSLAMs, and even very, very simple DSL modems (some of which > are nothing more than a USB dongle that connects to a PC, and many of which > are completely unmanaged, except locally by the end user). The idea that > these devices / network elements must support these requirements in order > to participate in a holistic scheme/architecture to collect meaningful > performance data is something I do not agree with. I also believe that it > is possible for a Management Client to be directly under the control of the > end user, and potentially not be upgradable or capable of an extensible set > of tests. I believe that the Windows Command Prompt is a very effective > (for me) end-user-controlled Measurement Client for performing a variety of > tests. But it does not meet the above requirements. Nor should it be > expected to. I use it often, though, for my on-demand connectivity and > performance testing. > > To me, this architecture draft assumes that a "federated" architecture is > necessary for useful collection of performance measurements and conducting > tests. I disagree with that premise. There also seems to be an assumption > that a "federated" architecture must be defined prior to defining > appropriate metrics. I disagree. I believe that a federated architecture > (that allows for all sorts of management and control and updating of > tests/schedules/etc.) is unnecessary. I believe it is possible to re-use a > lot of what exists, with minimal additions, to accomplish large scale > performance testing and on-demand diagnostics, that meet the basic "needs" > (not to be confused with "wants") of various parties. I believe it is > possible for 3rd parties and ANPs to develop one-to-one relationships with > an end user, to allow for on-demand and scheduled testing and collection of > test data. The end user can even allow for 3rd parties to access tools > under end-user control, which interact with ANP > Measurement Servers (taking advantage of the end user to ANP > relationship). For large scale performance data -- I believe it is possible > to collect such data and provide access to the resulting anonymized data > without enabling 3rd party management and control of the elements involved > in data collection. > > My recommendation would be to focus on defining appropriate metrics. Once > we understand the metrics, we can work on understanding the appropriate > measurements and tests, and the set of Measurement Clients and Measurements > Servers that could be used to perform those measurements and tests. If > something else is needed from a standards organization, that should become > evident once the metrics are identified. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap On Mon, Nov 19, 2012 at 4:52 PM, STARK, BARBARA H wrote: > [...] > > This architecture seems to imply that it's relevant to the end user > on-demand test case. My favorite test is "ping". I do my ping testing > against a variety of Internet websites, but also my access/Internet > provider's DNS servers and IP gateways. What I ping depends on what my > problem is. Anything that responds to my ping is a Measurement Server in my > book. > > As an end user, I have no need for the set of tests that such Measurement > Servers support. > I also have no need for my client to authenticate with them. I don't think > they've ever required authentication of my client. I have no idea if they > communicate with a Data Collector, and I really don't care. > > Similarly, I often use "bandwidth test" websites for on-demand bandwidth > tests. I really don't care whether they're extensible, authentication has > never been an issue, and I have no requirement for them to communicate with > a Data Collector > > > ---snip--- > > The following requirements are applicable to the overall LMAP Reference > > Architecture, as described in this document. How each of these > > requirements are applied to specific implementation of a particular type, > > category or class of measurement client, server, etc. device is > out-of-scope > > for this document. > > ---snip--- > > Is the above text good enough, for now? > > No. Unless the described architectural elements are changed to be > "LMAP Measurement Client" and "LMAP Measurement Server", with definition of > these being that an LMAP Measurement Client/Server is one that meets the > requirements enumerated in this document. I don't think it's possible to go > from the generic architecture elements of Measurement Client/Server to the > enumerated requirements that seem to have something very specific in mind > that is not at all consistent with the generic definitions provided in the > architecture section. > > > But, going back to Section 3.2, Measurement Server, I can see how it may > be > > useful to add some of the points you raised above. How about suggesting > to > > add something like the following to Section 3.2? > > ---snip--- > > Measurement servers likely will be owned and operated by a variety of > > administrative entities, some of which may be unrelated to the ISP > providing > > service to a measurement client. Therefore, measurement servers may > > need a facility where they can describe their capability to perform one, > or > > more, suites of active performance measurement tests to a measurement > > controller. In this way, the measurement controller may facilitate > > coordination of measurement clients to measurement servers that have > > matching active performance measurement capabilities. > > ---snip--- > > That still doesn't allow for a website that responds to a ping to be > considered a Measurement Server. Or a website that provides bandwidth > measurements. But they are Measurement Servers. They just aren't > Measurement Servers that meet these "LMAP" requirements. Do any of the use > cases create a need for all Measurement Servers (used by that use case) to > meet these requirements? I don't think so. Even the 3rd party testing use > case should allow for ping tests without insisting that the server > supporting the ping tests meet these requirements. > > This LMAP architecture is assuming that a requirement of the architecture > is to support all use cases, within a single architecture. I disagree with > that premise. > And I disagree that 3rd party management and control of "LMAP" > architecture elements is a compelling use case. The protocols may be > simple, but it's incredibly complex to manage and administer, and to put in > all the controls needed to prevent end users from being negatively impacted > by 3rd parties. A simple requirement of "make sure 3rd parties can't break > anything" is insufficient and ignores the fact that ensuring this is > incredibly complex and difficult. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap > --485b397dd3b503063e04cef105ae Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
DISCLAIMER: =A0The opinions expressed are those of the author and= do not necessarily=A0represent the views of the Federal Communications Com= mission or the=A0United States Government. =A0 or necessarily co-authors or= others on the list!

Barbara / Nick,

Thanks f= or everyone's discussions. =A0I wanted to chime in on some related poin= ts that seem to have congealed. =A0IMHO, the first three points relate more= to operational=A0concerns for implementing a LMAP program and go to scope = of IPPM and LMAP. =A0The point on metrics would be a good topic of an IPPM-= directed draft. =A0If I misunderstand any of the points quoted in these cha= ins below please clarify but I've picked up the following topics:

1) No large-scale management is necessary to collect me= trics
2) No carrier or other's business case requires large s= cale management standard
3) Privacy concerns=A0do not permit larg= e scale management
4) Metrics are largely sufficient or within scope of IPPM.

1) Management is Relevant to Evaluating the Quality of Data Collec= ted.

Barbara references some sources for "speed data" avai= lable to consumers and Nick Weaver also chimed on his great work. =A0The FC= C view, perhaps shared by others, is that data collected from a managed pla= tform provides a basis for making comparisons of broadband providers perfor= mance and supports an evaluation of offerings. =A0Where the technical condi= tions and methods of the tests (maybe defined as metrics) are executed by p= arties=A0committed=A0(and audited) to act in good faith in methodologically= sounds ways, the data produced is useful for consumers, academics, and bro= adband providers for many purposes, including for the FCC's review of t= he state of consumer broadband performance. =A0Thus understanding the archi= tecture and how it supports the collection of metrics I think is something = people care deeply about, and a focus solely on "metrics" without= understanding the platform is flawed. =A0The distinctions being asserted b= etween metrics and architecture=A0dissolve=A0when we=A0scrutinize=A0the con= ditions defining accuracy and precision for performance measurements. =A0Yo= u can't expect to use a kindergarten 12" ruler for all purposes or= expect what they taught you in kindergarten as process would be sufficient= for every measurement case, and the case holds for broadband measurement. = </humour>

Data that isn't managed and subjected to scrutiny p= resents undefined conditions, and may not be suitable for many analytic pur= poses. =A0Collections that I am aware of that are widely referenced employ = some management structure and this effort is to explore how the=A0architect= ures=A0could be standardized to reduce the costs and burdens of scaling pro= grams. =A0For FCC purposes we subject collected results to significant revi= ew in preparing our reports that have documented consumer fixed broadband p= erformance over the last two years. =A0In the LMAP draft we've put down= some of the major conditions that were relevant in the FCC studies, contro= l of the test schedule, protection of customer privacy, etc. =A0Implementat= ion as a=A0"federated" or other architecture is an issue of who s= chedules tests and collects the data, and as the draft states, is only one = of several=A0scenarios=A0but in the least *requires* authentication and oth= er management functions in the platform.

Nick makes the point that protocols may be an easier pr= oblem than metrics, but wanted to pull out he implements three=A0architectu= ral=A0management structures "depending on whether quick & easy or = server side or mutual authentication is require[d]." =A0So maybe exten= ding the point, the LMAP structure would at minimum support three classes o= f client server authentication: =A0Light and Easy (L&E), Server Side (S= S), and Mutual Authentication (MA). =A0I don't agree that you can ignor= e the problem with a focus on metrics as the same problems with then pop up= with how the tests are implemented, and the same problems will occur acros= s tests, demonstrating that there's a modularity in design that needs t= o be leveraged.

:: Standards around LMAP offer benefits such as reducin= g costs of components and management, and may otherwise be essential to sca= ling large-scale=A0measurement=A0to larger numbers. =A0As a baseline, do we= agree on the cost and burden reducing benefit of standards generally, and = that a standard around LMAP=A0would=A0reduce costs and make scalablity=A0of= large-scale=A0measurement=A0more=A0feasible=A0for the categories of users = defined in the document?
=A0
2) Performance Measurement is a Developing Business Requ= irement.=A0

Prior to the FCC program that began in= 2010, we are aware that several broadband providers of consumer broadband= =A0services=A0were contracting for technical solutions substantially=A0simi= lar or=A0identical=A0to the ones we contracted for in the FCC program but I= can't speculate what carriers had some sort of=A0similar=A0measurement= program in place. =A0 On the questions about how relevant large-scale meas= urement is for the industry, I would=A0suggest that there's plenty of e= vidence of adoption of "LMAP" style programs in the US prior to t= he FCC program, and interest has only grown both in the US and Internationa= lly. =A0

For the FCC program, 13 national ISPs providing residen= tial fixed service to consumers comprising 85% of the US market are tested = and participate in the program. =A0These companies were signatories to the = Code of Conduct and regularly participate in meetings and other aspects of = the program. =A0For reference, signatories to the 2012 Code of Conduct were= Adtran, AT&T, Cablevision, CenturyLink, Charter, Comcast, Corning,=A0C= ox, Fiber to the Home Council, =A0Frontier, Georgia Tech, Genband, Insight = (by TWC), Internet Society,=A0Mediacom, Motorola, Qwest, Time Warner Cable,= US Telecom, Verizon, Viasat and Windstream. =A0

I cannot speak to the specifics of carrier's indepe= ndent broadband performance efforts but there are references in the press a= nd national advertising that indicate that carriers do employ programs like= those described in the LMAP draft. =A0LMAP should support many use cases, = including the cases described in section 2 of the draft at a minimum, but t= he three discussed have been industry, regulator and researcher lead effort= s. =A0That there is established, strong interest in broadband metrics from = industry, academia and researchers, and regulators seems well founded, in a= ny event.


3) Privacy is Crucially Important and Can Be Protected.=

Without straying from the technical focus of the = list, let me first say that privacy issues are a serious topic that everyon= e in the field takes very seriously, but that they need not preclude a well= conceived and *managed* program. =A0While a best-practice list might be he= lpful in some fora, I think it's sufficient here to discuss how to ensu= re that the technical pieces are capable of enforcing whatever data integri= ty policy an implementor may adopt. =A0This is not the correct forum to def= ine a monolithic privacy policy, but is a good place to discuss the resilie= ncy of approaches to protecting a variety of policies, IMHO. =A0The draft o= bserves that existing=A0authentication=A0protocols and approaches could be = adopted for LMAP.

Barbara referenced US and EU law specifically, but I= 9;d point out that in the FCC program, governed by US law, the terms and co= nditions that volunteers agree to were crafted after much review that incor= porated the input of many=A0stakeholders, and worked together with other me= chanisms to protect privacy. =A0The FCC is conducting a review of the exist= ing privacy components for the mobile program and I would believe that any = large-scale program would likely tailor the specifics of a privacy strategy= to the specifics of the particular program and nature of the data to be co= llected. =A0

4) Broadband Performance Metric Needs are Diverse and G= rowing.

While Ping tests are valuable, there are l= imits to the value of ICMP ping for measurement purposes, but it's clea= r that hundreds of thousands of consumers expressed interest in=A0volunteer= ing=A0to be a part of the FCC efforts and other parties discussing consumer= interest in more detailed broadband statistics support a clear demand for = data in this area. The FCC's panel of U.S. broadband subscribers is dra= wn from a pool of over 145,000=A0volunteers following an ongoing recruitmen= t campaign that ran from May 2010=A0through February 2012. =A0As we discuss= ed in the BOF, it was the FCC's view, =A0largely confirmed by the parti= cipation and results of the study and echoed by other regulators globally, = researchers like those supporting M-LAB, and other stakeholders, that more = information about broadband performance is valuable, in many cases necessar= y, and has been insufficient. =A0

</lunchbreak>

=A0[combin= g threads referenced above]


On Wed, Nov 14, 2012 at 5:20 PM, STARK, BARBARA H=A0<bs76= 52@att.com>=A0wrote:
Shane suggested that the draft is a good start and asked m= e to provide comments against it. I've been avoiding doing that, becaus= e I'm not very fond of this draft, and am not sure if it can be "f= ixed", or if it should be completely overhauled, or totally scrapped. = But others apparently do like it, so let me explain some of my objections t= o the current draft.
=A0
[Barbara's email was the start fo= r much of the following discussion and included specifics spurred discussio= n of the topics above]

On Tu= e, Nov 20, 2012 at 9:16 AM, STARK, BARBARA H=A0<bs7652@att.com>= =A0wrote:

> [--snip--]
> > Data collector: I do= not see this as a necessary architectural element, either.
> > As with the Measurement controller, it has been assigned the role= of
> > providing multiple, separable functions: collecting data, = storing data, and
> > providing access to the data.
>
> I'm not sure I understand your concern. =A0Given the resource cons= traints on
> some classes of measurement clients, doesn't it make= sense for the
> measurement client to send it's performance meas= urement results to "the
> cloud" (a.k.a.: data collector) for longer-term archival, storage= and centralized
> access by the ISP and/or 3rd-parties?

<bhs> For the end user on-demand tests, I don't see that a disti= nct data collector necessarily needs to exist. The Measurement Client could= present collected data to the end user directly through a local user inter= face. The end user may or may not want to have the data saved for future us= e. The end user may not have a requirement for on-demand test data to be sh= ared with others. Also, since the LMAP Data Collector definition insists th= at many functions be combined together (data collection function, data stor= age function, and providing access to the data function), I do not agree th= at this combined-function Data Collector architectural element, as defined = in the draft, is necessary to providing functionality that some of the use = cases do call for. I agree that a data collection function, a data storage = function, and a data access function are needed by some of the use cases. I= believe it is possible for data collection and storage functions to be int= ernal to a single entity's
=A0 management architecture, and am not convinced that deployers and implem= enters of these functions are asking for IETF help in defining them.

> > Network parameter server: I do not see this as a necessary = architectural
> element. If there is a need for an externally-facing function that pro= vides
> access to access-provider-collected data for the regulatory u= se case to
> include corresponding (anonymous) information related to= subscribed line
> rate, etc., then that is a requirement on that external-facing functio= n. If an
> access provider wants help in figuring out how to get this= information to the
> external-facing function, then I would prefer f= or that access provider (or
> their proxy) to ask for such help.
>
> I would think that = if, say, an ISP already has a "network parameter server"
> = they already own & operate, then that's fine ... they can continue = to run it as
> they do today. =A0However, the key aspect we probably want to work on = is
> defining a _standards-based_ "data model" (e.g.: XML o= r JSON schema?)
> that can encapsulate appropriate/relevant bits of d= ata about subscribed lines
> in that (existing) Network Parameter Server. =A0This would allow for m= uch
> more straightforward access to and parsing of "network par= ameters" by, for
> example, 'authorized parties' as the = draft states (Section 3.4).

<bhs> I don't object to data model definition. I do thi= nk it may be premature, in the absence of a good understanding of the metri= cs. Without understanding the metrics, I believe the data model would need = to be very abstract. Metrics, to me, could include an expression of "s= ubscribed line rate" that has consistent meaning across all access tec= hnologies. This information is something I would consider a "network p= arameter". I would prefer to see IETF focus on metrics, rather than a = measurement management and control architecture. Metrics can drive a data m= odel that is independent of the architecture used to deliver the data.

> > If there is a desire to define an interworking function = that would allow a 3rd
> > party to get (not anonymous) customer-s= pecific line information directly
> > from the access provider (so= the 3rd party could correlate it with data they
> > collect from their own Measurement client(s)), then that would be= a distinct
> > function, as well. Personally, I would shy away fr= om trying to define this
> > function. But if someone else wants t= o tackle it, I might be willing to provide
> > comments on it.
>
> Hrm. =A0You raise a good point ab= ove about a 3rd-party acquiring "non-
> anonymous" customer= line information from the access provider so that the
> 3rd-party co= uld correlate it with data they collect from their own (3rd-party)
> measurement clients.
> 1) =A0I'm not sure what, if any, tech= nical mechanisms the IETF could develop to
> prevent such a thing fro= m happening, other than to NOT specify any data-
> models/schemas tha= t encapsulate such sensitive, non-anonymous
> information in a 'standards-based' data-model in the first pla= ce.
> 2) =A0That said, I would think that "privacy laws" in= place in the U.S. and other
> parts of the world (perhaps most signi= ficantly, the EU) would prevent the
> ISPs from [ever] offering access to that data for fear of violating bo= th the law
> and, more importantly, their customer's trust as wel= l.

<bhs> I'm not aware of any access provider who *w= ants* to do this. I couldn't quite tell if some of the requirements wer= e trying to provide support for this use case -- I could read some of the s= tatements either way. I'm glad we agree that this is not a use case tha= t we need to support.
Barbara
__________= _____________________________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap
<= br>


On Mon, Nov 19, 2012 at 5:12 PM= , STARK, BARBARA H=A0<bs7652@att.com>=A0wrote:
> > Measurement controller: This is not a neces= sary architectural element. This
> > is an element specific to this proposed physical architecture.>
> I'm unclear if your concern is related to:
> 1) hav= ing _strictly_ a single measurement controller be part of, or in control > of a single measurement "panel" (e.g.: group of measurement = clients &
> measurement server); or,
> 2) two, or more, mea= surement controllers (likely owned/operated by
> different parties) a= cting in concert to coordinate the rendezvous between an
> appropriate panel of measurement clients with measurement servers. =A0= See
> my previous e-mail re: "Section 3" for an example of = what I'm trying to
> describe here.
>
> If neither of= those are what concern you, can you point me in the right
> direction? =A0:-)

<bhs> Measurement Controllers are= not required by all use cases (e.g., end user on-demand testing that is co= ntrolled and managed by the end user, using devices under the direct contro= l and management of the end user, doesn't call for the existence of a M= easurement Controller).
I do not find the use case of allowing 3rd party management and control of = test infrastructure to be a compelling use case. IMO, this is not a necessa= ry function of any of the more general use cases for support of end-user sp= ecific on-demand testing and large scale performance measurements.
Outside of this 3rd party use case, none of the other use cases really care= about how control is achieved -- it's a function internal to an archit= ecture of a single entity. And as I mentioned in my other emails, not all M= easurement Clients or Measurement Servers need to allow for measurement-spe= cific remote control.
We have no clue as to what the desired metrics are, or the recommended meas= urements and tests to achieve those metrics; so assuming a Measurement Cont= roller is needed to coordinate any of these tests is premature. Therefore, = I don't consider it to be a necessary element of an architecture.
Barbara
__________= _____________________________________
lmap mailing list
lmap@ietf.org




On Tue, Nov 20, 2012 at 11:01 AM, Nicholas Weaver=A0<<= a href=3D"mailto:nweaver@icsi.berkeley.edu" target=3D"_blank">nweaver@icsi.= berkeley.edu>=A0wrote:

On Nov 20, 2012, at 7:31 AM, STARK, BARBARA H wrote:
>> -----O= riginal Message-----
>> From: Marc Linsner [mailto:mlinsner@cisco.com]
>>= ; As far as metrics, my assumption is any metric defined by IPPM or related=
>> SDO body, with additions as desired. From the plenary presentation= I would
>> expect at least the following metrics: Sustained Downl= oad, Sustained Upload,
>> Web Browsing Download, UDP Packet Loss, = VoIP Measure, DNS Failures,
>> ICMP Packet Loss, Total Bytes Downloaded, Burst Download, Burst Up= load,
>> UDP Latency, Video Streaming Measure, DNS Resolution, Lat= ency Under
>> Load, Total Bytes Uploaded. =A0Obviously, extensibil= ity is a key feature.
>
> <bhs> How key is extensibility, how dynamic does extensi= bility have to be, and are various approaches to achieving extensibility al= lowed? Can extensibility be achieved by firmware upgrades that happen maybe= once a year? Do all clients and servers have to be extensible, or is it ok= to achieve extensibility through the addition of new clients and servers, = potentially in new/different/separate physical devices/elements? Does exten= sibility imply near infinite extensibility or is it reasonable for extensib= ility to be constrained by lack of resources available on the device that h= osts a client?
> Barbara

I've said it once, I'll say it one final time:<= br>
This focus on architecture is deeply misguided. =A0As someone who= 9;s built one such architecture (Netalyzr), working on two others (Fathom, = a browser extension, and techniques using GuruPlugs/Raspberry Pis), the arc= hitecture is the easy part.

In all this work, having some "standard" way to communicate b= etween my measurement clients and servers is useless. =A0I just use HTTP, H= TTPS, or SSH depending on whether quick & easy or server side or mutual= authentication is required, with whatever load-balancing is needed handled= by random assignment of remote test node.

The internal data store is whatever is appropriate for the project (eg.= for Netalyzr its python pickled objects for storage on our web server, dat= a in a Postgress database for analysis).

Updates are handled appropr= iately (Netalyzr is updateless, its an applet so its always up-to-date. =A0= Our android version will just use the app store update mechanism. =A0Fathom= we use Firefox's update mechanism. =A0The plugs are a hack using SSH, = but its a fair amount of custom goop, and they update nightly).

And all three end up having rather different architectural requirements= .



The hard part is developing the test metrics and measureme= nts, both in the abstract and within the API constraints of the target plat= form. =A0E.g. why I like the GuruPlugs and Raspberry Pi is that I can run J= ava, so therefore I can just "Run Netalyzr" for all Netalyzr test= s. =A0But the same isn't the case for the Ripe Atlas or Bismark platfor= m.

And Marc's list is actually just a start: you also have generic pro= xy presence & location detection, application specific proxy detection = and analysis, packet injection detection, differential application behavior= detection, application agnostic traffic shaping detection, DNSSEC support,= DNS manipulation detection, path MTU, IPv6 for all of the previous, and a = ton of others. =A0Netalyzr is currently composed of >100 distinct tests.= =A0And this amount just continues to grow.


If the IETF actually wants to have an impact in this area, don'= t bother focusing on protocols for communication, focus on tests with given= attributes. =A0Because people like me who actually build these things are = going to ignore an architectural focus because that has no value. =A0But ha= ving ACCEPTED STANDARD tests would be very valuable.

E.g. a good, standard latency-under-load test that can be written in Fl= ash (so TCP/UDP connections only to cooperating servers) would be great. = =A0Our current one in Netalyzr is only OK: we believe that one could do a l= ot better.

If you could then figure out WHERE the bottleneck is (using a server th= at can also do simultaneous traceroutes), this would be even more valuable.=

_______________________________________________
lmap mailing lis= t
lmap@ietf.org
https= ://www.ietf.org/mailman/listinfo/lmap

<= br>


On Mon, Nov 19, 2012 at 1:58 PM, STARK, BARBARA H=A0<<= a href=3D"mailto:bs7652@att.com" target=3D"_blank">bs7652@att.com>=A0wrote:
=A0[--snip--]
> > Here are some specific comments on the arch= itectural elements:
> > Measurement client: I agree that if measur= ements (active or passive) are
> > going to be made, then one or m= ore Measurement clients are a mandatory
> > element. I agree with the first 3 sentences of the Measurement cl= ient
> > definition.
>
> Well 3 out of 4 (sentences) i= s a good start. :-)
>
>
> > But the idea that *all* Me= asurement clients must be manageable and
> > "provide for communications within different network segment= s" is not
> > something I agree with.
>
> I'm= not sure I detect an implication that all measurement clients must be
> manageable. =A0What the fourth sentence says is the following:
>= "Client measurement functionality must be implementable in a variety = of
> user contexts ..."

<bhs> As I mentioned,= I thought it was possible to agree on the existence of a Measurement Clien= t (as defined by the first 3 sentences) as an architectural element. The pr= oblem really arises with the assumptions as to the nature of this element t= hat are implied by certain of the requirements:

=A0 =A0A-2: =A0Measurement clients and servers MUST support an extensib= le set
=A0 =A0 =A0 of performance metrics.
=A0 =A0A-4: =A0Measurement= clients MUST be able perform both active and
=A0 =A0 =A0 passive measur= ements.
=A0 =A0M-3: =A0A measurement client MUST be able to register wit= h the data
=A0 =A0 =A0 collection platform automatically, announcing its availability = and
=A0 =A0 =A0 relevant system parameters. =A0(For example, a cable or = DSL modem
=A0 =A0 =A0 may indicate its make and model number.)
=A0 = =A0M-4: =A0A measurement client MUST be able to declare what kind of
=A0 =A0 =A0 measurements it can perform, e.g., by enumerating a set of
= =A0 =A0 =A0 measurement identifiers.
=A0 =A0C-4: =A0A measurement client= MUST be able to authenticate and authorize
=A0 =A0 =A0 the measurement = controller.

As I mentioned in my last email, I believe Measurement C= lients may exist inside ONTs, DSLAMs, and even very, very simple DSL modems= (some of which are nothing more than a USB dongle that connects to a PC, a= nd many of which are completely unmanaged, except locally by the end user).= The idea that these devices / network elements must support these requirem= ents in order to participate in a holistic scheme/architecture to collect m= eaningful performance data is something I do not agree with. I also believe= that it is possible for a Management Client to be directly under the contr= ol of the end user, and potentially not be upgradable or capable of an exte= nsible set of tests. I believe that the Windows Command Prompt is a very ef= fective (for me) end-user-controlled Measurement Client for performing a va= riety of tests. But it does not meet the above requirements. Nor should it = be expected to. I use it often, though, for my on-demand connectivity and p= erformance testing.

To me, this architecture draft assumes that a "federated" arc= hitecture is necessary for useful collection of performance measurements an= d conducting tests. I disagree with that premise. There also seems to be an= assumption that a "federated" architecture must be defined prior= to defining appropriate metrics. I disagree. I believe that a federated ar= chitecture (that allows for all sorts of management and control and updatin= g of tests/schedules/etc.) is unnecessary. I believe it is possible to re-u= se a lot of what exists, with minimal additions, to accomplish large scale = performance testing and on-demand diagnostics, that meet the basic "ne= eds" (not to be confused with "wants") of various parties. I= believe it is possible for 3rd parties and ANPs to develop one-to-one rela= tionships with an end user, to allow for on-demand and scheduled testing an= d collection of test data. The end user can even allow for 3rd parties to a= ccess tools under end-user control, which interact with ANP
=A0 Measurement Servers (taking advantage of the end user to ANP relationsh= ip). For large scale performance data -- I believe it is possible to collec= t such data and provide access to the resulting anonymized data without ena= bling 3rd party management and control of the elements involved in data col= lection.

My recommendation would be to focus on defining appropriate metrics. On= ce we understand the metrics, we can work on understanding the appropriate = measurements and tests, and the set of Measurement Clients and Measurements= Servers that could be used to perform those measurements and tests. If som= ething else is needed from a standards organization, that should become evi= dent once the metrics are identified.
Barbara
__________= _____________________________________
lmap mailing list
lmap@ietf.org




On Mon, Nov 19, 2012 at 4:52 PM, STARK, BARBARA H=A0<<= a href=3D"mailto:bs7652@att.com" target=3D"_blank">bs7652@att.com>=A0wrote:
[...]

This architecture seems to imply that it'= ;s relevant to the end user on-demand test case. My favorite test is "= ping". I do my ping testing against a variety of Internet websites, bu= t also my access/Internet provider's DNS servers and IP gateways. What = I ping depends on what my problem is. Anything that responds to my ping is = a Measurement Server in my book.

As an end user, I have no need for the set of tests that such Measureme= nt Servers support.
I also have no need for my client to authenticate wi= th them. I don't think they've ever required authentication of my c= lient. I have no idea if they communicate with a Data Collector, and I real= ly don't care.

Similarly, I often use "bandwidth test" websites for on-deman= d bandwidth tests. I really don't care whether they're extensible, = authentication has never been an issue, and I have no requirement for them = to communicate with a Data Collector

> ---snip---
> The following requirements are applicable = to the overall LMAP Reference
> Architecture, as described in this do= cument. =A0How each of these
> requirements are applied to specific i= mplementation of a particular type,
> category or class of measurement client, server, etc. device is out-of= -scope
> for this document.
> ---snip---
> Is the above t= ext good enough, for now?

<bhs> No. Unless the described= architectural elements are changed to be "LMAP Measurement Client&quo= t; and "LMAP Measurement Server", with definition of these being = that an LMAP Measurement Client/Server is one that meets the requirements e= numerated in this document. I don't think it's possible to go from = the generic architecture elements of Measurement Client/Server to the enume= rated requirements that seem to have something very specific in mind that i= s not at all consistent with the generic definitions provided in the archit= ecture section.

> But, going back to Section 3.2, Measurement Server, I can see= how it may be
> useful to add some of the points you raised above. = =A0How about suggesting to
> add something like the following to Sect= ion 3.2?
> ---snip---
> Measurement servers likely will be owned and operat= ed by a variety of
> administrative entities, some of which may be un= related to the ISP providing
> service to a measurement client. =A0Th= erefore, measurement servers may
> need a facility where they can describe their capability to perform on= e, or
> more, suites of active performance measurement tests to a mea= surement
> controller. In this way, the measurement controller may fa= cilitate
> coordination of measurement clients to measurement servers that have> matching active performance measurement capabilities.
> ---sni= p---

That still doesn't allow for a website that responds = to a ping to be considered a Measurement Server. Or a website that provides= bandwidth measurements. But they are Measurement Servers. They just aren&#= 39;t Measurement Servers that meet these "LMAP" requirements. Do = any of the use cases create a need for all Measurement Servers (used by tha= t use case) to meet these requirements? I don't think so. Even the 3rd = party testing use case should allow for ping tests without insisting that t= he server supporting the ping tests meet these requirements.

This LMAP architecture is assuming that a requirement of the architectu= re is to support all use cases, within a single architecture. I disagree wi= th that premise.
And I disagree that 3rd party management and control of= "LMAP" architecture elements is a compelling use case. The proto= cols may be simple, but it's incredibly complex to manage and administe= r, and to put in all the controls needed to prevent end users from being ne= gatively impacted by 3rd parties. A simple requirement of "make sure 3= rd parties can't break anything" is insufficient and ignores the f= act that ensuring this is incredibly complex and difficult.
Barbara
__________= _____________________________________
lmap mailing list
lmap@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/lmap
<= br>

--485b397dd3b503063e04cef105ae-- From bs7652@att.com Sat Nov 24 14:20:52 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CB49B21F84C6 for ; Sat, 24 Nov 2012 14:20:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FW6a52KKEMb7 for ; Sat, 24 Nov 2012 14:20:52 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id EBC6621F8442 for ; Sat, 24 Nov 2012 14:20:51 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 34841b05.0.1873113.00-368.5115212.nbfkord-smmo08.seg.att.com (envelope-from ); Sat, 24 Nov 2012 22:20:51 +0000 (UTC) X-MXL-Hash: 50b14843433adb6c-8502b0c1f0fdf6905fdd4f49a26d1b396b724281 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAOMKpWj018635 for ; Sat, 24 Nov 2012 17:20:51 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAOMKjnd018570 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 24 Nov 2012 17:20:46 -0500 Received: from GAALPA1MSGHUB9A.ITServices.sbc.com (gaalpa1msghub9a.itservices.sbc.com [130.8.36.87]) by sflint04.pst.cso.att.com (RSA Interceptor) for ; Sat, 24 Nov 2012 17:20:04 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9A.ITServices.sbc.com ([130.8.36.87]) with mapi id 14.02.0318.001; Sat, 24 Nov 2012 17:20:03 -0500 From: "STARK, BARBARA H" To: "lmap@ietf.org" Thread-Topic: Is there a requirement for combined management and control architecture? Thread-Index: Ac3Kkdi1EUw3eU8MSJWjD4h8PhFVmA== Date: Sat, 24 Nov 2012 22:20:02 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6112F5E3784@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [130.10.170.141] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=OqD4PVDt c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=YRHmaC0L4fkA:10 a=tqfkoo-rZPgA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=LSRi9OjKplwA:10 a=8rfRQ_BatVZl3UJOOUMA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Subject: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Nov 2012 22:20:53 -0000 Thanks, James, for the email you sent. It really helped to improve my under= standing of what you believe to be important to a regulatory use case, and = explain why you put certain requirements into the draft. If I read your email correctly, I believe that you support separating discu= ssion of recommendations for measurements and metrics, from discussions on = management and control of the elements participating in metric data collect= ion and dissemination. I agree with this. I believe that the draft that you= co-authored was very much focused on a management and control architecture= , so I am going to focus my comments now purely on requirements of manageme= nt and control architectures for measurement collection architectures of th= e various use cases we have been considering. There is a key element of the draft that I disagree with, that I would like= to better understand your thoughts on. It appears to me that the draft attempts to combine the management and cont= rol requirements for all use cases (end user, access network provider, appl= ication service provider, regulatory/research) into a single management and= control architecture (that therefore imposes the requirements of the most = complex use case on all use cases). This appears to me to be a core assumpt= ion of the draft, and is my fundamental objection to the draft. This assump= tion is made without any attempt to justify it, and I saw nothing in your e= mail that would justify it. In my opinion, just because all of the use case= s are "good" or "valid", does not imply that combined management and contro= l of them is desirable or good. Most of the statements from my prior emails= were to explain why I find (mandated) combination of management and contro= l architectures to be undesirable. To me, the requirements of the different= use cases all seem to be very independent of each other.=20 Why do you (or do you?) consider it important that a single management and = control architecture be defined that meets the combined needs of all use ca= ses? If the draft authors would agree to consider the management and control arc= hitecture requirements of each use case independently, I think we might be = able to progress to doing such independent requirement assessment. But if t= he draft authors insist that they must be combined into a single architectu= re, then I would like for some attempt to be made to provide a convincing r= eason for their combination. Barbara From acmorton@att.com Sun Nov 25 06:21:15 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5462021F8461; Sun, 25 Nov 2012 06:21:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.46 X-Spam-Level: X-Spam-Status: No, score=-105.46 tagged_above=-999 required=5 tests=[AWL=-0.783, BAYES_00=-2.599, HTML_MESSAGE=0.001, MIME_HTML_ONLY=1.457, RCVD_IN_DNSWL_MED=-4, SARE_UNSUB31=0.464, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ADndwNkjL50t; Sun, 25 Nov 2012 06:21:11 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id 8ADFB21F8465; Sun, 25 Nov 2012 06:21:03 -0800 (PST) Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id b4922b05.0.24520.00-489.64368.nbfkord-smmo08.seg.att.com (envelope-from ); Sun, 25 Nov 2012 14:21:08 +0000 (UTC) X-MXL-Hash: 50b229543d1d382f-42ac44bc3616803a49e9031e439d84aa26548599 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAPEKxkj011567; Sun, 25 Nov 2012 09:20:59 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAPEKogZ011539; Sun, 25 Nov 2012 09:20:55 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAPEEoVK001791; Sun, 25 Nov 2012 09:14:50 -0500 Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAPEEio8001744; Sun, 25 Nov 2012 09:14:45 -0500 Received: from lt-hp1044652.att.com (vpn-135-70-161-25.vpn.mwst.att.com[135.70.161.25](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121125141442gw100632iqe>; Sun, 25 Nov 2012 14:14:50 +0000 X-Originating-IP: [135.70.161.25] Message-Id: <7.0.1.0.0.20121125083420.085ee458@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Sun, 25 Nov 2012 09:12:36 -0500 To: Brian Trammell , Nicholas Weaver From: Al Morton Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.145] X-AnalysisOut: [v=2.0 cv=TspRckrh c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a] X-AnalysisOut: [=xm6Tbr5ENoQA:10 a=2xSh8RXx3toA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=glbQQ1A4] X-AnalysisOut: [_rkA:10 a=48vgC7mUAAAA:8 a=n8IGn1O_320mK60UEwYA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10 a=_W_S_7VecoQA:10 a=lZB815dzVvQA:10 a=zWZLj9ytA2] X-AnalysisOut: [irFZkr:21 a=XMF_64RsddoOS0_M:21 a=h7Qxi_pJRNgTow2r:21] Cc: lmap@ietf.org, ippm@ietf.org Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 14:21:15 -0000 LMAP and IPPM,

After catching-up on many messages, I would like to add a key
consideration and area for discussion and agreement for
development of standardized tests:

- For each test we eventually specify, it's necessary to specify
  the equivalent measurement points along the user's packet transfer
  path/architecture of each different access technology.

This effort certainly needs folks with standards experience in all of the
access architectures to complete the task, because working out
measurement point details will be a critical aspect of conducting comparable
measurements. So, for anyone who read Nick's and Brian's messages
below (which I mostly agree with) and was about to un-subscribe or add "lmap"
to your spam filter, don't.  There's plenty to do here that's
architecture-related. The various measurement points needed depend
on the list of tests of course, and that list needs discussion, too.

As an example needing some discussion:
All packet-transfer services have a demarcation point where the customer
connects their own equipment to the network. It is valuable from both the
diagnostic and service verification points of view to have a measurement
point very close to the demarcation point. But direct access to this
point is often impossible for a measurement agent - where can we agree
are the functionally equivalent alternatives in each access architecture?
(don't forget wireless access)

regards,
Al

At 11:59 AM 11/20/2012, Brian Trammell wrote:
Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:
>
> This focus on architecture is deeply misguided.  As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.
>
> In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.  
>
> The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis). 
>
> Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.  Our android version will just use the app store update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly). 
>
> And all three end up having rather different architectural requirements.

Architectures, I would submit (and as I've said here before), are useful to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same isn't the case for the Ripe Atlas or Bismark platform.
>
> And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.  Netalyzr is currently composed of >100 distinct tests.  And this amount just continues to grow.
>
>
> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build these things are going to ignore an architectural focus because that has no value.  But having ACCEPTED STANDARD tests would be very valuable. 

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary.

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.  Our current one in Netalyzr is only OK: we believe that one could do a lot better.
>
> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load).

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm
From mattmathis@google.com Sun Nov 25 13:20:54 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3A03321F8446 for ; Sun, 25 Nov 2012 13:20:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.435 X-Spam-Level: X-Spam-Status: No, score=-102.435 tagged_above=-999 required=5 tests=[AWL=0.077, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_UNSUB31=0.464, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zMYs5AI18wAW for ; Sun, 25 Nov 2012 13:20:52 -0800 (PST) Received: from mail-we0-f172.google.com (mail-we0-f172.google.com [74.125.82.172]) by ietfa.amsl.com (Postfix) with ESMTP id DE26121F843F for ; Sun, 25 Nov 2012 13:20:51 -0800 (PST) Received: by mail-we0-f172.google.com with SMTP id r3so2828158wey.31 for ; Sun, 25 Nov 2012 13:20:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=U+6EAnArwL2485LEb/QF/nUGekUh7Ht5xC+FKYuagOw=; b=MCAOInBxXeSJs+vh77qyYGjv2eTCzwan1hmTyEuEXQ53Psy7+/7eA1K8iSShliTQ0q iIVs1qIMT06m0Umt+5ttUzNjos6tBWZqoPAPwVeKZDLLiTwlFPx++3MPQ2QzpJL990wy 7pCK+RBhUNZFU4RDK1ZglTm3ykEo1MneU05/4VhWkyVErvjOTTo9bF6gR9T6j4np8kVr w8a8q0MWJZ8TUQyeRfRv29NebM4E3tIRlGVOZhAC5Z+GjhGrfKCUb4SMxKSyWWH0l/0P jRajvbYuI6PKCbfIJvtc66sQeJmTf1M+64x5lmN5VM5AAb/nU0qJyrKF2IYw6cs7fVBa uqFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=U+6EAnArwL2485LEb/QF/nUGekUh7Ht5xC+FKYuagOw=; b=CCrXu34nFfEr5Cs3fQfHapncb4yXEqrl6ZWalXnRTfWDSkpl6aZ5cMv6aGB/EAhbrJ +lHNRowB/GLmSOqwTX/xXwHXAohOwLPIZMAHo2/ZtiE5tcuu7X8QE3v2PXfRM/9MXrOW cK8gVmRvJwiZRP2CkBlQf4FQlfTdyy7eTGFQgX0FCbFoN58k/ju+15plXR5IPwcOf2Cb F+CazV9nqTNKkEGcVMBIB8i91ZH1r/3RiRH1seKEOTml30a7fnsHxMVns1ePG2T9AfVU lnOfZh2jU71k3RI4IXZ05SzAyneLGnx3cqolUnuWJFXa8apoCLGn29i4p/LpnIa17+Qp WYAg== MIME-Version: 1.0 Received: by 10.180.100.132 with SMTP id ey4mr17844864wib.9.1353878450502; Sun, 25 Nov 2012 13:20:50 -0800 (PST) Received: by 10.227.9.71 with HTTP; Sun, 25 Nov 2012 13:20:50 -0800 (PST) In-Reply-To: <7.0.1.0.0.20121125083420.085ee458@att.com> References: <7.0.1.0.0.20121125083420.085ee458@att.com> Date: Sun, 25 Nov 2012 13:20:50 -0800 Message-ID: From: Matt Mathis To: Al Morton Content-Type: multipart/alternative; boundary=f46d0444ec311c7ea904cf58646f X-Gm-Message-State: ALoCoQlvXQNVTu+6ruH7nZGe13/3puwJe3P1rI+DkfNGW7qz6AOrohWJm5eXPnyzTHRaYcU0VzVG7568fF7AaXvSMxSPmv31dPwIhs/OhgKXzPaopPCD83OMtsuRkdmOIcsYTvbHeoPqlIrs8p6RIa7t72y9/bY/fhC0VtANAW1uw2XPVkItxdV187ZVolwGRvMwVLuHMQOu Cc: lmap@ietf.org, Nicholas Weaver , ippm@ietf.org, Brian Trammell Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 21:20:54 -0000 --f46d0444ec311c7ea904cf58646f Content-Type: text/plain; charset=ISO-8859-1 I would like to disagree here. Another missing requirement from 2330 is that measurement vantage point should not matter. Consider the standards for milk: milk fat is a parameter that anybody can measure (producer, consumer, independent lab, school classroom, etc). It can be process controlled by the manufacturer (tweaking centrifuge settings), and is useful to the consumer. As an extremely robust metric it stabilizes the industry and brings sanity to the marketplace. A metric that can only be measured by the producer and does not agree with the users intuition as to the value they are getting for their money will always be perceived as crooked, even if it is not. lmap is a logical step down this path..... What we really need are better metrics. The Model Based Metrics ID is my first attempt here, although the words in the draft fall far short of telling the right story. I have started a "Model Based Metrics working doc" at https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit# I believe that the "Sustained burst test" at the bottom is (close to) the metric that we need. It has (is expected to have) a whole list of cool properties, including actionable by the ISPs, useful to the users, RTT insensitivity, and consequently vantage insensitivity. BTW: I enable "public comment" on the doc, so all of you can add margin notes. I would like to request an extended slot in IPPM, to present this. It should be in a proper ID by that time. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. On Sun, Nov 25, 2012 at 6:12 AM, Al Morton wrote: > LMAP and IPPM, > > After catching-up on many messages, I would like to add a key > consideration and area for discussion and agreement for > development of standardized tests: > > - For each test we eventually specify, it's necessary to specify > the equivalent measurement points along the user's packet transfer > path/architecture of each different access technology. > > This effort certainly needs folks with standards experience in all of the > access architectures to complete the task, because working out > measurement point details will be a critical aspect of conducting > comparable > measurements. So, for anyone who read Nick's and Brian's messages > below (which I mostly agree with) and was about to un-subscribe or add > "lmap" > to your spam filter, don't. There's plenty to do here that's > architecture-related. The various measurement points needed depend > on the list of tests of course, and that list needs discussion, too. > > As an example needing some discussion: > All packet-transfer services have a demarcation point where the customer > connects their own equipment to the network. It is valuable from both the > diagnostic and service verification points of view to have a measurement > point very close to the demarcation point. But direct access to this > point is often impossible for a measurement agent - where can we agree > are the functionally equivalent alternatives in each access architecture? > (don't forget wireless access) > > regards, > Al > > > At 11:59 AM 11/20/2012, Brian Trammell wrote: > > Hi, Nick, > > (Copying this over to IPPM as well, as the focus-on-metrics discussion > will be interesting there too). > > On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote: > > > I've said it once, I'll say it one final time: > > > > This focus on architecture is deeply misguided. As someone who's built > one such architecture (Netalyzr), working on two others (Fathom, a browser > extension, and techniques using GuruPlugs/Raspberry Pis), the architecture > is the easy part. > > > > In all this work, having some "standard" way to communicate between my > measurement clients and servers is useless. I just use HTTP, HTTPS, or SSH > depending on whether quick & easy or server side or mutual authentication > is required, with whatever load-balancing is needed handled by random > assignment of remote test node. > > > > The internal data store is whatever is appropriate for the project (eg. > for Netalyzr its python pickled objects for storage on our web server, data > in a Postgress database for analysis). > > > > Updates are handled appropriately (Netalyzr is updateless, its an applet > so its always up-to-date. Our android version will just use the app store > update mechanism. Fathom we use Firefox's update mechanism. The plugs are > a hack using SSH, but its a fair amount of custom goop, and they update > nightly). > > > > And all three end up having rather different architectural requirements. > > Architectures, I would submit (and as I've said here before), are useful > to hang terminology off of: you need boxes and lines so everyone > understands (roughly) the same thing when you say "client" and "server" and > "controller" with reference to a set of measurements you define. > > *But yes, specifying a way to specify a measurement is largely arbitrary. > I'd even advocate separating the representation from the transport, > especially as the general problem of measurement interoperability has data > sources operating and wildly different scales, but as you say deciding this > is the easy part. > * > > The hard part is developing the test metrics and measurements, both in > the abstract and within the API constraints of the target platform. E.g. > why I like the GuruPlugs and Raspberry Pi is that I can run Java, so > therefore I can just "Run Netalyzr" for all Netalyzr tests. But the same > isn't the case for the Ripe Atlas or Bismark platform. > > > > And Marc's list is actually just a start: you also have generic proxy > presence & location detection, application specific proxy detection and > analysis, packet injection detection, differential application behavior > detection, application agnostic traffic shaping detection, DNSSEC support, > DNS manipulation detection, path MTU, IPv6 for all of the previous, and a > ton of others. Netalyzr is currently composed of >100 distinct tests. And > this amount just continues to grow. > > > > > > If the IETF actually wants to have an impact in this area, don't bother > focusing on protocols for communication, focus on tests with given > attributes. Because people like me who actually build these things are > going to ignore an architectural focus because that has no value. But > having ACCEPTED STANDARD tests would be very valuable. > > + 1e. Stated another way, the protocol gives you the > grammar. What you need to make this at all interoperable is the vocabulary. > > IPPM has defined a several of these at a low level, with a focus on active > measurement methods that produce comparable results. There appears to be > interest in IPPM both in adding passive methods for the existing metrics > (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more > complex metrics based in part on the simple ones (see > draft-mathis-ippm-model-based-metrics). > > > E.g. a good, standard latency-under-load test that can be written in > Flash (so TCP/UDP connections only to cooperating servers) would be great. > Our current one in Netalyzr is only OK: we believe that one could do a lot > better. > > > > If you could then figure out WHERE the bottleneck is (using a server > that can also do simultaneous traceroutes), this would be even more > valuable. > > (This seems to make the assumption that traceroute-measurable latency is > strongly correlated with application latency at a given hop, but I get your > point...) > > The efforts of IPPM to date have been focused on simply describable, > easily repeatable metrics... there's not a lot of specification of > arbitrarily complex or unknown preconditions (as would be useful in the > latency-under-load example you give), because these tend to be counter to > the goals of maximizing repeatability and minimizing uncertainty. There's > also a focus on active measurement for the same reason. *On the other > hand, a lot of work in distributed measurement systems is focused on > measuring the network as it is, with implicit or explicit passive > components (again, returning to "latency under load", the load traffic will > be made up of generated as well as existing background load). > * > *So, what might be useful would be to open another line of work within > IPPM on interoperable descriptions of some of these existing tests as > empirically specified metrics, perhaps with relaxed compliance with the > IPPM framework as appropriate.* This could be used to provide a more > complex vocabulary to LMAP, but could also be useful independently -- *if > LMAP is eventually defined to apply only to a restricted set of use cases > (i.e. specifically mandated tests for regulatory compliance, but not > research or troubleshooting), comparable metrics with incompatible > architecture and protocols could still lead to wider interoperability in > the measurement space*, the need for protocol shims (which are after all > easy to write if the number inside has the same meaning) aside. > > Best regards, > > Brian > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm > > > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm > > --f46d0444ec311c7ea904cf58646f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I= would like to disagree here. =A0 Another missing requirement from 2330 is = that measurement vantage=A0point=A0should not matter. =A0 Consider the stan= dards for milk: milk fat is a parameter that anybody can measure (producer,= consumer, independent lab, school classroom, etc). =A0It can be process co= ntrolled by the=A0manufacturer (tweaking centrifuge settings), and is usefu= l to the consumer. =A0As an extremely robust metric it=A0stabilizes=A0the i= ndustry and brings sanity to the marketplace.

A metric that can only be measured by the producer and does = not agree with the users intuition as to the value they are=A0getting for t= heir money will always be=A0perceived=A0as crooked, even if it is not. =A0 = lmap is a logical step down this path.....

What we really need are better metrics.

<= /div>
The Model Based Metrics ID is my first attempt here, although the= words in the draft fall far short of telling the right story. =A0 I have s= tarted a "Model Based Metrics working doc" at=A0https://docs.google.com/document/d/1KDJMuZ2cWfXbnWj= gwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#=A0 =A0 I believe that the "Sust= ained burst test" at the bottom is (close to) the metric that we need.= =A0It has (is expected to have) a whole list of cool properties, including= actionable by the ISPs, useful to the users, RTT=A0insensitivity, and cons= equently vantage insensitivity.

BTW: I enable "public comment" on the doc, so= all of you can add margin notes.

I would like to = request =A0an extended slot in IPPM, to present this. =A0It should be in a = proper ID by that time.

Thanks,
--MM--
The best way to pred= ict the future is to create it. =A0- Alan Kay

Privacy matters! =A0We= know from recent events that people are using our services to speak in def= iance of unjust governments. =A0 We treat privacy and security as matters o= f life and death, because for some users, they are.


On Sun, Nov 25, 2012 at 6:12 AM, Al Mort= on <acmorton@att.com> wrote:
Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:
>
> This focus on architecture is deeply misguided.=A0 As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.
>
> In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.=A0 I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.=A0=A0
>
> The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis).=A0
>
> Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.=A0 Our android version will just use the app store update mechanism.=A0 Fathom we use Firefox's update mechanism.=A0 The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly).=A0
>
> And all three end up having rather different architectural requirements.

Architectures, I would submit (and as I've said here before), are usefu= l to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.=A0 E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.=A0 But the same isn't the case for the Ripe Atlas or Bismark platform.
>
> And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.=A0 Netalyzr is currently composed of >100 distinct tests.=A0 And this amount just continues to grow.
>
>
> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.=A0 Because people like me who actually build these things are going to ignore an architectural focus because that has no value.=A0 But having ACCEPTED STANDARD tests would be very valuable.=A0

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary.

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.=A0 Our current one in Netalyzr is only OK: we believe that one could do a lot better.
>
> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There'= ;s also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load).

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm

_______________________________________________
ippm mailing list
ippm@ietf.org
ht= tps://www.ietf.org/mailman/listinfo/ippm


--f46d0444ec311c7ea904cf58646f-- From acmorton@att.com Sun Nov 25 21:23:24 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4A64621F85CB; Sun, 25 Nov 2012 21:23:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.077 X-Spam-Level: X-Spam-Status: No, score=-102.077 tagged_above=-999 required=5 tests=[BAYES_50=0.001, HTML_MESSAGE=0.001, MIME_HTML_ONLY=1.457, RCVD_IN_DNSWL_MED=-4, SARE_UNSUB31=0.464, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cjJRBFvCecMA; Sun, 25 Nov 2012 21:23:21 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 7F16821F85C2; Sun, 25 Nov 2012 21:23:17 -0800 (PST) Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 4ccf2b05.0.136702.00-494.369389.nbfkord-smmo06.seg.att.com (envelope-from ); Mon, 26 Nov 2012 05:23:20 +0000 (UTC) X-MXL-Hash: 50b2fcc87d2c1681-9513ecd829a305884c75a3faa518ebc1bb83ea57 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAQ5NGAT015038; Mon, 26 Nov 2012 00:23:16 -0500 Received: from sflint02.pst.cso.att.com (sflint02.pst.cso.att.com [144.154.234.229]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAQ5NDDU015030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 26 Nov 2012 00:23:13 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by sflint02.pst.cso.att.com (RSA Interceptor); Mon, 26 Nov 2012 00:22:57 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAQ5MvZd012596; Mon, 26 Nov 2012 00:22:57 -0500 Received: from mailgw1.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qAQ5Mmjc012511; Mon, 26 Nov 2012 00:22:53 -0500 Received: from lt-hp1044652.att.com (vpn-135-70-161-25.vpn.mwst.att.com[135.70.161.25](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121126052250gw100632ive>; Mon, 26 Nov 2012 05:22:51 +0000 X-Originating-IP: [135.70.161.25] Message-Id: <7.0.1.0.0.20121125225022.085ee310@att.com> X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0 Date: Mon, 26 Nov 2012 00:20:44 -0500 To: Matt Mathis From: Al Morton In-Reply-To: References: <7.0.1.0.0.20121125083420.085ee458@att.com> Mime-Version: 1.0 Content-Type: text/html; charset="us-ascii" X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.145] X-AnalysisOut: [v=2.0 cv=AMgvexcp c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a] X-AnalysisOut: [=xm6Tbr5ENoQA:10 a=2xSh8RXx3toA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=glbQQ1A4] X-AnalysisOut: [_rkA:10 a=hZG83p_yAAAA:8 a=B6KMzFptAAAA:20 a=48vgC7mUAAAA:] X-AnalysisOut: [8 a=oz9-V9JTie3uQvoq9_oA:9 a=CjuIK1q_8ugA:10 a=_W_S_7VecoQ] X-AnalysisOut: [A:10 a=Hz7IrDYlS0cA:10 a=lZB815dzVvQA:10 a=ALAIfxRDjUGEzYC] X-AnalysisOut: [9:21 a=Ar26coLjBKrL5sK-:21 a=oJVC8dsoSXraIRNQ:21] Cc: lmap@ietf.org, Nicholas Weaver , ippm@ietf.org, Brian Trammell Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 05:23:24 -0000 Matt,

I think we agree that better metrics are needed for LMAP,
and that ideally a metric can be applied at any point or pair of points.

RFC 2330 and most IPPM metrics have a decidedly Source-to-Destination
host construction. Frankly, we did a better job with arbitrary
measurement points in Rec Y.1540. It's worth a look:
http://www.itu.int/rec/T-REC-Y.1540/en

There are many goals the new metrics could serve:
 - externally observable
 - measurable by users
 - easily related to user application performance
 - service verification
 - support consumer choice of ???
 - etc.
We need to agree on the set that are needed.

While there may be some information in a measurement that covers
a combination of networks, measuring more than one network
is only useful when the combined network path delivers
the target performance - the sunny day outcome. Otherwise,
they are inconclusive, as you said in your memo. Some target
performance levels only apply in the scope where they are offered.

We don't really have the milk fat scenario here, unless the producer,
the truck driver, the convenience store, and your home refrigerator
can all add some unknown amount of fat to the milk. ;-) 

When considering this problem, I've been thinking of the fuel
efficiency ratings provided for new cars in the US and Europe (at least).
The specs illustrate very well that the quantity measured is variable
depending on the conditions of the measurement (in the US, the miles per
gallon for city and highway driving), and they imply the
range of values that normal users can experience (a fairly wide range).
These are some aspects that would be useful to reproduce in our
metrics, I think.

regards, and YMMV,
Al


At 04:20 PM 11/25/2012, Matt Mathis wrote:
I would like to disagree here.   Another missing requirement from 2330 is that measurement vantage point should not matter.   Consider the standards for milk: milk fat is a parameter that anybody can measure (producer, consumer, independent lab, school classroom, etc).  It can be process controlled by the manufacturer (tweaking centrifuge settings), and is useful to the consumer.  As an extremely robust metric it stabilizes the industry and brings sanity to the marketplace.

A metric that can only be measured by the producer and does not agree with the users intuition as to the value they are getting for their money will always be perceived as crooked, even if it is not.   lmap is a logical step down this path.....

What we really need are better metrics.

The Model Based Metrics ID is my first attempt here, although the words in the draft fall far short of telling the right story.   I have started a "Model Based Metrics working doc" at https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#     I believe that the "Sustained burst test" at the bottom is (close to) the metric that we need.  It has (is expected to have) a whole list of cool properties, including actionable by the ISPs, useful to the users, RTT insensitivity, and consequently vantage insensitivity.

BTW: I enable "public comment" on the doc, so all of you can add margin notes.

I would like to request  an extended slot in IPPM, to present this.  It should be in a proper ID by that time.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our services to speak in defiance of unjust governments.   We treat privacy and security as matters of life and death, because for some users, they are.


On Sun, Nov 25, 2012 at 6:12 AM, Al Morton <acmorton@att.com> wrote:
LMAP and IPPM,

After catching-up on many messages, I would like to add a key
consideration and area for discussion and agreement for
development of standardized tests:

- For each test we eventually specify, it's necessary to specify
  the equivalent measurement points along the user's packet transfer
  path/architecture of each different access technology.

This effort certainly needs folks with standards experience in all of the
access architectures to complete the task, because working out
measurement point details will be a critical aspect of conducting comparable
measurements. So, for anyone who read Nick's and Brian's messages
below (which I mostly agree with) and was about to un-subscribe or add "lmap"
to your spam filter, don't.  There's plenty to do here that's
architecture-related. The various measurement points needed depend
on the list of tests of course, and that list needs discussion, too.

As an example needing some discussion:
All packet-transfer services have a demarcation point where the customer
connects their own equipment to the network. It is valuable from both the
diagnostic and service verification points of view to have a measurement
point very close to the demarcation point. But direct access to this
point is often impossible for a measurement agent - where can we agree
are the functionally equivalent alternatives in each access architecture?
(don't forget wireless access)

regards,
Al


At 11:59 AM 11/20/2012, Brian Trammell wrote:
Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:
>
> This focus on architecture is deeply misguided.  As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.
>
> In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.  
>
> The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis). 
>
> Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.  Our android version will just use the app store update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly). 
>
> And all three end up having rather different architectural requirements.

Architectures, I would submit (and as I've said here before), are useful to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same isn't the case for the Ripe Atlas or Bismark platform.
>
> And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.  Netalyzr is currently composed of >100 distinct tests.  And this amount just continues to grow.
>
>
> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build these things are going to ignore an architectural focus because that has no value.  But having ACCEPTED STANDARD tests would be very valuable. 

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary.

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.  Our current one in Netalyzr is only OK: we believe that one could do a lot better.
>
> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load).

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian
_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm

_______________________________________________
ippm mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm

From shane@castlepoint.net Mon Nov 26 20:40:06 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EEA0F21F84D0 for ; Mon, 26 Nov 2012 20:40:06 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.437 X-Spam-Level: X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Am21Pv3FYoX0 for ; Mon, 26 Nov 2012 20:40:06 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 45ED721F8442 for ; Mon, 26 Nov 2012 20:40:06 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 05A9B21BC for ; Tue, 27 Nov 2012 04:40:05 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id C42EF21B7; Mon, 26 Nov 2012 21:40:03 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: Date: Mon, 26 Nov 2012 21:40:02 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5DD2D9@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6112F5DD81F@GAALPA1MSGUSR9L.ITServices.sbc.com> <52C5FD9F-E083-4075-AE24-763306DBBFA7@icsi.berkeley.edu> <2D09D61DDFA73D4C884805CC7865E6112F5DD96D@GAALPA1MSGUSR9L.ITServices.sbc.com> To: KEN KO X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Mon Nov 26 21:40:05 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b44425199631207171716 X-DSPAM-Factors: 27, Mime-Version*OS+X, 0.01000, a+#+#+#+that, 0.01000, a+#+#+#+that, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, a+single, 0.01000, From*Amante+shane, 0.01000, there+#+be, 0.01000, this+is, 0.01000, as+well, 0.01000, the+#+#+the, 0.01000, 2012+at, 0.01000, Mime-Version*Mail+#+1499, 0.01000, com+wrote, 0.01000, I+#+#+#+a, 0.01000, be+#+to, 0.01000, be+#+to, 0.01000, of+those, 0.01000, of+those, 0.01000, want+to, 0.01000, Mime-Version*6.2+1499, 0.01000, for+the, 0.01000, tests+may, 0.01000, of+#+#+are, 0.01000, the+#+#+is, 0.01000, I+believe, 0.01000, to+the, 0.01000 Cc: Marc Linsner , "lmap@ietf.org" , Nicholas Weaver , "STARK, BARBARA H" , "ippm@ietf.org" Subject: Re: [lmap] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 04:40:07 -0000 On Nov 20, 2012, at 10:53 AM, KEN KO wrote: >> Since client and controller are deployed by the same entity, it = should be up to that entity to figure out their control architecture, = and that entity should have full say in how to do their own control = architecture. >=20 > I don't think it's a universally held assumption that the client and = controller are deployed by the same entity. +1. > For instance, a regulator use case spanning multiple operators could = use a controller function (and, a data collection function) that = communicates with clients deployed by each of those operators. So I = don't think we should ignore architecture. Agreed. I also want to stress that there are other applicable, = non-regulatory use cases, as well, where (for example) a single = controller speaks to LMAP measurement clients deployed across a = multitude of operators. > That said, I do believe that focusing on tests is an excellent idea = and that that work could progress in ippm while other lmap issues are = still being discussed. Agreed. I'd also add that there may be two solution spaces emerging wrt = "metrics" (in IPPM) that we discover. First, as Brian Trammell pointed = out in his e-mail on this thread: > The efforts of IPPM to date have been focused on simply describable, = easily repeatable metrics... there's not a lot of specification of = arbitrarily complex or unknown preconditions (as would be = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^= ^^^^^^^^^ > useful in the latency-under-load example you give), because these tend = to be counter to the goals = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > of maximizing repeatability and minimizing uncertainty. There's also a = focus on active measurement ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > for the same reason. ... these may be applicable for an extremely large number of (or, all?) = "LMAP Measurement Clients" and provide 'basic' measurement capabilities, = e.g.: UDP packet loss, UDP latency, etc. On the other end of the spectrum, I think this is where the research = community is still (?) in the midst of exploring + defining "new" = metrics that can then be used in small to medium-size experiments, = (e.g.: on a smaller base of "LMAP Measurement Clients"), whose = scientific results can ultimately inform which of these tests may get = formalized into standards by the IETF and, if appropriate, incorporated = as "new" tests or replace one or more existing tests (completely). = Obviously, how those tests ultimately get rolled out to the field = (potentially through firmware updates or only when new generations of HW = + firmware are shipped), will be determined by the extent of the change, = etc. While I have an interest in following the new(er) metrics being = explored, in the 2nd paragraph, I am more interested in and do not want = us to lose sight of those basic metrics, in the first paragraph, as I = believe a lot of valuable data can still be gleaned from them, today. -shane= From shane@castlepoint.net Mon Nov 26 21:59:32 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0862121F8468 for ; Mon, 26 Nov 2012 21:59:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.437 X-Spam-Level: X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 33UVDb3lWmNL for ; Mon, 26 Nov 2012 21:59:31 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 672D421F8465 for ; Mon, 26 Nov 2012 21:59:30 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 4321421BD for ; Tue, 27 Nov 2012 05:59:30 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 846B721B9; Mon, 26 Nov 2012 22:59:29 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6112F5E3784@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Mon, 26 Nov 2012 22:59:28 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5E3784@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Mon Nov 26 22:59:30 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b456c2199631412312585 X-DSPAM-Factors: 27, to+#+the, 0.01000, to+#+the, 0.01000, PM+#+#+#+bs7652, 0.01000, To*STARK+BARBARA, 0.01000, 2012+#+3, 0.01000, 3+20, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, then+I, 0.01000, of+#+#+#+in, 0.01000, I+#+that, 0.01000, I+#+that, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, a+single, 0.01000, a+single, 0.01000, 20+#+#+BARBARA, 0.01000, From*Amante+shane, 0.01000, PM+#+BARBARA, 0.01000, Nov+#+#+#+3, 0.01000, be+#+#+#+the, 0.01000, be+#+#+#+the, 0.01000, regulatory+use, 0.01000, as+well, 0.01000, the+#+#+the, 0.01000, the+#+#+the, 0.01000, PM+#+#+H, 0.01000, 2012+at, 0.01000 Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 05:59:32 -0000 Barbara, I'm not James :-), but let me step in anyway to ask a clarifying = question, below. On Nov 24, 2012, at 3:20 PM, "STARK, BARBARA H" wrote: > Thanks, James, for the email you sent. It really helped to improve my = understanding of what you believe to be important to a regulatory use = case, and explain why you put certain requirements into the draft. > If I read your email correctly, I believe that you support separating = discussion of recommendations for measurements and metrics, from = discussions on management and control of the elements participating in = metric data collection and dissemination. I agree with this. I believe = that the draft that you co-authored was very much focused on a = management and control architecture, so I am going to focus my comments = now purely on requirements of management and control architectures for = measurement collection architectures of the various use cases we have = been considering. >=20 > There is a key element of the draft that I disagree with, that I would = like to better understand your thoughts on. >=20 > It appears to me that the draft attempts to combine the management and = control requirements for all use cases (end user, access network = provider, application service provider, regulatory/research) into a = single management and control architecture (that therefore imposes the = requirements of the most complex use case on all use cases). This = appears to me to be a core assumption of the draft, and is my = fundamental objection to the draft. This assumption is made without any = attempt to justify it, and I saw nothing in your email that would = justify it. In my opinion, just because all of the use cases are "good" = or "valid", does not imply that combined management and control of them = is desirable or good. Most of the statements from my prior emails were = to explain why I find (mandated) combination of management and control = architectures to be undesirable. To me, the requirements of the = different use cases all seem to be very independent of each other.=20 >=20 > Why do you (or do you?) consider it important that a single management = and control architecture be defined that meets the combined needs of all = use cases? Isn't one of the fundamental benefits of standards that it allows an = _entire_ industry to benefit from the economies of scale by _allowing_ = (not, forcing) ISPs to specify to manufacturers, in RFP's, that they = build equipment (HW + SW) that follows a single specification? Thus, = isn't it better to find commonalities across several use cases and = *attempt* to engineer a solution that covers all of them in a consistent = fashion? At this stage of the LMAP work, I believe that it's a good idea to = combine the "management and control requirements" for all use cases. I = can certainly see validity in that approach, particularly relative to = services on my own network, other small- to mid-size networks and, = potentially, other use cases as well. -shane > If the draft authors would agree to consider the management and = control architecture requirements of each use case independently, I = think we might be able to progress to doing such independent requirement = assessment. But if the draft authors insist that they must be combined = into a single architecture, then I would like for some attempt to be = made to provide a convincing reason for their combination. > Barbara > _______________________________________________ > lmap mailing list > lmap@ietf.org > https://www.ietf.org/mailman/listinfo/lmap >=20 From bs7652@att.com Tue Nov 27 06:20:29 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6ADD521F850D for ; Tue, 27 Nov 2012 06:20:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.599 X-Spam-Level: X-Spam-Status: No, score=-101.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, GB_SUMOF=5, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Oh+OxsII-zKK for ; Tue, 27 Nov 2012 06:20:28 -0800 (PST) Received: from nbfkord-smmo06.seg.att.com (nbfkord-smmo06.seg.att.com [209.65.160.94]) by ietfa.amsl.com (Postfix) with ESMTP id 7B17321F84B9 for ; Tue, 27 Nov 2012 06:20:26 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo06.seg.att.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id a2cc4b05.6f8e2940.610671.00-543.1702385.nbfkord-smmo06.seg.att.com (envelope-from ); Tue, 27 Nov 2012 14:20:26 +0000 (UTC) X-MXL-Hash: 50b4cc2a53d9bc27-16fee98df8465df46bc995d00d1bd9e61a2d50e3 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo06.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 62cc4b05.0.610647.00-421.1702314.nbfkord-smmo06.seg.att.com (envelope-from ); Tue, 27 Nov 2012 14:20:23 +0000 (UTC) X-MXL-Hash: 50b4cc272b129b28-c4fced1d01cc8fd9ba99b8850e392b3cf6bcef3a Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAREKMa9009645; Tue, 27 Nov 2012 09:20:22 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qAREK7Pu009539 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 27 Nov 2012 09:20:13 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint04.pst.cso.att.com (RSA Interceptor); Tue, 27 Nov 2012 09:18:12 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Tue, 27 Nov 2012 09:17:13 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] Is there a requirement for combined management and control architecture? Thread-Index: AQHNzGRlXmv8yMV9uUajetql15fYrJf9sITw Date: Tue, 27 Nov 2012 14:17:13 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6113020E51B@GAALPA1MSGUSR9L.ITServices.sbc.com> References: <2D09D61DDFA73D4C884805CC7865E6112F5E3784@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [90.164.202.155] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=AMgvexcp c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=DNnosmBG25IA:10 a=n70DyX6EGRYA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=8KS6RNUCkZEA:10 a=_DtCM6sAD6Ksv4z8N3EA:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2012 14:20:29 -0000 > Isn't one of the fundamental benefits of standards that it allows an _ent= ire_ > industry to benefit from the economies of scale by _allowing_ (not, forci= ng) > ISPs to specify to manufacturers, in RFP's, that they build equipment (HW= + > SW) that follows a single specification? Thus, isn't it better to find > commonalities across several use cases and *attempt* to engineer a soluti= on > that covers all of them in a consistent fashion? > > At this stage of the LMAP work, I believe that it's a good idea to combin= e the > "management and control requirements" for all use cases. I can certainly= see > validity in that approach, particularly relative to services on my own ne= twork, > other small- to mid-size networks and, potentially, other use cases as we= ll. Commonalities, to me, implies the intersection of requirements of all= use cases. Not the union. To understand what the intersection is, the set = of requirements for each use case needs to be understood independently, as = its own stand-alone set of requirements. Where the requirements intersect i= s the "commonality". The current draft is attempting to define the union. I= t is imposing the sum of all requirements for all use cases on all elements= .=20 I'm absolutely in favor of seeing if there is commonality. Commonality is i= mpossible to determine unless the needs of each use case are first identifi= ed independent of the other use cases. From Ruediger.Geib@telekom.de Mon Nov 26 01:26:31 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B9F021F869D; Mon, 26 Nov 2012 01:26:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.017 X-Spam-Level: X-Spam-Status: No, score=-3.017 tagged_above=-999 required=5 tests=[AWL=-0.233, BAYES_00=-2.599, HELO_EQ_DE=0.35, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_UNSUB31=0.464] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F164jGUh+t5m; Mon, 26 Nov 2012 01:26:29 -0800 (PST) Received: from tcmail93.telekom.de (tcmail93.telekom.de [80.149.113.205]) by ietfa.amsl.com (Postfix) with ESMTP id B599321F8564; Mon, 26 Nov 2012 01:26:26 -0800 (PST) Received: from he113445.emea1.cds.t-internal.com ([10.134.93.105]) by tcmail91.telekom.de with ESMTP/TLS/AES128-SHA; 26 Nov 2012 10:26:24 +0100 Received: from HE111643.EMEA1.CDS.T-INTERNAL.COM ([10.134.93.12]) by HE113445.emea1.cds.t-internal.com ([::1]) with mapi; Mon, 26 Nov 2012 10:26:24 +0100 From: To: , Date: Mon, 26 Nov 2012 10:26:22 +0100 Thread-Topic: [ippm] [lmap] Focus on Tests, Not Architectures... Thread-Index: Ac3LUstbSzAZ8HWIRfKli94Z55UZBQAYrXgg Message-ID: References: <7.0.1.0.0.20121125083420.085ee458@att.com> In-Reply-To: Accept-Language: en-US, de-DE Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, de-DE Content-Type: multipart/alternative; boundary="_000_CA7A7C64CC4ADB458B74477EA99DF6F5A2873CCFHE111643EMEA1CD_" MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 27 Nov 2012 09:13:12 -0800 Cc: ippm@ietf.org, lmap@ietf.org Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 09:26:31 -0000 --_000_CA7A7C64CC4ADB458B74477EA99DF6F5A2873CCFHE111643EMEA1CD_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Matt, Al, one can measure accross a terminal device, home gateway, provider access sy= stem, provider policy management node, provider gateway, central lmap capture sys= tem. >From what I've seen and done, an undesired measurement result only tells me= , I have a problem. But it doesn't explain a black box, it doesn't explain, w= hether and why a reference I compare an actual result against is relevant, what ar= e the differences and so on. I doubt that all issues can be solved by a better metric alone. Regards, R=FCdiger ________________________________ From: ippm-bounces@ietf.org [mailto:ippm-bounces@ietf.org] On Behalf Of Mat= t Mathis Sent: Sunday, November 25, 2012 10:21 PM To: Al Morton Cc: lmap@ietf.org; ippm@ietf.org Subject: Re: [ippm] [lmap] Focus on Tests, Not Architectures... I would like to disagree here. Another missing requirement from 2330 is t= hat measurement vantage point should not matter. Consider the standards f= or milk: milk fat is a parameter that anybody can measure (producer, consum= er, independent lab, school classroom, etc). It can be process controlled = by the manufacturer (tweaking centrifuge settings), and is useful to the co= nsumer. As an extremely robust metric it stabilizes the industry and bring= s sanity to the marketplace. A metric that can only be measured by the producer and does not agree with = the users intuition as to the value they are getting for their money will a= lways be perceived as crooked, even if it is not. lmap is a logical step = down this path..... What we really need are better metrics. The Model Based Metrics ID is my first attempt here, although the words in = the draft fall far short of telling the right story. I have started a "Mo= del Based Metrics working doc" at https://docs.google.com/document/d/1KDJMu= Z2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit# I believe that the "Sustain= ed burst test" at the bottom is (close to) the metric that we need. It has= (is expected to have) a whole list of cool properties, including actionabl= e by the ISPs, useful to the users, RTT insensitivity, and consequently van= tage insensitivity. BTW: I enable "public comment" on the doc, so all of you can add margin not= es. I would like to request an extended slot in IPPM, to present this. It sho= uld be in a proper ID by that time. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our serv= ices to speak in defiance of unjust governments. We treat privacy and sec= urity as matters of life and death, because for some users, they are. On Sun, Nov 25, 2012 at 6:12 AM, Al Morton > wrote: LMAP and IPPM, After catching-up on many messages, I would like to add a key consideration and area for discussion and agreement for development of standardized tests: - For each test we eventually specify, it's necessary to specify the equivalent measurement points along the user's packet transfer path/architecture of each different access technology. This effort certainly needs folks with standards experience in all of the access architectures to complete the task, because working out measurement point details will be a critical aspect of conducting comparabl= e measurements. So, for anyone who read Nick's and Brian's messages below (which I mostly agree with) and was about to un-subscribe or add "lma= p" to your spam filter, don't. There's plenty to do here that's architecture-related. The various measurement points needed depend on the list of tests of course, and that list needs discussion, too. As an example needing some discussion: All packet-transfer services have a demarcation point where the customer connects their own equipment to the network. It is valuable from both the diagnostic and service verification points of view to have a measurement point very close to the demarcation point. But direct access to this point is often impossible for a measurement agent - where can we agree are the functionally equivalent alternatives in each access architecture? (don't forget wireless access) regards, Al At 11:59 AM 11/20/2012, Brian Trammell wrote: Hi, Nick, (Copying this over to IPPM as well, as the focus-on-metrics discussion will= be interesting there too). On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote: > I've said it once, I'll say it one final time: > > This focus on architecture is deeply misguided. As someone who's built o= ne such architecture (Netalyzr), working on two others (Fathom, a browser e= xtension, and techniques using GuruPlugs/Raspberry Pis), the architecture i= s the easy part. > > In all this work, having some "standard" way to communicate between my me= asurement clients and servers is useless. I just use HTTP, HTTPS, or SSH d= epending on whether quick & easy or server side or mutual authentication is= required, with whatever load-balancing is needed handled by random assignm= ent of remote test node. > > The internal data store is whatever is appropriate for the project (eg. f= or Netalyzr its python pickled objects for storage on our web server, data = in a Postgress database for analysis). > > Updates are handled appropriately (Netalyzr is updateless, its an applet = so its always up-to-date. Our android version will just use the app store = update mechanism. Fathom we use Firefox's update mechanism. The plugs are= a hack using SSH, but its a fair amount of custom goop, and they update ni= ghtly). > > And all three end up having rather different architectural requirements. Architectures, I would submit (and as I've said here before), are useful to= hang terminology off of: you need boxes and lines so everyone understands = (roughly) the same thing when you say "client" and "server" and "controller= " with reference to a set of measurements you define. But yes, specifying a way to specify a measurement is largely arbitrary. I'= d even advocate separating the representation from the transport, especiall= y as the general problem of measurement interoperability has data sources o= perating and wildly different scales, but as you say deciding this is the e= asy part. > The hard part is developing the test metrics and measurements, both in th= e abstract and within the API constraints of the target platform. E.g. why= I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore= I can just "Run Netalyzr" for all Netalyzr tests. But the same isn't the = case for the Ripe Atlas or Bismark platform. > > And Marc's list is actually just a start: you also have generic proxy pre= sence & location detection, application specific proxy detection and analys= is, packet injection detection, differential application behavior detection= , application agnostic traffic shaping detection, DNSSEC support, DNS manip= ulation detection, path MTU, IPv6 for all of the previous, and a ton of oth= ers. Netalyzr is currently composed of >100 distinct tests. And this amou= nt just continues to grow. > > > If the IETF actually wants to have an impact in this area, don't bother f= ocusing on protocols for communication, focus on tests with given attribute= s. Because people like me who actually build these things are going to ign= ore an architectural focus because that has no value. But having ACCEPTED = STANDARD tests would be very valuable. + 1e. Stated another way, the protocol gives you the grammar.= What you need to make this at all interoperable is the vocabulary. IPPM has defined a several of these at a low level, with a focus on active = measurement methods that produce comparable results. There appears to be in= terest in IPPM both in adding passive methods for the existing metrics (see= draft-trammell-ippm-hybrid-ps) as well as adding definitions of more compl= ex metrics based in part on the simple ones (see draft-mathis-ippm-model-ba= sed-metrics). > E.g. a good, standard latency-under-load test that can be written in Flas= h (so TCP/UDP connections only to cooperating servers) would be great. Our= current one in Netalyzr is only OK: we believe that one could do a lot bet= ter. > > If you could then figure out WHERE the bottleneck is (using a server that= can also do simultaneous traceroutes), this would be even more valuable. (This seems to make the assumption that traceroute-measurable latency is st= rongly correlated with application latency at a given hop, but I get your p= oint...) The efforts of IPPM to date have been focused on simply describable, easily= repeatable metrics... there's not a lot of specification of arbitrarily co= mplex or unknown preconditions (as would be useful in the latency-under-loa= d example you give), because these tend to be counter to the goals of maxim= izing repeatability and minimizing uncertainty. There's also a focus on act= ive measurement for the same reason. On the other hand, a lot of work in di= stributed measurement systems is focused on measuring the network as it is,= with implicit or explicit passive components (again, returning to "latency= under load", the load traffic will be made up of generated as well as exis= ting background load). So, what might be useful would be to open another line of work within IPPM = on interoperable descriptions of some of these existing tests as empiricall= y specified metrics, perhaps with relaxed compliance with the IPPM framewor= k as appropriate. This could be used to provide a more complex vocabulary t= o LMAP, but could also be useful independently -- if LMAP is eventually def= ined to apply only to a restricted set of use cases (i.e. specifically mand= ated tests for regulatory compliance, but not research or troubleshooting),= comparable metrics with incompatible architecture and protocols could stil= l lead to wider interoperability in the measurement space, the need for pro= tocol shims (which are after all easy to write if the number inside has the= same meaning) aside. Best regards, Brian _______________________________________________ ippm mailing list ippm@ietf.org https://www.ietf.org/mailman/listinfo/ippm _______________________________________________ ippm mailing list ippm@ietf.org https://www.ietf.org/mailman/listinfo/ippm --_000_CA7A7C64CC4ADB458B74477EA99DF6F5A2873CCFHE111643EMEA1CD_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Matt, Al,
 
one can measure accross a terminal device, home gatew= ay,=20 provider access system,
provider policy management node, provider gateway, ce= ntral=20 lmap capture system.
 
From what I've seen and done, an undesired measuremen= t result=20 only tells me,
I have a problem. But it doesn't explain a black box,= it=20 doesn't explain, whether
and why a reference I compare an actual result a= gainst=20 is relevant, what are
the differences and so=20 on.
 
I doubt that all issues can be solved by a better met= ric=20 alone.
 
Regards,
 
R=FCdiger
 
 
From:=20 ippm-bounces@ietf.org [mailto:ippm-bounces@ietf.org] On Behalf Of Ma= tt=20 Mathis
Sent: Sunday, November 25, 2012 10:21 PM
To: Al= =20 Morton
Cc: lmap@ietf.org; ippm@ietf.org
Subject: Re: [i= ppm]=20 [lmap] Focus on Tests, Not Architectures...

I= would=20 like to disagree here.   Another missing requirement from 2330 is that= =20 measurement vantage point should not matter.   Consider the= =20 standards for milk: milk fat is a parameter that anybody can measure (produ= cer,=20 consumer, independent lab, school classroom, etc).  It can be process= =20 controlled by the manufacturer (tweaking centrifuge settings), and is= =20 useful to the consumer.  As an extremely robust metric=20 it stabilizes the industry and brings sanity to the marketplace.

A metric that can only be measured by the producer and does not agree = with=20 the users intuition as to the value they are getting for their money w= ill=20 always be perceived as crooked, even if it is not.   lmap is= a=20 logical step down this path.....

What we really need are better metrics.

The Model Based Metrics ID is my first attempt here, although the word= s in=20 the draft fall far short of telling the right story.   I have started = a=20 "Model Based Metrics working doc" at https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w= 5smvZLfCL-gTk6Xgx1mI/edit# =20   I believe that the "Sustained burst test" at the bottom is (close to= ) the=20 metric that we need.  It has (is expected to have) a whole list of coo= l=20 properties, including actionable by the ISPs, useful to the users,=20 RTT insensitivity, and consequently vantage insensitivity.

BTW: I enable "public comment" on the doc, so all of you can add margi= n=20 notes.

I would like to request  an extended slot in IPPM, to present thi= s.=20  It should be in a proper ID by that time.

Thanks,
--MM--
The best way to predict the future is= to=20 create it.  - Alan Kay

Privacy matters!  We know from rece= nt=20 events that people are using our services to speak in defiance of unjust=20 governments.   We treat privacy and security as matters of life and de= ath,=20 because for some users, they are.


On Sun, Nov 25, 2012 at 6:12 AM, Al Morton <acmorton@att.com> wrote:
LMAP and IPPM,

After catching-up on many messages, I would li= ke to=20 add a key
consideration and area for discussion and agreement=20 for
development of standardized tests:

- For each test we event= ually=20 specify, it's necessary to specify
  the equivalent measurement p= oints=20 along the user's packet transfer
  path/architecture of each diff= erent=20 access technology.

This effort certainly needs folks with standard= s=20 experience in all of the
access architectures to complete the task, be= cause=20 working out
measurement point details will be a critical aspect of=20 conducting comparable
measurements. So, for anyone who read Nick's and= =20 Brian's messages
below (which I mostly agree with) and was about to=20 un-subscribe or add "lmap"
to your spam filter, don't.  There's p= lenty=20 to do here that's
architecture-related. The various measurement point= s=20 needed depend
on the list of tests of course, and that list needs=20 discussion, too.

As an example needing some discussion:
All=20 packet-transfer services have a demarcation point where the=20 customer
connects their own equipment to the network. It is valuable f= rom=20 both the
diagnostic and service verification points of view to have a= =20 measurement
point very close to the demarcation point. But direct acce= ss to=20 this
point is often impossible for a measurement agent - where can we = agree=20
are the functionally equivalent alternatives in each access=20 architecture?
(don't forget wireless access)

regards,
Al


At 11:59 AM 11/20/2012, Brian Trammell wrote:
Hi, Nick,

(Copying this over to IPPM as well, as the=20 focus-on-metrics discussion will be interesting there too).

On 2= 0 Nov=20 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:
>
&g= t;=20 This focus on architecture is deeply misguided.  As someone who's = built=20 one such architecture (Netalyzr), working on two others (Fathom, a brow= ser=20 extension, and techniques using GuruPlugs/Raspberry Pis), the architect= ure=20 is the easy part.
>
> In all this work, having some "stand= ard"=20 way to communicate between my measurement clients and servers is=20 useless.  I just use HTTP, HTTPS, or SSH depending on whether quic= k=20 & easy or server side or mutual authentication is required, with=20 whatever load-balancing is needed handled by random assignment of remot= e=20 test node.  
>
> The internal data store is wha= tever=20 is appropriate for the project (eg. for Netalyzr its python pickled obj= ects=20 for storage on our web server, data in a Postgress database for=20 analysis). 
>
> Updates are handled appropriately=20 (Netalyzr is updateless, its an applet so its always up-to-date.  = Our=20 android version will just use the app store update mechanism.  Fat= hom=20 we use Firefox's update mechanism.  The plugs are a hack using SSH= , but=20 its a fair amount of custom goop, and they update nightly). 
&= gt;=20
> And all three end up having rather different architectural=20 requirements.

Architectures, I would submit (and as I've said here before), are= =20 useful to hang terminology off of: you need boxes and lines so everyone= =20 understands (roughly) the same thing when you say "client" and "server"= and=20 "controller" with reference to a set of measurements you=20 define.

But yes, specifying a way to specify a measurement is= =20 largely arbitrary. I'd even advocate separating the representation from= the=20 transport, especially as the general problem of measurement interoperab= ility=20 has data sources operating and wildly different scales, but as you say= =20 deciding this is the easy part.

> The hard part is developing the test metrics and measurements= ,=20 both in the abstract and within the API constraints of the target=20 platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that = I can=20 run Java, so therefore I can just "Run Netalyzr" for all Netalyzr=20 tests.  But the same isn't the case for the Ripe Atlas or Bismark= =20 platform.
>
> And Marc's list is actually just a start: yo= u=20 also have generic proxy presence & location detection, application= =20 specific proxy detection and analysis, packet injection detection,=20 differential application behavior detection, application agnostic traff= ic=20 shaping detection, DNSSEC support, DNS manipulation detection, path MTU= ,=20 IPv6 for all of the previous, and a ton of others.  Netalyzr is=20 currently composed of >100 distinct tests.  And this amount jus= t=20 continues to grow.
>
>
> If the IETF actually wants= to=20 have an impact in this area, don't bother focusing on protocols for=20 communication, focus on tests with given attributes.  Because peop= le=20 like me who actually build these things are going to ignore an architec= tural=20 focus because that has no value.  But having ACCEPTED STANDARD tes= ts=20 would be very valuable. 

+ 1e<large number>. Stated another way, the protocol gives y= ou=20 the grammar. What you need to make this at all interoperable is the=20 vocabulary.

IPPM has defined a several of these at a low level,= with=20 a focus on active measurement methods that produce comparable results. = There=20 appears to be interest in IPPM both in adding passive methods for the=20 existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding= =20 definitions of more complex metrics based in part on the simple ones (s= ee=20 draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be wri= tten=20 in Flash (so TCP/UDP connections only to cooperating servers) would be= =20 great.  Our current one in Netalyzr is only OK: we believe that on= e=20 could do a lot better.
>
> If you could then figure out WH= ERE=20 the bottleneck is (using a server that can also do simultaneous=20 traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable late= ncy=20 is strongly correlated with application latency at a given hop, but I g= et=20 your point...)

The efforts of IPPM to date have been focused on= =20 simply describable, easily repeatable metrics... there's not a lot of=20 specification of arbitrarily complex or unknown preconditions (as would= be=20 useful in the latency-under-load example you give), because these tend = to be=20 counter to the goals of maximizing repeatability and minimizing uncerta= inty.=20 There's also a focus on active measurement for the same reason. On t= he=20 other hand, a lot of work in distributed measurement systems is focused= on=20 measuring the network as it is, with implicit or explicit passive compo= nents=20 (again, returning to "latency under load", the load traffic will be mad= e up=20 of generated as well as existing background load).

So, w= hat=20 might be useful would be to open another line of work within IPPM on=20 interoperable descriptions of some of these existing tests as empirical= ly=20 specified metrics, perhaps with relaxed compliance with the IPPM framew= ork=20 as appropriate. This could be used to provide a more complex vocabu= lary=20 to LMAP, but could also be useful independently -- if LMAP is eventu= ally=20 defined to apply only to a restricted set of use cases (i.e. specifical= ly=20 mandated tests for regulatory compliance, but not research or=20 troubleshooting), comparable metrics with incompatible architecture and= =20 protocols could still lead to wider interoperability in the measurement= =20 space, the need for protocol shims (which are after all easy to wri= te if=20 the number inside has the same meaning) aside.

Best=20 regards,

Brian
______________________________________________= _
ippm=20 mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm=20

____________________________________________= ___
ippm=20 mailing list
ippm@ietf.org
https://www.ietf.org/mailman/listinfo/ippm


--_000_CA7A7C64CC4ADB458B74477EA99DF6F5A2873CCFHE111643EMEA1CD_-- From shane@castlepoint.net Tue Nov 27 17:59:14 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0DDB621F8766 for ; Tue, 27 Nov 2012 17:59:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 2.063 X-Spam-Level: ** X-Spam-Status: No, score=2.063 tagged_above=-999 required=5 tests=[AWL=-2.500, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, GB_SUMOF=5, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zJrybVWt1H9R for ; Tue, 27 Nov 2012 17:59:13 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 83F0421F86E7 for ; Tue, 27 Nov 2012 17:59:13 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id F36612244 for ; Wed, 28 Nov 2012 01:59:12 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 50E3A2217; Tue, 27 Nov 2012 18:59:12 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6113020E51B@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Tue, 27 Nov 2012 18:59:11 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6112F5E3784@GAALPA1MSGUSR9L.ITServices.sbc.com> <2D09D61DDFA73D4C884805CC7865E6113020E51B@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Tue Nov 27 18:59:12 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b56ff0199631315414768 X-DSPAM-Factors: 27, to+#+#+#+to, 0.01000, what+the, 0.01000, to+#+the, 0.01000, to+#+the, 0.01000, To*STARK+BARBARA, 0.01000, to+#+to, 0.01000, to+#+to, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, I+#+that, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, I+#+#+see, 0.01000, a+single, 0.01000, the+#+#+#+to, 0.01000, and+#+#+#+to, 0.01000, From*Amante+shane, 0.01000, as+well, 0.01000, the+#+#+the, 0.01000, 2012+at, 0.01000, to+#+in, 0.01000, Mime-Version*Mail+#+1499, 0.01000, the+#+#+#+of, 0.01000, To*bs7652+att.com, 0.01000, the+#+to, 0.01000, can+#+#+#+in, 0.01000, com+wrote, 0.01000, the+#+use, 0.01000 Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 01:59:14 -0000 On Nov 27, 2012, at 7:17 AM, "STARK, BARBARA H" wrote: >> Isn't one of the fundamental benefits of standards that it allows an = _entire_ >> industry to benefit from the economies of scale by _allowing_ (not, = forcing) >> ISPs to specify to manufacturers, in RFP's, that they build equipment = (HW + >> SW) that follows a single specification? Thus, isn't it better to = find >> commonalities across several use cases and *attempt* to engineer a = solution >> that covers all of them in a consistent fashion? >>=20 >> At this stage of the LMAP work, I believe that it's a good idea to = combine the >> "management and control requirements" for all use cases. I can = certainly see >> validity in that approach, particularly relative to services on my = own network, >> other small- to mid-size networks and, potentially, other use cases = as well. >=20 > Commonalities, to me, implies the intersection of requirements = of all use cases. Not the union. To understand what the intersection is, = the set of requirements for each use case needs to be understood = independently, as its own stand-alone set of requirements. Where the = requirements intersect is the "commonality". The current draft is = attempting to define the union. It is imposing the sum of all = requirements for all use cases on all elements.=20 >=20 > I'm absolutely in favor of seeing if there is commonality. Commonality = is impossible to determine unless the needs of each use case are first = identified independent of the other use cases. Sometimes it's more efficient not to reinvent the wheel = over-and-over-and-over ... :-) Hence, I can understand why the draft = authors have taken the approach of identifying requirements that _are_ = common across all Use Cases defined in the draft. This suggests, to me = at least, that those requirements already in the draft that are = incompatible with one, or more, Use Cases should on a case-by-case = basis: a) have technical evidence presented as to _why_ they are incompatible = with a Use Case; and, b) suggestions as to how to change the text to make it compatible with = the envisioned Use Case(s). Is this not how the editing process of a IETF draft works? -shane= From bs7652@att.com Wed Nov 28 12:48:32 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B951421F84D2 for ; Wed, 28 Nov 2012 12:48:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FMB-6nQ5g2gR for ; Wed, 28 Nov 2012 12:48:32 -0800 (PST) Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id DD7EF21F84D0 for ; Wed, 28 Nov 2012 12:48:31 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo08.seg.att.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id f9876b05.62596940.116962.00-598.330930.nbfkord-smmo08.seg.att.com (envelope-from ); Wed, 28 Nov 2012 20:48:31 +0000 (UTC) X-MXL-Hash: 50b6789f2ee026ae-46c9730ee3e22693bf93c36986558878793fd306 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id d9876b05.0.116940.00-474.330853.nbfkord-smmo08.seg.att.com (envelope-from ); Wed, 28 Nov 2012 20:48:29 +0000 (UTC) X-MXL-Hash: 50b6789d7caef4b1-ed8985b143587a8efe59dbf4fab87e3f79cf655b Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qASKmRIf030470; Wed, 28 Nov 2012 15:48:29 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qASKmK0P030328 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 28 Nov 2012 15:48:23 -0500 Received: from GAALPA1MSGHUB9E.ITServices.sbc.com (gaalpa1msghub9e.itservices.sbc.com [130.8.36.91]) by sflint04.pst.cso.att.com (RSA Interceptor); Wed, 28 Nov 2012 15:47:30 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9E.ITServices.sbc.com ([130.8.36.91]) with mapi id 14.02.0318.001; Wed, 28 Nov 2012 15:47:30 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] Is there a requirement for combined management and control architecture? Thread-Index: Ac3NqZD+1DikzBKXS+yubA2FFmeeFA== Date: Wed, 28 Nov 2012 20:47:29 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6113020ED51@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.157.219] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=XZQ+OvF5 c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=yBf4MsZ4bRUA:10 a=6nTaFMBW0SAA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=8KS6RNUCkZEA:10 a=EULW10tbEzvsH_YV4_4A:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 20:48:32 -0000 > Sometimes it's more efficient not to reinvent the wheel over-and-over-and= - > over ... :-) Hence, I can understand why the draft authors have taken t= he > approach of identifying requirements that _are_ common across all Use > Cases defined in the draft. This suggests, to me at least, that those > requirements already in the draft that are incompatible with one, or more= , > Use Cases should on a case-by-case basis: > a) have technical evidence presented as to _why_ they are incompatible > with a Use Case; and, > b) suggestions as to how to change the text to make it compatible with th= e > envisioned Use Case(s). >=20 > Is this not how the editing process of a IETF draft works? >=20 > -shane If this draft were a working group draft under a chartered working group, I= might agree with what you suggest. But it isn't. There is no working group that has this effort as part of its charter. This is a personal draft. There is no agreement to use this personal draft as a baseline for anything= .=20 What I thought we were asked to consider was: Is there a need for something new to be done by IETF? To figure this out, I thought we were supposed to explore what needs might = exist, that were not being handled elsewhere (in other SDOs or other IETF W= Gs), and whether these needs were sufficiently compelling to charter an exi= sting or new WG to deal with them. We have identified the need of defining metrics, measurements, and points o= f measurement for all or some of the proposed use cases, and I think I have= seen that there is a belief that this can be covered by the existing ippm = wg, perhaps under its existing charter. I'm happy to leave that discussion = to ippm, and don't think that this lmap list needs to try to hash out these= things that ippm can deal with.=20 There appears to be this other (IMO, not well defined) need around managing= and controlling the taking of measurements, along with data collection, st= orage, and presentation, and such.=20 I am trying to suggest a path that will allow for this "need" to be defined= in such a way that it will (a) satisfy the expressers of a use cases's "ne= ed", (b) satisfy those who have experience in this area of device managemen= t, control and operational support systems, and operations architectures (l= ike me, though I'm sure there are others) and might be willing to work on i= t if they better understood it, and (c) allow for sufficient assessment to = see if there is consensus on needs that would be part of a new or modified = WG charter that would pass muster. I have seen 3 approaches proposed to proceed from where we are: (1) use the current "combined use case" draft as a starting point and a= ttempt to edit it to the point that it provides sufficient assessment that = would allow charter-able work efforts to be identified, (2) assess the needs of each use case independently (and identify commo= nality), to see if charter-able work can be identified (the draft can be co= nsidered input to this work, especially for the regulatory use case), or (3) do nothing in the area of management and control of performance mea= surements.=20 I recommend (2). I have seen no justification presented for why approach (1= ) (combine the needs of all use cases; do not identify which requirement ap= plies to which use case; do not consider the individual needs of individual= use cases) is preferred by some. I've seen nothing to indicate that anyone= else has an interest in (2), so I'm glad to do no work on progressing that= approach. I've seen a few emails that perhaps suggest a preference for app= roach (1). In the absence of consensus to proceed with (1) or (2), or sugge= st another approach, I suspect that (3) will win. If there is a desire to p= roceed with approach (1), I can only say "Have at it". I want no part in it= , and I'll be happy to step back and stay out of the way. Barbara From Michael.K.Bugenhagen@centurylink.com Wed Nov 28 14:13:44 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A1A7121F890B for ; Wed, 28 Nov 2012 14:13:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d3Gyzg2PEV7p for ; Wed, 28 Nov 2012 14:13:39 -0800 (PST) Received: from sudnp799.qwest.com (sudnp799.qwest.com [155.70.32.99]) by ietfa.amsl.com (Postfix) with ESMTP id C31FE21F8910 for ; Wed, 28 Nov 2012 14:13:38 -0800 (PST) Received: from lxomavmpc030.qintra.com (lxomavmpc030.qintra.com [151.117.207.30]) by sudnp799.qwest.com (8.14.4/8.14.4) with ESMTP id qASMDXZj003139 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 28 Nov 2012 15:13:33 -0700 (MST) Received: from lxomavmpc030.qintra.com (unknown [127.0.0.1]) by IMSA (Postfix) with ESMTP id 4C7EE1E0032; Wed, 28 Nov 2012 16:13:28 -0600 (CST) Received: from suomp60i.qintra.com (unknown [10.6.10.61]) by lxomavmpc030.qintra.com (Postfix) with ESMTP id 3091E1E005D; Wed, 28 Nov 2012 16:13:28 -0600 (CST) Received: from suomp60i.qintra.com (localhost [127.0.0.1]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id qASMDRRd019906; Wed, 28 Nov 2012 16:13:27 -0600 (CST) Received: from vodcwhubex502.ctl.intranet (vodcwhubex502.qintra.com [151.117.206.28]) by suomp60i.qintra.com (8.14.4/8.14.4) with ESMTP id qASMDRwq019899 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 28 Nov 2012 16:13:27 -0600 (CST) Received: from PODCWMBXEX505.ctl.intranet ([fe80::f87e:fe44:ad72:b610]) by vodcwhubex502.ctl.intranet ([2002:9775:ce1c::9775:ce1c]) with mapi id 14.02.0318.001; Wed, 28 Nov 2012 16:13:27 -0600 From: "Bugenhagen, Michael K" To: "'STARK, BARBARA H'" , Shane Amante Thread-Topic: [lmap] Is there a requirement for combined management and control architecture? Thread-Index: Ac3NqZD+1DikzBKXS+yubA2FFmeeFAAC8h6Q Date: Wed, 28 Nov 2012 22:13:25 +0000 Message-ID: References: <2D09D61DDFA73D4C884805CC7865E6113020ED51@GAALPA1MSGUSR9L.ITServices.sbc.com> In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6113020ED51@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [151.117.206.7] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 22:13:44 -0000 Personally I think working on the Charter to either modify the scope of IPP= M to tackle the work or make this it's own project is very important becaus= e the statements in that charter will fix all the questions being asked her= e. I'm hoping Henning and James are working on one - I was / am holding off fo= r that so far. Regards - Mike -----Original Message----- From: lmap-bounces@ietf.org [mailto:lmap-bounces@ietf.org] On Behalf Of STA= RK, BARBARA H Sent: Wednesday, November 28, 2012 2:47 PM To: Shane Amante Cc: lmap@ietf.org Subject: Re: [lmap] Is there a requirement for combined management and cont= rol architecture? > Sometimes it's more efficient not to reinvent the wheel=20 > over-and-over-and- over ... :-) Hence, I can understand why the=20 > draft authors have taken the approach of identifying requirements that=20 > _are_ common across all Use Cases defined in the draft. This=20 > suggests, to me at least, that those requirements already in the draft=20 > that are incompatible with one, or more, Use Cases should on a case-by-ca= se basis: > a) have technical evidence presented as to _why_ they are incompatible=20 > with a Use Case; and, > b) suggestions as to how to change the text to make it compatible with=20 > the envisioned Use Case(s). >=20 > Is this not how the editing process of a IETF draft works? >=20 > -shane If this draft were a working group draft under a chartered working group, I= might agree with what you suggest. But it isn't. There is no working group that has this effort as part of its charter. This is a personal draft. There is no agreement to use this personal draft as a baseline for anything= .=20 What I thought we were asked to consider was: Is there a need for something new to be done by IETF? To figure this out, I thought we were supposed to explore what needs might = exist, that were not being handled elsewhere (in other SDOs or other IETF W= Gs), and whether these needs were sufficiently compelling to charter an exi= sting or new WG to deal with them. We have identified the need of defining metrics, measurements, and points o= f measurement for all or some of the proposed use cases, and I think I have= seen that there is a belief that this can be covered by the existing ippm = wg, perhaps under its existing charter. I'm happy to leave that discussion = to ippm, and don't think that this lmap list needs to try to hash out these= things that ippm can deal with.=20 There appears to be this other (IMO, not well defined) need around managing= and controlling the taking of measurements, along with data collection, st= orage, and presentation, and such.=20 I am trying to suggest a path that will allow for this "need" to be defined= in such a way that it will (a) satisfy the expressers of a use cases's "ne= ed", (b) satisfy those who have experience in this area of device managemen= t, control and operational support systems, and operations architectures (l= ike me, though I'm sure there are others) and might be willing to work on i= t if they better understood it, and (c) allow for sufficient assessment to = see if there is consensus on needs that would be part of a new or modified = WG charter that would pass muster. I have seen 3 approaches proposed to proceed from where we are: (1) use the current "combined use case" draft as a starting point and a= ttempt to edit it to the point that it provides sufficient assessment that = would allow charter-able work efforts to be identified, (2) assess the needs of each use case independently (and identify commo= nality), to see if charter-able work can be identified (the draft can be co= nsidered input to this work, especially for the regulatory use case), or (3) do nothing in the area of management and control of performance mea= surements.=20 I recommend (2). I have seen no justification presented for why approach (1= ) (combine the needs of all use cases; do not identify which requirement ap= plies to which use case; do not consider the individual needs of individual= use cases) is preferred by some. I've seen nothing to indicate that anyone= else has an interest in (2), so I'm glad to do no work on progressing that= approach. I've seen a few emails that perhaps suggest a preference for app= roach (1). In the absence of consensus to proceed with (1) or (2), or sugge= st another approach, I suspect that (3) will win. If there is a desire to p= roceed with approach (1), I can only say "Have at it". I want no part in it= , and I'll be happy to step back and stay out of the way. Barbara _______________________________________________ lmap mailing list lmap@ietf.org https://www.ietf.org/mailman/listinfo/lmap From shane@castlepoint.net Wed Nov 28 17:40:09 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 031251F0C5C for ; Wed, 28 Nov 2012 17:40:09 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.188 X-Spam-Level: X-Spam-Status: No, score=0.188 tagged_above=-999 required=5 tests=[AWL=0.625, BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qp4H1YnNXeVH for ; Wed, 28 Nov 2012 17:40:08 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 504AB1F0C4A for ; Wed, 28 Nov 2012 17:40:08 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id A1D98220E for ; Thu, 29 Nov 2012 01:40:07 +0000 (UTC) Received: from mbp.castlepoint.net (174-29-211-99.hlrn.qwest.net [174.29.211.99]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id CF7B62204; Wed, 28 Nov 2012 18:40:06 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6113020ED51@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Wed, 28 Nov 2012 18:40:05 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <68361C3A-8E66-4B9B-A071-CE155CB2629B@castlepoint.net> References: <2D09D61DDFA73D4C884805CC7865E6113020ED51@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Nov 28 18:40:07 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b6bcf7199631468127746 X-DSPAM-Factors: 27, to+#+#+#+to, 0.01000, to+#+#+#+to, 0.01000, and+#+#+#+the, 0.01000, that+are, 0.01000, that+are, 0.01000, the+#+I, 0.01000, to+#+the, 0.01000, to+#+the, 0.01000, how+to, 0.01000, PM+#+#+#+bs7652, 0.01000, To*STARK+BARBARA, 0.01000, to+#+to, 0.01000, to+#+to, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, to+#+this, 0.01000, to+#+this, 0.01000, of+#+requirements, 0.01000, might+#+#+to, 0.01000, not+#+#+the, 0.01000, I+#+that, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, and+#+of, 0.01000, and+#+of, 0.01000, the+#+#+#+to, 0.01000, the+#+#+#+to, 0.01000, and+#+#+#+to, 0.01000 Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 01:40:09 -0000 Barbara, On Nov 28, 2012, at 1:47 PM, "STARK, BARBARA H" wrote: >> Sometimes it's more efficient not to reinvent the wheel = over-and-over-and- >> over ... :-) Hence, I can understand why the draft authors have = taken the >> approach of identifying requirements that _are_ common across all Use >> Cases defined in the draft. This suggests, to me at least, that = those >> requirements already in the draft that are incompatible with one, or = more, >> Use Cases should on a case-by-case basis: >> a) have technical evidence presented as to _why_ they are = incompatible >> with a Use Case; and, >> b) suggestions as to how to change the text to make it compatible = with the >> envisioned Use Case(s). >>=20 >> Is this not how the editing process of a IETF draft works? >>=20 >> -shane >=20 > If this draft were a working group draft under a chartered working = group, I might agree with what you suggest. > But it isn't. > There is no working group that has this effort as part of its charter. > This is a personal draft. Fair enough. > There is no agreement to use this personal draft as a baseline for = anything.=20 FWIW, others have commented, on this list, about this draft and the = impression I had was that most, but not all, thought it was a relatively = decent starting place for an architecture draft as to the coordinating = the functions performed by a measurement controller, server, network = parameter server, etc. I'd say that's potentially rough consensus, but = certainly not unanimity. Regardless, as you say, there is no WG, yet, = so no one could "officially" declare agreement, consensus, etc. > What I thought we were asked to consider was: > Is there a need for something new to be done by IETF? > To figure this out, I thought we were supposed to explore what needs = might exist, that were not being handled elsewhere (in other SDOs or = other IETF WGs), and whether these needs were sufficiently compelling to = charter an existing or new WG to deal with them. > We have identified the need of defining metrics, measurements, and = points of measurement for all or some of the proposed use cases, and I = think I have seen that there is a belief that this can be covered by the = existing ippm wg, perhaps under its existing charter. I'm happy to leave = that discussion to ippm, and don't think that this lmap list needs to = try to hash out these things that ippm can deal with.=20 > There appears to be this other (IMO, not well defined) need around = managing and controlling the taking of measurements, along with data = collection, storage, and presentation, and such.=20 > I am trying to suggest a path that will allow for this "need" to be = defined in such a way that it will (a) satisfy the expressers of a use = cases's "need", (b) satisfy those who have experience in this area of = device management, control and operational support systems, and = operations architectures (like me, though I'm sure there are others) and = might be willing to work on it if they better understood it, and (c) = allow for sufficient assessment to see if there is consensus on needs = that would be part of a new or modified WG charter that would pass = muster. >=20 > I have seen 3 approaches proposed to proceed from where we are: > (1) use the current "combined use case" draft as a starting point = and attempt to edit it to the point that it provides sufficient = assessment that would allow charter-able work efforts to be identified, > (2) assess the needs of each use case independently (and identify = commonality), to see if charter-able work can be identified (the draft = can be considered input to this work, especially for the regulatory use = case), or > (3) do nothing in the area of management and control of performance = measurements.=20 >=20 > I recommend (2). I have seen no justification presented for why = approach (1) (combine the needs of all use cases; do not identify which = requirement applies to which use case; do not consider the individual = needs of individual use cases) is preferred by some. I've seen nothing = to indicate that anyone else has an interest in (2), so I'm glad to do = no work on progressing that approach. Just to clear. In your last sentence above, you said: "so I'm glad to = do no work on progressing that approach", even though you're saying "I = recommend (2)" just before that. Did you really mean to say: "so I'm = glad to do work on progressing that approach", (notice the missing = "no")? And, if so, are you offering to write a (rough) draft to share = your thoughts on approach #2? Or, am I misunderstanding your comment = entirely? :-( > I've seen a few emails that perhaps suggest a preference for approach = (1). In the absence of consensus to proceed with (1) or (2), or suggest = another approach, I suspect that (3) will win. If there is a desire to = proceed with approach (1), I can only say "Have at it". I want no part = in it, and I'll be happy to step back and stay out of the way. Thanks for this note. If I understand you correctly and you are = offering to take a lead on approach #2 that may provide more clarity = what are the perceived or actual deficiencies in the present draft, = which will only help clarify things for everyone. -shane= From bs7652@att.com Thu Nov 29 08:17:04 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C1A9621F84C1 for ; Thu, 29 Nov 2012 08:17:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -106.599 X-Spam-Level: X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BbTo3brm59Ur for ; Thu, 29 Nov 2012 08:17:04 -0800 (PST) Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id 0632421F84BB for ; Thu, 29 Nov 2012 08:17:03 -0800 (PST) Received: from unknown [144.160.20.146] (EHLO nbfkord-smmo05.seg.att.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) with ESMTP id 08a87b05.53ca9940.1459645.00-526.4081080.nbfkord-smmo05.seg.att.com (envelope-from ); Thu, 29 Nov 2012 16:17:04 +0000 (UTC) X-MXL-Hash: 50b78a807a01f6d6-f26619f9511b98c0d8e4bed91a1a8677f1ae8649 Received: from unknown [144.160.20.146] (EHLO mlpd194.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id c7a87b05.0.1459628.00-428.4081035.nbfkord-smmo05.seg.att.com (envelope-from ); Thu, 29 Nov 2012 16:17:00 +0000 (UTC) X-MXL-Hash: 50b78a7c1c122160-f6de4dfcbb2862ba60944d214c2c3f20f55c0092 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qATGGxXc019182; Thu, 29 Nov 2012 11:16:59 -0500 Received: from sflint04.pst.cso.att.com (sflint04.pst.cso.att.com [144.154.234.231]) by mlpd194.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qATGGqPq019157 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 29 Nov 2012 11:16:56 -0500 Received: from GAALPA1MSGHUB9A.ITServices.sbc.com (gaalpa1msghub9a.itservices.sbc.com [130.8.36.87]) by sflint04.pst.cso.att.com (RSA Interceptor); Thu, 29 Nov 2012 11:16:20 -0500 Received: from GAALPA1MSGUSR9L.ITServices.sbc.com ([130.8.36.69]) by GAALPA1MSGHUB9A.ITServices.sbc.com ([130.8.36.87]) with mapi id 14.02.0318.001; Thu, 29 Nov 2012 11:16:19 -0500 From: "STARK, BARBARA H" To: Shane Amante Thread-Topic: [lmap] Is there a requirement for combined management and control architecture? Thread-Index: Ac3OTNyNsAJV2moqTiiNH/l49S/8oA== Date: Thu, 29 Nov 2012 16:16:18 +0000 Message-ID: <2D09D61DDFA73D4C884805CC7865E6113020F217@GAALPA1MSGUSR9L.ITServices.sbc.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.70.126.211] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-RSA-Inspected: yes X-RSA-Classifications: public X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)] X-MAIL-FROM: X-SOURCE-IP: [144.160.20.146] X-AnalysisOut: [v=2.0 cv=fuJrkiEf c=1 sm=0 a=Qs8R1XBwmid1qBFB/a8mmA==:17 a] X-AnalysisOut: [=3lUGnpqktvgA:10 a=6nTaFMBW0SAA:10 a=ofMgfj31e3cA:10 a=BLc] X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=XIqpo32R] X-AnalysisOut: [AAAA:8 a=8KS6RNUCkZEA:10 a=jONZRDf631Vytx6fNt0A:9 a=CjuIK1] X-AnalysisOut: [q_8ugA:10] Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 16:17:04 -0000 > > I have seen 3 approaches proposed to proceed from where we are: > > (1) use the current "combined use case" draft as a starting point an= d > attempt to edit it to the point that it provides sufficient assessment th= at > would allow charter-able work efforts to be identified, > > (2) assess the needs of each use case independently (and identify > commonality), to see if charter-able work can be identified (the draft ca= n be > considered input to this work, especially for the regulatory use case), o= r > > (3) do nothing in the area of management and control of performance > measurements. > > > > I recommend (2). I have seen no justification presented for why approac= h > (1) (combine the needs of all use cases; do not identify which requiremen= t > applies to which use case; do not consider the individual needs of indivi= dual > use cases) is preferred by some. I've seen nothing to indicate that anyon= e > else has an interest in (2), so I'm glad to do no work on progressing tha= t > approach. >=20 > Just to clear. In your last sentence above, you said: "so I'm glad to do= no > work on progressing that approach", even though you're saying "I > recommend (2)" just before that. Did you really mean to say: "so I'm gla= d to > do work on progressing that approach", (notice the missing "no")? And, i= f so, > are you offering to write a (rough) draft to share your thoughts on appro= ach > #2? Or, am I misunderstanding your comment entirely? :-( No, I'm not willing to write a draft. I meant what I said. A recommendation= is just a recommendation. Those to whom I make the recommendation are welc= ome to act upon my recommendation, express an interest in better understand= ing my recommendation, or not. So far, I've seen "not". While I'm *not* willing to write a draft, I am willing to participate in di= scussions on this list. I'm also willing to further clarify (by email) how = I would recommend constructing independent use case assessments that might = subsequently allow for gap-in-existing-standards analysis and commonality a= nalysis. But only if there is interest in exploring such an approach.=20 I would expect those who see real benefit in making something happen to be = the ones to write a draft or propose a charter. That is, either those who p= erceive themselves to be customers of the work, or who otherwise believe th= at they have something to gain by seeing the work done. I'm just here as a = subject matter expert, with extensive experience in consumer device require= ments, consumer device management and control, home networking, network and= service operations architectures, and network element operations requireme= nts. My experience in the area of measurements and metrics is very rudiment= ary, and mostly geared towards pragmatic troubleshooting of problems experi= enced by individual consumers with home networks and Internet access; so I'= m not going to pretend to know a lot about measurements and metrics. Barbara From shane@castlepoint.net Thu Nov 29 11:10:58 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8961421F8B90 for ; Thu, 29 Nov 2012 11:10:58 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.437 X-Spam-Level: X-Spam-Status: No, score=-0.437 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_ORG=0.611, RDNS_NONE=0.1] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WN0PaZBQnCUm for ; Thu, 29 Nov 2012 11:10:56 -0800 (PST) Received: from mail.friendswithtools.org (unknown [64.78.239.70]) by ietfa.amsl.com (Postfix) with ESMTP id 875FB21F8B86 for ; Thu, 29 Nov 2012 11:10:56 -0800 (PST) Received: from dspam (unknown [127.0.0.1]) by mail.friendswithtools.org (Postfix) with SMTP id 1EBAB2338 for ; Thu, 29 Nov 2012 19:10:56 +0000 (UTC) Received: from [10.9.0.10] (web.hollyman.com [64.78.239.73]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.friendswithtools.org (Postfix) with ESMTPSA id 847C422CC; Thu, 29 Nov 2012 12:10:55 -0700 (MST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) From: Shane Amante In-Reply-To: <2D09D61DDFA73D4C884805CC7865E6113020F217@GAALPA1MSGUSR9L.ITServices.sbc.com> Date: Thu, 29 Nov 2012 12:10:53 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D09D61DDFA73D4C884805CC7865E6113020F217@GAALPA1MSGUSR9L.ITServices.sbc.com> To: "STARK, BARBARA H" X-Mailer: Apple Mail (2.1499) X-DSPAM-Result: Innocent X-DSPAM-Processed: Thu Nov 29 12:10:56 2012 X-DSPAM-Confidence: 0.9899 X-DSPAM-Improbability: 1 in 9809 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 50b7b340199631732682418 X-DSPAM-Factors: 27, each+#+case, 0.01000, to+#+#+#+to, 0.01000, to+#+the, 0.01000, to+#+#+in, 0.01000, To*STARK+BARBARA, 0.01000, requirements+for, 0.01000, to+#+to, 0.01000, to+#+to, 0.01000, Subject*lmap+#+#+a, 0.01000, Mime-Version*OS+X, 0.01000, bs7652+#+#+wrote, 0.01000, to+#+this, 0.01000, of+#+requirements, 0.01000, that+approach, 0.01000, that+approach, 0.01000, of+each, 0.01000, Subject*management+and, 0.01000, Cc*ietf.org+#+ietf.org, 0.01000, and+#+of, 0.01000, that+#+#+has, 0.01000, the+#+#+#+to, 0.01000, to+#+#+#+that, 0.01000, of+#+use, 0.01000, of+#+use, 0.01000, of+#+#+control, 0.01000, From*Amante+shane, 0.01000, 2012+#+#+#+AM, 0.01000 Cc: "lmap@ietf.org" Subject: Re: [lmap] Is there a requirement for combined management and control architecture? X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Nov 2012 19:10:58 -0000 Barbara, On Nov 29, 2012, at 9:16 AM, "STARK, BARBARA H" wrote: >>> I have seen 3 approaches proposed to proceed from where we are: >>> (1) use the current "combined use case" draft as a starting point = and >> attempt to edit it to the point that it provides sufficient = assessment that >> would allow charter-able work efforts to be identified, >>> (2) assess the needs of each use case independently (and identify >> commonality), to see if charter-able work can be identified (the = draft can be >> considered input to this work, especially for the regulatory use = case), or >>> (3) do nothing in the area of management and control of = performance >> measurements. >>>=20 >>> I recommend (2). I have seen no justification presented for why = approach >> (1) (combine the needs of all use cases; do not identify which = requirement >> applies to which use case; do not consider the individual needs of = individual >> use cases) is preferred by some. I've seen nothing to indicate that = anyone >> else has an interest in (2), so I'm glad to do no work on progressing = that >> approach. >>=20 >> Just to clear. In your last sentence above, you said: "so I'm glad = to do no >> work on progressing that approach", even though you're saying "I >> recommend (2)" just before that. Did you really mean to say: "so I'm = glad to >> do work on progressing that approach", (notice the missing "no")? = And, if so, >> are you offering to write a (rough) draft to share your thoughts on = approach >> #2? Or, am I misunderstanding your comment entirely? :-( >=20 > No, I'm not willing to write a draft. I meant what I said. A = recommendation is just a recommendation. Those to whom I make the = recommendation are welcome to act upon my recommendation, express an = interest in better understanding my recommendation, or not. So far, I've = seen "not". >=20 > While I'm *not* willing to write a draft, I am willing to participate = in discussions on this list. I'm also willing to further clarify (by = email) how I would recommend constructing independent use case = assessments that might subsequently allow for gap-in-existing-standards = analysis and commonality analysis. But only if there is interest in = exploring such an approach.=20 OK, in that case, I'll bite. Can you lay out a rough & high-level = sketch (ideally, bullets in outline form?) of what you would consider a = better framework upon which to construct "independent use case = assessments", because otherwise I think it will be difficult to = impossible to have more people agree whether approach #2 is a reasonable = approach or too much work, etc. > I would expect those who see real benefit in making something happen = to be the ones to write a draft or propose a charter. That is, either = those who perceive themselves to be customers of the work, or who = otherwise believe that they have something to gain by seeing the work = done. FWIW, I perceive myself (my network) to actually be beneficiaries of = this work, hence why I'm interested in trying to move this work forward = and be inclusive of requirements beyond my own network. Hence, my = request for your help in laying out what you believe to be a better = framework upon which to form a valid base of overall requirements for = this work. > I'm just here as a subject matter expert, with extensive experience in = consumer device requirements, consumer device management and control, = home networking, network and service operations architectures, and = network element operations requirements. My experience in the area of = measurements and metrics is very rudimentary, and mostly geared towards = pragmatic troubleshooting of problems experienced by individual = consumers with home networks and Internet access; so I'm not going to = pretend to know a lot about measurements and metrics. And, I would say that I a similar level of experience and knowledge wrt = the underlying measurements. I do not consider myself an expert, but do = want to learn more in that area as we go. Thanks, -shane= From Henning.Schulzrinne@fcc.gov Fri Nov 30 15:53:14 2012 Return-Path: X-Original-To: lmap@ietfa.amsl.com Delivered-To: lmap@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 24A0221F866B; Fri, 30 Nov 2012 15:53:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AH1kdURjJ2oU; Fri, 30 Nov 2012 15:53:13 -0800 (PST) Received: from dmz-mail1.fcc.gov (dmz-mail1.fcc.gov [192.104.54.105]) by ietfa.amsl.com (Postfix) with ESMTP id 1B61121F8AEA; Fri, 30 Nov 2012 15:53:12 -0800 (PST) Received: from P2PXMB13.fccnet.win.fcc.gov ([fe80::6593:6526:65f8:66b7]) by p2pxcas01.fccnet.win.fcc.gov ([fe80::1d27:de47:ad2e:9062%13]) with mapi id 14.01.0421.002; Fri, 30 Nov 2012 18:53:08 -0500 From: Henning Schulzrinne To: Al Morton , Matt Mathis Thread-Topic: [lmap] [ippm] Focus on Tests, Not Architectures... Thread-Index: AQHNy5XIAIpq06ASp0eBzU25n0xt15gDDoZh Date: Fri, 30 Nov 2012 23:53:08 +0000 Message-ID: References: <7.0.1.0.0.20121125083420.085ee458@att.com> , <7.0.1.0.0.20121125225022.085ee310@att.com> In-Reply-To: <7.0.1.0.0.20121125225022.085ee310@att.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.104.54.63] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "ippm@ietf.org" , Nicholas Weaver , Brian Trammell , "lmap@ietf.org" Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... X-BeenThere: lmap@ietf.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Large Scale Measurement of Access network Performance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2012 23:53:14 -0000 One thing that I have learned through the MBA experience that it is hard to= find a single metric that addresses all use cases. For example, consumers = (and national bodies charged with consumer protection) might want to know w= hether advertised rates agree with actual rates. Unless ISPs are advertisin= g the model-based metric, it may not be as useful. For other applications, = such as VoIP, latency and packet loss for specific traffic types matter. Ha= ving a metric that is RTT-independent and uses a different traffic pattern = is not helpful in that case, as much as it might be otherwise. On a more general note, from a consumer perspective, there are probably thr= ee kinds of metrics that are useful: (a) Comparison between advertised and real-life metric, to allow fair compa= rison of different providers. (b) Relative metrics: I don't much care what the number is (it could be exp= ressed in some number between 0 and 100, rather than Mb/s or ms), but it sh= ould preserve relative ordering. If provider A is significantly better acco= rding to the metric X, I should expect my application (related to that metr= ic) to perform significantly better, even if I don't get the exact same per= formance. A rough example for that is cellular throughput: It is very unlik= ely that an individual consumer will get exactly the average throughput mea= sured by whatever entity, given geographic and device variations, but they = should still be able to pick a better-performing carrier based on that metr= ic. That's similar to Al's gas mileage example, "your mileage may vary", bu= t any Prius is still likely to get better gas mileage than a Lincoln Naviga= tor. (c) Threshold metrics: The metric should tell me whether a particular servi= ce is likely to be suitable for my need. Latency is the classical example: = If my access link has a round-trip latency of 400 ms, I may die prematurely= in a first-person-shooter game. Absolute accuracy isn't that important (sa= y, 400 vs. 420 ms), but consistent definition of the path component and oth= er attributes is. Some metrics might serve all three roles, but the emphasis differs. I would be interested in comments on the metrics that the FCC Measurement B= roadband America project uses. (a) Which ones need standardization first? (Some are already based on IPPM = RFCs, but most aren't.) (b) Which ones might be less valuable and/or should be substantially modifi= ed? See http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-A= ppendix.pdf (pg. 22) for the current list. Henning ________________________________ From: lmap-bounces@ietf.org [lmap-bounces@ietf.org] on behalf of Al Morton = [acmorton@att.com] Sent: Monday, November 26, 2012 12:20 AM To: Matt Mathis Cc: lmap@ietf.org; Nicholas Weaver; ippm@ietf.org; Brian Trammell Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures... Matt, I think we agree that better metrics are needed for LMAP, and that ideally a metric can be applied at any point or pair of points. RFC 2330 and most IPPM metrics have a decidedly Source-to-Destination host construction. Frankly, we did a better job with arbitrary measurement points in Rec Y.1540. It's worth a look: http://www.itu.int/rec/T-REC-Y.1540/en There are many goals the new metrics could serve: - externally observable - measurable by users - easily related to user application performance - service verification - support consumer choice of ??? - etc. We need to agree on the set that are needed. While there may be some information in a measurement that covers a combination of networks, measuring more than one network is only useful when the combined network path delivers the target performance - the sunny day outcome. Otherwise, they are inconclusive, as you said in your memo. Some target performance levels only apply in the scope where they are offered. We don't really have the milk fat scenario here, unless the producer, the truck driver, the convenience store, and your home refrigerator can all add some unknown amount of fat to the milk. ;-) When considering this problem, I've been thinking of the fuel efficiency ratings provided for new cars in the US and Europe (at least). The specs illustrate very well that the quantity measured is variable depending on the conditions of the measurement (in the US, the miles per gallon for city and highway driving), and they imply the range of values that normal users can experience (a fairly wide range). These are some aspects that would be useful to reproduce in our metrics, I think. regards, and YMMV, Al