From nobody Mon Jun 3 04:59:45 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 5D4B8120114; Mon, 3 Jun 2019 04:59:28 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Francesca Palombini via Datatracker To: Cc: draft-ietf-netvc-testing.all@ietf.org, video-codec@ietf.org, ietf@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Francesca Palombini Message-ID: <155956316822.24369.4530339060904691343@ietfa.amsl.com> Date: Mon, 03 Jun 2019 04:59:28 -0700 Archived-At: Subject: [video-codec] Genart last call review of draft-ietf-netvc-testing-08 X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 11:59:29 -0000 Reviewer: Francesca Palombini Review result: Ready with Nits I am the assigned Gen-ART reviewer for this draft. The General Area Review Team (Gen-ART) reviews all IETF documents being processed by the IESG for the IETF Chair. Please treat these comments just like any other last call comments. For more information, please see the FAQ at . Document: draft-ietf-netvc-testing-08 Reviewer: Francesca Palombini Review Date: 2019-06-02 IETF LC End Date: 2019-06-04 IESG Telechat date: Not scheduled for a telechat Summary: This draft is basically ready for publication, but has nits that should be fixed before publication. Major issues: N/A Minor issues: N/A Nits/editorial comments: * The document is missing the IANA section and Security Considerations section. Although they do not apply (as the shepherd noted), these sections are required. (See https://tools.ietf.org/html/rfc7322#section-4.8.5 and https://tools.ietf.org/html/rfc7322#section-4.8.3) * Outdated reference: A later version (-09) exists of draft-ietf-netvc-requirements-08 All following comments are suggestions that would have helped a non-expert reader (feel free to disregard): * Section 2 - Subjective testing is mentioned in the first sentence with no introduction. Although quite clear, a short definition or a reference to literature would have been helpful. * Section 2, second to last paragraph - The text is somewhat not well formulated. "... even if the group agrees that a particular test is important... then the test should be discarded. This ensures that only important tests be done; in particular, the tests that are important to participants" I understand that the paragraph was meant to underline the necessity of selecting tests that participants can/will complete in a timely fashion. But the formulation is not great: the group agrees on a particular test is important, but if it is not completed it is not important? * There are several acronym across the document that could have been expanded on first use, or used a reference to the definition: PMF, PSNR, PSNR-HVS-M, CQP * Section 3.7 - "This metric is focused on quality degradation due compression and rescaling" missing "to" * Section 5.2 - It would have been good to explicitly write out here what the parenthesis after each test indicates (resolution, bit depth, etc) * Section 5.2.2 - "High bit depth" could you quantify more precisely what high is here? * Section 5.2.4 and 5.2.5 - Does "old version" mean this should be seen as deprecated? (Please keep my address in the To: field if you want to make sure I see any response to this thread) Thanks, Francesca From nobody Wed Jun 5 12:06:30 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id CC1361201BC; Wed, 5 Jun 2019 12:06:25 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: =?utf-8?q?Mirja_K=C3=BChlewind_via_Datatracker?= To: "The IESG" Cc: draft-ietf-netvc-testing@ietf.org, Matthew Miller , netvc-chairs@ietf.org, linuxwolf+ietf@outer-planes.net, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: =?utf-8?q?Mirja_K=C3=BChlewind?= Message-ID: <155976158573.22377.17833340509275946542.idtracker@ietfa.amsl.com> Date: Wed, 05 Jun 2019 12:06:25 -0700 Archived-At: Subject: [video-codec] =?utf-8?q?Mirja_K=C3=BChlewind=27s_Abstain_on_draf?= =?utf-8?q?t-ietf-netvc-testing-08=3A_=28with_COMMENT=29?= X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2019 19:06:28 -0000 Mirja Kühlewind has entered the following ballot position for draft-ietf-netvc-testing-08: Abstain When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/ ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- This document reads more like a user manual of the Daala tools repository (together with the test sequences). I wonder why this is not simply achieved within the repo? What’s the benefit of having this in an RFC? Especially I’m worried that this document is basically useless in case the repo and test sequences disappear, and are therefore not available anymore in future, or change significantly. I understand that this is referenced by OAM and therefore publication is desired, however, I don't think that makes my concern about the standalone usefulness of this document invalid. If you really want to publish in the RFC series, I would recommend to reduce the dependencies to these repos and try to make this document more useful as a standalone test description (which would probably mean removing most of section 4 and adding some additional information to other parts). Also, the shepherd write-up seems to indicate that this document has an IPR disclosure that was filed after WG last call. Is the wg aware of this? Has this been discussed in the wg? Other more concrete comments: 1) Quick question on 2.1: Is the tester supposed to view one image after the other or both at the same time? And if one ofter the other, could the order impact the results (and should maybe be randomly chosen therfore)? 2) Sec 2.3: Would it make sense to provide a (normative) reference to MOS? Or is that supposed to be so well know that that is not even necessary? 3) Sec 3.1: maybe spell out PSNR on first occurrence. And would it make sense to provide a reference for PSNR? 4) Sec 3.2: “ The weights used by the dump_pnsrhvs.c tool in the Daala repository have been found to be the best match to real MOS scores.” Maybe document these weights in this document as well…? 5) Sec 5.3: Maybe spell out CQP at first occurrence From nobody Wed Jun 5 12:31:54 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 2461412022B; Wed, 5 Jun 2019 12:31:49 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: =?utf-8?q?Mirja_K=C3=BChlewind_via_Datatracker?= To: "The IESG" Cc: draft-ietf-netvc-requirements@ietf.org, Mo Zanaty , netvc-chairs@ietf.org, mzanaty@cisco.com, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: =?utf-8?q?Mirja_K=C3=BChlewind?= Message-ID: <155976310906.22306.2176020957912099391.idtracker@ietfa.amsl.com> Date: Wed, 05 Jun 2019 12:31:49 -0700 Archived-At: Subject: [video-codec] =?utf-8?q?Mirja_K=C3=BChlewind=27s_Abstain_on_draf?= =?utf-8?q?t-ietf-netvc-requirements-09=3A_=28with_COMMENT=29?= X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2019 19:31:51 -0000 Mirja Kühlewind has entered the following ballot position for draft-ietf-netvc-requirements-09: Abstain When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- I would expect that this document also mentions the interaction with congestion control, e.g. see RMCAT. All I can find is this, which seems very high level: "3.1.7. Specifications providing integration with system and delivery layers should be developed." Further the security consideration section is rather brief while it actually does mention security requirements. In a requirement document I would have expected that these requirements are clearly spelled out and explained, as well as risks/attacks are discussed. And why is section 4 in this document and not in draft-ietf-netvc-testing? Another minor comment: Sec 2: "for instance, wired channels are considerably more error- free than wireless channels and therefore require different QoS approaches." I don't think this is generally true as most mobile networks repair losses on the lower layers. From nobody Wed Jun 5 12:33:30 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 9816712021B; Wed, 5 Jun 2019 12:33:28 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: =?utf-8?q?Mirja_K=C3=BChlewind_via_Datatracker?= To: "The IESG" Cc: draft-ietf-netvc-testing@ietf.org, Matthew Miller , netvc-chairs@ietf.org, linuxwolf+ietf@outer-planes.net, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: =?utf-8?q?Mirja_K=C3=BChlewind?= Message-ID: <155976320861.22415.998674089792385186.idtracker@ietfa.amsl.com> Date: Wed, 05 Jun 2019 12:33:28 -0700 Archived-At: Subject: [video-codec] =?utf-8?q?Mirja_K=C3=BChlewind=27s_Abstain_on_draf?= =?utf-8?q?t-ietf-netvc-testing-08=3A_=28with_COMMENT=29?= X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2019 19:33:28 -0000 Mirja Kühlewind has entered the following ballot position for draft-ietf-netvc-testing-08: Abstain When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/ ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Update: This document has no security considerations section, while having this section is required. This document reads more like a user manual of the Daala tools repository (together with the test sequences). I wonder why this is not simply achieved within the repo? What’s the benefit of having this in an RFC? Especially I’m worried that this document is basically useless in case the repo and test sequences disappear, and are therefore not available anymore in future, or change significantly. I understand that this is referenced by OAM and therefore publication is desired, however, I don't think that makes my concern about the standalone usefulness of this document invalid. If you really want to publish in the RFC series, I would recommend to reduce the dependencies to these repos and try to make this document more useful as a standalone test description (which would probably mean removing most of section 4 and adding some additional information to other parts). Also, the shepherd write-up seems to indicate that this document has an IPR disclosure that was filed after WG last call. Is the wg aware of this? Has this been discussed in the wg? Other more concrete comments: 1) Quick question on 2.1: Is the tester supposed to view one image after the other or both at the same time? And if one ofter the other, could the order impact the results (and should maybe be randomly chosen therfore)? 2) Sec 2.3: Would it make sense to provide a (normative) reference to MOS? Or is that supposed to be so well know that that is not even necessary? 3) Sec 3.1: maybe spell out PSNR on first occurrence. And would it make sense to provide a reference for PSNR? 4) Sec 3.2: “ The weights used by the dump_pnsrhvs.c tool in the Daala repository have been found to be the best match to real MOS scores.” Maybe document these weights in this document as well…? 5) Sec 5.3: Maybe spell out CQP at first occurrence From nobody Fri Jun 7 13:42:24 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 5EA2E12012B; Fri, 7 Jun 2019 13:42:21 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: IETF Meeting Session Request Tool To: Cc: netvc-chairs@ietf.org, adam@nostrum.com, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <155994014130.27432.10172323973305168138.idtracker@ietfa.amsl.com> Date: Fri, 07 Jun 2019 13:42:21 -0700 Archived-At: Subject: [video-codec] netvc - Not having a session at IETF 105 X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2019 20:42:22 -0000 Adam Roach, a chair of the netvc working group, indicated that the netvc working group does not plan to hold a session at IETF 105. This message was generated and sent by the IETF Meeting Session Request Tool. From nobody Tue Jun 11 18:24:54 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 3FBD01200B4; Tue, 11 Jun 2019 18:24:45 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: Roman Danyliw via Datatracker To: "The IESG" Cc: draft-ietf-netvc-requirements@ietf.org, Mo Zanaty , netvc-chairs@ietf.org, mzanaty@cisco.com, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Roman Danyliw Message-ID: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> Date: Tue, 11 Jun 2019 18:24:45 -0700 Archived-At: Subject: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 01:24:45 -0000 Roman Danyliw has entered the following ballot position for draft-ietf-netvc-requirements-09: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ ---------------------------------------------------------------------- DISCUSS: ---------------------------------------------------------------------- (1) I worry that the level of detail in the in the Compression Performance Evaluation (Section 4.1) is insufficient for implementation. Specifically: (a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated.” -- How should a set of QPs be specified? --How should the quality values be calculated? -- What does the text “(in a separate document on Internet video codec testing)” mean? (b) Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that? Is it draft-ietf-netvc-testing? (2) Per the Security Considerations Section (Section 5) -- What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean? -- Please add additional language that codec should be written in a defensive style as they will be processing untrusted input. ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- (1) There is something odd about the document formatting – the title and the first author last name in the footer is wrapped in “< … >” (2) It would be helpful to forward reference that acronyms are explained in Appendix A. (3) This draft uses the words should and must to prescribe action. Why wasn’t RFC2119 cited to explain these words? (4) Section 2.0. A reference to explain “YCbCr 4:2:0” would be helpful (5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I initiate read that as a qualitative measure. However, the text says “up to 10x and more” so that implies some quantitative measure. What is that? (6) Section 2.1. Expand QP values on first use (7) Section 2.x. The language around content doesn’t appear to be consist. For example: -- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, and animation.” -- Section 2.2, IPTV says “television content” -- Section 2.5, Screen casting says “business presentations …, animation (cartoons), gaming content, data visualization, …, virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automation display, wireless display, display wall, digital operating room (DiOR), etc. What the difference between Section 2.1’s animation and Section 2.5’s cartoons? What’s the different between Section 2.1’s “movies, TV series …” and Section 2.2’s “television content”? (8) Section 2.5. The sentence “Currently, …” is very challenging to parse as it includes inline “i.e.,” and “etc”. (9) Section 2.5. Per “powerpoint, word documents”, these are specific Microsoft products. I recommend using more generic names. (10) Section 4. I found it confusing that an evaluation methodology was in a requirements document. I would have expected it in the draft-ietf-netvc-testing (11) Section 4.1. VP9 needs a reference. (12) Section 6. I don’t think this entire section is necessary. (13) Editorial Nits: -- Section 3. Style nit. s/chapter/section/ -- Section 4.1. Typo. s/computged/computed From nobody Wed Jun 12 08:37:38 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id DC00B120123; Wed, 12 Jun 2019 08:37:36 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: Alvaro Retana via Datatracker To: "The IESG" Cc: draft-ietf-netvc-requirements@ietf.org, Mo Zanaty , netvc-chairs@ietf.org, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Alvaro Retana Message-ID: <156035385682.2165.15790635678260097166.idtracker@ietfa.amsl.com> Date: Wed, 12 Jun 2019 08:37:36 -0700 Archived-At: Subject: [video-codec] Alvaro Retana's No Objection on draft-ietf-netvc-requirements-09: (with COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 15:37:37 -0000 Alvaro Retana has entered the following ballot position for draft-ietf-netvc-requirements-09: No Objection When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- §4.2 mentions "Reference software provided to the NETVC WG for candidate codecs", but, if I understand correctly, the WG will not evaluate the candidates. From nobody Wed Jun 12 11:07:46 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 7D3E9120025; Wed, 12 Jun 2019 11:07:37 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Alissa Cooper via Datatracker To: "The IESG" Cc: draft-ietf-netvc-testing@ietf.org, Matthew Miller , netvc-chairs@ietf.org, linuxwolf+ietf@outer-planes.net, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Alissa Cooper Message-ID: <156036285743.14175.5585160791726970945.idtracker@ietfa.amsl.com> Date: Wed, 12 Jun 2019 11:07:37 -0700 Archived-At: Subject: [video-codec] Alissa Cooper's Discuss on draft-ietf-netvc-testing-08: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 18:07:38 -0000 Alissa Cooper has entered the following ballot position for draft-ietf-netvc-testing-08: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/ ---------------------------------------------------------------------- DISCUSS: ---------------------------------------------------------------------- RFC 7322 requires all Internet-drafts to have IANA considerations and security considerations sections. Please add them. ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Please respond to the Gen-ART review. From nobody Wed Jun 12 11:08:55 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E1208120158; Wed, 12 Jun 2019 11:08:37 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.701 X-Spam-Level: X-Spam-Status: No, score=-2.701 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=cooperw.in header.b=uKaN0JQh; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=elHX1WIV Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zyrt4DlZUipk; Wed, 12 Jun 2019 11:08:34 -0700 (PDT) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 513E6120075; Wed, 12 Jun 2019 11:08:31 -0700 (PDT) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 472B721EA0; Wed, 12 Jun 2019 14:08:30 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Wed, 12 Jun 2019 14:08:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cooperw.in; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm3; bh=5 GDXPLUEuO81OzbWr68S2GJWhdAYBgp7i5C5FIZQPFw=; b=uKaN0JQhLgnb05buT aCAAc1VvgAxAVBPxvdLd7KYHCTCCb/52dMPycQy9gygDu9ixOB67BwCWRqNCsgv7 6CH4msdl1VVzLjnLfrgAN+TZGFOQPLHDcncH9dhSAfw1wLXS38msMO2w1G5ImQQ/ xP21AcfgMXTo71zVi8xdLzYxAWojwTb2bLrS9aLWM9vLiOz71sQpWwqjsB46io+6 khcmue6EJ/mVf++kg+Miid4J+/CAp9s90nEvf1PHhvomh9J/wU9vM8OazLTq9zGL xBQSXebHyLVjxdUH19DWUmwVNUFisWiIiAcYeDinjonbB30roYTylibHw45sJkPN u+ZRA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=5GDXPLUEuO81OzbWr68S2GJWhdAYBgp7i5C5FIZQP Fw=; b=elHX1WIVHtyenTcMdp1Ft46kD3tkkLC89pA/OI5rgQwe9cQyr3NbGdhud wD/SL30Ng1HibP8JKL8XMBCIEaX4fk1cCtMVWMsdydVKMAEQOYOScOFqXlGv7Iwx iHWegOOVy8eacUL6dGv6d/h1jbddLBw9KkWSG5bmUkseZbZDxhEqvKq68OP1Bzae I86lDWxSWd7BTeMrTIGVUHLvunIRxp/dvDvUG5wmPNsQln0J4918n0WpRf/4f1wZ vISqQTcEZpTCB6nGGFobMk3dMnYiDqcPPPEtiZIhUYODzqcbt9wiLlPV7vIOmgMw WquxnLiY+UHgbGfEsyWlb721Z5oQQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudehjedguddvjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpegtggfuhfgjfffgkfhfvffosehtqhhmtdhhtddvnecuhfhrohhmpeetlhhi shhsrgcuvehoohhpvghruceorghlihhsshgrsegtohhophgvrhifrdhinheqnecuffhomh grihhnpehivghtfhdrohhrghenucfkphepudelledrudekjedrvdduledrfeelnecurfgr rhgrmhepmhgrihhlfhhrohhmpegrlhhishhsrgestghoohhpvghrfidrihhnnecuvehluh hsthgvrhfuihiivgeptd X-ME-Proxy: Received: from static-219-39.meetings.nanog.org (unknown [199.187.219.39]) by mail.messagingengine.com (Postfix) with ESMTPA id 72E5038008D; Wed, 12 Jun 2019 14:08:29 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Alissa Cooper In-Reply-To: <155956316822.24369.4530339060904691343@ietfa.amsl.com> Date: Wed, 12 Jun 2019 14:08:28 -0400 Cc: IETF Gen-ART , draft-ietf-netvc-testing.all@ietf.org, video-codec@ietf.org, ietf@ietf.org Content-Transfer-Encoding: quoted-printable Message-Id: <13A74BDC-D5CB-4516-878F-098673E15641@cooperw.in> References: <155956316822.24369.4530339060904691343@ietfa.amsl.com> To: Francesca Palombini X-Mailer: Apple Mail (2.3445.9.1) Archived-At: Subject: Re: [video-codec] [Gen-art] Genart last call review of draft-ietf-netvc-testing-08 X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 18:08:38 -0000 Francesca, thanks for your review. I entered a DISCUSS ballot to get = IANA considerations and security considerations added. Alissa > On Jun 3, 2019, at 7:59 AM, Francesca Palombini via Datatracker = wrote: >=20 > Reviewer: Francesca Palombini > Review result: Ready with Nits >=20 > I am the assigned Gen-ART reviewer for this draft. The General Area > Review Team (Gen-ART) reviews all IETF documents being processed > by the IESG for the IETF Chair. Please treat these comments just > like any other last call comments. >=20 > For more information, please see the FAQ at >=20 > . >=20 > Document: draft-ietf-netvc-testing-08 > Reviewer: Francesca Palombini > Review Date: 2019-06-02 > IETF LC End Date: 2019-06-04 > IESG Telechat date: Not scheduled for a telechat >=20 > Summary: This draft is basically ready for publication, but has nits = that > should be fixed before publication. >=20 > Major issues: N/A >=20 > Minor issues: N/A >=20 > Nits/editorial comments: >=20 > * The document is missing the IANA section and Security Considerations = section. > Although they do not apply (as the shepherd noted), these sections are > required. (See https://tools.ietf.org/html/rfc7322#section-4.8.5 and > https://tools.ietf.org/html/rfc7322#section-4.8.3) >=20 > * Outdated reference: A later version (-09) exists of > draft-ietf-netvc-requirements-08 >=20 > All following comments are suggestions that would have helped a = non-expert > reader (feel free to disregard): >=20 > * Section 2 - Subjective testing is mentioned in the first sentence = with no > introduction. Although quite clear, a short definition or a reference = to > literature would have been helpful. >=20 > * Section 2, second to last paragraph - The text is somewhat not well > formulated. "... even if the group agrees that a particular test is > important... then the test should be discarded. This ensures that only > important tests be done; in particular, the tests that are important = to > participants" I understand that the paragraph was meant to underline = the > necessity of selecting tests that participants can/will complete in a = timely > fashion. But the formulation is not great: the group agrees on a = particular > test is important, but if it is not completed it is not important? >=20 > * There are several acronym across the document that could have been = expanded > on first use, or used a reference to the definition: PMF, PSNR, = PSNR-HVS-M, CQP >=20 > * Section 3.7 - "This metric is focused on quality degradation due = compression > and rescaling" missing "to" >=20 > * Section 5.2 - It would have been good to explicitly write out here = what the > parenthesis after each test indicates (resolution, bit depth, etc) >=20 > * Section 5.2.2 - "High bit depth" could you quantify more precisely = what high > is here? >=20 > * Section 5.2.4 and 5.2.5 - Does "old version" mean this should be = seen as > deprecated? >=20 > (Please keep my address in the To: field if you want to make sure I = see any > response to this thread) >=20 > Thanks, > Francesca >=20 > _______________________________________________ > Gen-art mailing list > Gen-art@ietf.org > https://www.ietf.org/mailman/listinfo/gen-art From nobody Wed Jun 12 12:56:16 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id C36641200EC; Wed, 12 Jun 2019 12:56:14 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Alissa Cooper via Datatracker To: "The IESG" Cc: draft-ietf-netvc-requirements@ietf.org, Mo Zanaty , netvc-chairs@ietf.org, mzanaty@cisco.com, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Alissa Cooper Message-ID: <156036937472.13994.1608185001147975030.idtracker@ietfa.amsl.com> Date: Wed, 12 Jun 2019 12:56:14 -0700 Archived-At: Subject: [video-codec] Alissa Cooper's No Objection on draft-ietf-netvc-requirements-09: (with COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Jun 2019 19:56:15 -0000 Alissa Cooper has entered the following ballot position for draft-ietf-netvc-requirements-09: No Objection When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Please respond to the Gen-ART review. I think sections 4.2 and 6 should be removed. From nobody Wed Jun 12 19:56:28 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CB7F312008B; Wed, 12 Jun 2019 19:56:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.68 X-Spam-Level: X-Spam-Status: No, score=-1.68 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, T_SPF_HELO_PERMERROR=0.01, T_SPF_PERMERROR=0.01] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (1024-bit key) reason="fail (message has been altered)" header.d=nostrum.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 14nDLQdbPo1i; Wed, 12 Jun 2019 19:56:20 -0700 (PDT) Received: from nostrum.com (raven-v6.nostrum.com [IPv6:2001:470:d:1130::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F2EF3120074; Wed, 12 Jun 2019 19:56:19 -0700 (PDT) Received: from MacBook-Pro.roach.at (99-152-146-228.lightspeed.dllstx.sbcglobal.net [99.152.146.228]) (authenticated bits=0) by nostrum.com (8.15.2/8.15.2) with ESMTPSA id x5D2u2N1016300 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 12 Jun 2019 21:56:04 -0500 (CDT) (envelope-from adam@nostrum.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=nostrum.com; s=default; t=1560394567; bh=9iOzSTa22r4NyVp5LyGFY7NgL7XaoQdiWMHRSgD053M=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=GmmBuOZbPCyoH4CFl027ZJiy5qKTAqw4Hm6XHvi4equQKFnejYwZL0IBmej4jan6F eOm93a4WiFBTgazm31OP3zC9xWL/BfTf+OCfaorSmBrCqOVdRxPS/XOHo3gzaZkR7g wcVVFFFp0y8EliNjOQHcjmLHj3cQ6dLDo9hMTyoQ= X-Authentication-Warning: raven.nostrum.com: Host 99-152-146-228.lightspeed.dllstx.sbcglobal.net [99.152.146.228] claimed to be MacBook-Pro.roach.at To: Roman Danyliw , The IESG Cc: netvc-chairs@ietf.org, video-codec@ietf.org, draft-ietf-netvc-requirements@ietf.org, mzanaty@cisco.com References: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> From: Adam Roach Message-ID: <8d9e74af-cbd4-24de-b4a9-e1ee0d9eb969@nostrum.com> Date: Wed, 12 Jun 2019 21:55:57 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.7.0 MIME-Version: 1.0 In-Reply-To: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Archived-At: Subject: Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 02:56:21 -0000 On 6/11/19 8:24 PM, Roman Danyliw via Datatracker wrote: > -- What does “codec implementation (for both an encoder and a decoder) should > cover the worst case of computational complexity, memory bandwidth, and > physical memory size” mean? Would this be clearer if it were rephrased as "...should take into consideration the worst-case..." ? /a From nobody Wed Jun 12 22:47:31 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 27172120048; Wed, 12 Jun 2019 22:47:29 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Benjamin Kaduk via Datatracker To: "The IESG" Cc: draft-ietf-netvc-testing@ietf.org, Matthew Miller , netvc-chairs@ietf.org, linuxwolf+ietf@outer-planes.net, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Benjamin Kaduk Message-ID: <156040484907.27201.12939157708122861483.idtracker@ietfa.amsl.com> Date: Wed, 12 Jun 2019 22:47:29 -0700 Archived-At: Subject: [video-codec] Benjamin Kaduk's Discuss on draft-ietf-netvc-testing-08: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 05:47:29 -0000 Benjamin Kaduk has entered the following ballot position for draft-ietf-netvc-testing-08: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/ ---------------------------------------------------------------------- DISCUSS: ---------------------------------------------------------------------- I suspect I will end up balloting Abstain on this document, given how far it is from something I could support publishing (e.g., a freestanding clear description of test procedures), but I do think there are some key issues that need to be resolved before publication. Perhaps some of them stem from a misunderstanding of the intended goal of the document -- I am reading this document as attempting to lay out procedures that are of general utility in evaluating a codec or codecs, but it is possible that (e.g.) it is intended as an informal summary of some choices made in a specific operating environment to make a specific decision. Additional text to set the scope of the discussion could go a long way. Section 2 There's a lot of assertions here without any supporting evidence or reasoning. Why is subjective better than objective? What if objective gets a lot better in the future? What if a test should be important but the interested people don't have the qualifications and the qualified people are too busy doing other things? Section 2.1 Why is p<0.5 an appropriate criterion? Even where p-values are still used in the scientific literature (which is decreasing in popularity), the threshold is more often 0.05, or even 0.00001 (e.g., for high-energy physics). Section 3 Normative C code contained outside of the RFC being published is hardly an archival way to describe an algorithm. There isn't even a git commit hash listed to ensure that the referenced material doesn't change! Section 3.5, 3.6, 3.7 I don't see how MSSSIM, CIEDE2000, VMAF, etc. are not normative references. If you want to use the indicated metric, you have to follow the reference. Section 4.2 There is a dearth of references here. This document alone is far from sufficient to perform these calculations. Section 4.3 There is a dearth of references here as well. What are libaom and libvpx? What is the overlap "BD-Rate method" and where is it specified? Section 5.2 This mention of "[a]ll current test sets" seems to imply that this document is part of a broader set of work. The Introduction should make clear what broader context this document is to be interpreted within. (I only note this once in the Discuss portion, but noted some other examples in the Comment section.) ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Section 1 Please give the reader a background reading list to get up to speed with the general concepts, terminology, etc. (E.g., I happen to know what the "luma plane" is, but that's not the case for all consumers of the RFC series.) Section 2.1 It seems likely that we should note that the ordering of the algorithms in question should be randomized (presented as left vs. right, first vs. second, etc.) Section 2.3 A Mean Opinion Score (MOS) viewing test is the preferred method of evaluating the quality. The subjective test should be performed as either consecutively showing the video sequences on one screen or on two screens located side-by-side. The testing procedure should When would it be appropriate to perform the test differently? normally follow rules described in [BT500] and be performed with non- expert test subjects. The result of the test will be (depending on (I couldn't follow the links to [BT500] and look; is this a restricted-distribution document?) Section 3.4 A forward reference or other expansion for BD-Rate would be helpful. Section 3.7 perception of video quality [VMAF]. This metric is focused on quality degradation due compression and rescaling. VMAF estimates nit: "due to" Section 4.1 Decibel is a logarithmic scale that requires a fixed reference value in order for numerical values to be defined (i.e., to "cancel out the units" before the transcendental logarthmic function is applied). I assume this is intended to take the reference as the full-fidelity unprocessed original signal, but it may be worth making that explicit. Section 4.2 Why is it necessary to mandate trapezoidal integration for the numerical integration? There are fairly cheap numerical methods available that have superior performance and are well-known. Section 5.2.x How important is it to have what is effectively a directory listing in the final RFC? Section 5.2.2, 5.2.3 This test set requires compiling with high bit depth support. Compiling? Compiling what? Again, this needs to be set in the broader context. Section 5.3 Please expand CQP on first usage. I don't think the broader scope in which the "operating modes" are defined has been made clear. Section 5.3.4, 5.3.5 supported. One parameter is provided to adjust bitrate, but the units are arbitrary. Example configurations follow: Example configurations *of what*? Section 6.2 Normally, the encoder should always be run at the slowest, highest quality speed setting (cpu-used=0 in the case of AV1 and VP9). However, in the case of computation time, both the reference and What is "the case of computation time"? changed encoder can be built with some options disabled. For AV1, - disable-ext_partition and -disable-ext_partition_types can be passed to the configure script to substantially speed up encoding, but the usage of these options must be reported in the test results. Again, this is assuming some context of command-line tools that is not clear from the document. From nobody Thu Jun 13 05:50:42 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id CAF1612004E; Thu, 13 Jun 2019 05:50:32 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: Roman Danyliw via Datatracker To: "The IESG" Cc: draft-ietf-netvc-testing@ietf.org, Matthew Miller , netvc-chairs@ietf.org, linuxwolf+ietf@outer-planes.net, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Roman Danyliw Message-ID: <156043023275.12320.14106545312452480484.idtracker@ietfa.amsl.com> Date: Thu, 13 Jun 2019 05:50:32 -0700 Archived-At: Subject: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-testing-08: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 12:50:33 -0000 Roman Danyliw has entered the following ballot position for draft-ietf-netvc-testing-08: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-testing/ ---------------------------------------------------------------------- DISCUSS: ---------------------------------------------------------------------- (1) There appear to be deep and implicit dependencies in the document to the references [DAALA-GIT] and [TESTSEQUENCES]. I applaud the intent to provide tangible advice on testing and evaluation to the community with them. I have a few questions around their use. (1.a) Why aren’t [DAALA-GIT] and [TESTSEQUENCES] normative references as they are needed to fully understand the testing approach and provide the test data? (1.b) What should readers of the RFC do should these external references no longer be available? How is the change control of these references handled? (1.c) In the case of [DAALA-GIT] which version of the code in the repo should be used? Formally, what version of C is in that repo? (1.d) Per the observation that there are implicit assumptions made by the document about familiarity with [DAALA-GIT] and [TESTSEQUENCES], here are a few places where additional clarity is required: -- Section 4.3, Per “For individual feature changes in libaom or libvpx , the overlap BD-Rate method with quantizers 20, 32, 43, and 55 must be used”, what are libaom and libvpx and what is their role? -- Section 5.3. Multiple subsection in 5.3.* list what look like settings for tools (e.g., “av1: -codec=av1 -ivf -frame-parallel=0 …”). What exactly are those? How to read them/use them? (2) The full details of some of the testing regimes need to be more fully specified (or cited as normative): -- Section 3.1. The variable MAX is not explained in either equation. -- Section 3.1. This section doesn’t explain or provide a reference to calculate PSNR. I’m not sure how to calculate or implement it. -- Section 4.2. Reference needed for Bjontegaard rate difference to explain its computation -- The references [SSIM], [MSSIM], [CIEDE2000] and [VMAF] are needed to fully explain a given testing metric so they need to be normative (3) An IANA Considerations section isn’t present in the document. (4) A Security Considerations sections isn’t present in the document. ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- A few comments: (5) Consider qualifying the title to more accurately capture the substance of this draft “Video Codec Testing and Quality Measurement {using the Daala Tool Suite or Xiph Tools and Data}.” (6) Section 3.1 and 3.3. Cite a reference for the source code files names in question – dump_psnr.c and dump_pnsrhvs.c (which are somewhere in the [DAALA-GIT] repo?) (7) Editorial Nits -- Section 2.1. Expand PMF (Probability Mass Function) on first use. -- Section 2.1. Explain floor. -- Section 2.2. Typo. s/vidoes/videos/ -- Section 2.2. Typo. s/rewatched/re-watched/ -- Section 2.3. Typo. s/comparisions/comparisons/ -- Section 3.1. Expand PSNR (Peak signal to noise ratio) on first use. -- Section 3.1. Typo. s/drived/derived/ From nobody Thu Jun 13 06:23:28 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 27EC1120153; Thu, 13 Jun 2019 06:23:20 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Benjamin Kaduk via Datatracker To: "The IESG" Cc: draft-ietf-netvc-requirements@ietf.org, Mo Zanaty , netvc-chairs@ietf.org, mzanaty@cisco.com, video-codec@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Benjamin Kaduk Message-ID: <156043220008.12531.8325545739299673939.idtracker@ietfa.amsl.com> Date: Thu, 13 Jun 2019 06:23:20 -0700 Archived-At: Subject: [video-codec] Benjamin Kaduk's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 13:23:21 -0000 Benjamin Kaduk has entered the following ballot position for draft-ietf-netvc-requirements-09: Discuss When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html for more information about IESG DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ ---------------------------------------------------------------------- DISCUSS: ---------------------------------------------------------------------- I support Roman's Discuss. I am sympathetic to the tsv-art reviewer's concerns that this document is focused on video technology of 5 years ago and may lack relevant in the current world. I don't intend to hold a Discuss point for any specific resolution, but I do think the IESG should discuss whether this concern affects the value of publishing this document as an RFC. ---------------------------------------------------------------------- COMMENT: ---------------------------------------------------------------------- Section 2.1 What do "PAM" and "RA" mean? Moving Appendix A earlier (before Section 2) or referring to it from the Introduction would be helpful. Note that RFC style is to expand on first use... . High Dynamic Range (HDR), Wide Color Gamut (WCG), high resolution (currently, up to 4K), high frame-rate content are important use cases, the codec should be able to encode such content efficiently. nits: missing "and" in serial list, and the last comma is a comma splice. Section 2.5 [Google didn't help me find reference [9].] Section 2.6 The (long) list in Section 2.5 includes "cloud gaming"; how much overlap does that have with this service? Section 3.1, 3.2 What is the difference between "General Requirements" and "Basic Requirements"? ection 3.2.1 Is "Exemplary input source formats" supposed to just be an example, or an indication of the pinnacle of possible values? Section 4.1 Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP (quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated. [...] This seems to suggest ("Initially", "for the codec selected") that the evaulation requirements are not yet complete. Are they intended to be a single set of requirements for the codec's development, or customized to some per-application requirements? (Is there a reference needed to ongoing work to solidify these requirements?) QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) }, i in R I would suggest defining the argmin function. It's surprising to see no reference to draft-ietf-netvc-testing from this document. Section 6 I don't see much need for a "Conclusions" section of this nature, in this document. Appendix B Defining (e.g.) "high dynamic range" and "wide color gamut" with respect to "normal" or "conventional" mechanisms does not really provide a stable and archival reference for comparison. From nobody Thu Jun 13 07:37:37 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E17E12030F; Thu, 13 Jun 2019 07:37:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X2j9koZbHBD5; Thu, 13 Jun 2019 07:37:31 -0700 (PDT) Received: from huawei.com (lhrrgout.huawei.com [185.176.76.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AAA75120306; Thu, 13 Jun 2019 07:37:30 -0700 (PDT) Received: from LHREML710-CAH.china.huawei.com (unknown [172.18.7.106]) by Forcepoint Email with ESMTP id 8BD80D7715F0CC3BDDAD; Thu, 13 Jun 2019 15:37:28 +0100 (IST) Received: from fraeml704-chm.china.huawei.com (10.206.15.53) by LHREML710-CAH.china.huawei.com (10.201.108.33) with Microsoft SMTP Server (TLS) id 14.3.408.0; Thu, 13 Jun 2019 15:37:28 +0100 Received: from fraeml705-chm.china.huawei.com (10.206.15.54) by fraeml704-chm.china.huawei.com (10.206.15.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Thu, 13 Jun 2019 16:37:27 +0200 Received: from fraeml705-chm.china.huawei.com ([10.206.112.183]) by fraeml705-chm.china.huawei.com ([10.206.112.183]) with mapi id 15.01.1713.004; Thu, 13 Jun 2019 16:37:27 +0200 From: Filippov Alexey To: Benjamin Kaduk , The IESG CC: "draft-ietf-netvc-requirements@ietf.org" , Mo Zanaty , "netvc-chairs@ietf.org" , "video-codec@ietf.org" Thread-Topic: Benjamin Kaduk's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) Thread-Index: AQHVIevha/t94duTcEKqqE2CqdML4qaZlNzQ Date: Thu, 13 Jun 2019 14:37:27 +0000 Message-ID: <6546942c71054cc884b6cb671cfcd918@huawei.com> References: <156043220008.12531.8325545739299673939.idtracker@ietfa.amsl.com> In-Reply-To: <156043220008.12531.8325545739299673939.idtracker@ietfa.amsl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.198.51.251] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: Subject: Re: [video-codec] Benjamin Kaduk's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 14:37:36 -0000 RGVhciBCZW5qYW1pbiwgYWxsLA0KDQpUaGFuayB5b3UgYSBsb3QgZm9yIGNvbW1lbnRpbmcgb3Vy IGRyYWZ0Lg0KDQo+IEkgc3VwcG9ydCBSb21hbidzIERpc2N1c3MuDQpHb3QgaXQgYW5kIHRyeSB0 byBhbnN3ZXIgdGhlIHJhaXNlZCBxdWVzdGlvbiBhcyBzb29uIGFzIHBvc3NpYmxlLg0KDQo+IEkg YW0gc3ltcGF0aGV0aWMgdG8gdGhlIHRzdi1hcnQgcmV2aWV3ZXIncyBjb25jZXJucyB0aGF0IHRo aXMgZG9jdW1lbnQgaXMgZm9jdXNlZCBvbiB2aWRlbyB0ZWNobm9sb2d5IG9mIDUgeWVhcnMgYWdv IGFuZCBtYXkgbGFjayByZWxldmFudCBpbiB0aGUgY3VycmVudCB3b3JsZC4NCkknZCBsaWtlIHRv IGNvbW1lbnQgdGhlIHRzdi1hcnQgcmV2aWV3ZXIncyBjb25jZXJucy4gQWNjb3JkaW5nIHRvIG15 IHVuZGVyc3RhbmRpbmcsIHRoZSByZXZpZXcgaXMgZm9jdXNlZCBvbiB0aGUgZmFjdCB0aGF0IHN1 cHBvcnQgZm9yIHF1YWxpdHkgYW5kIHJlc29sdXRpb24gc2NhbGFiaWxpdHkgaXMgbm90IG1hbmRh dG9yeS4gVGhlc2UgdHlwZXMgb2Ygc2NhbGFiaWxpdHkgYXJlIGltcG9ydGFudCBidXQgdXN1YWxs eSBzb21lIHByaWNlIGluIHRlcm1zIG9mIGNvbXByZXNzaW9uIHBlcmZvcm1hbmNlIHNob3VsZCBi ZSBwYWlkIGZvciB0aGVpciBzdXBwb3J0LiBTbywgaWYgaXQgaXMgZWZmaWNpZW50bHkgaW1wbGVt ZW50ZWQgaW4gYSBjYW5kaWRhdGUgY29kZWMgKGkuZS4gIndpdGggY29tcHJlc3Npb24gZWZmaWNp ZW5jeSBwZW5hbHR5IHVwIHRvIDUlIG9mIEJELXJhdGUgaW5jcmVhc2UgcGVyIGxheWVyIiBhcyBw b2ludGVkIG91dCBpbiBTZWN0aW9uIDMuMy4yKSwgdGhleSB3aWxsIGJlIGluIHRoZSBtYWluIHBy b2ZpbGUuIE90aGVyd2lzZSwgIiBhIHNlcGFyYXRlIHByb2ZpbGUgaXMgbmVlZGVkIC4uLiIuIElu IG15IG9waW5pb24sIHN1cHBvcnRpbmcgZm9yIHF1YWxpdHkgYW5kIHJlc29sdXRpb24gc2NhbGFi aWxpdHkgd2l0aG91dCB0YWtpbmcgaW50byBhY2NvdW50IGNvbXByZXNzaW9uIGVmZmljaWVuY3kg cGVuYWx0eSBtYXkgcHJldmVudCBzdWNjZXNzZnVsIGFkb3B0aW9uIG9mIE5FVFZDIGNvZGVjIGJ5 IGluZHVzdHJ5Lg0KDQpBbm90aGVyIGNvbmNlcm4gcmFpc2VkIGJ5IHRoZSB0c3YtYXJ0IHJldmll d2VyIGlzIHRoYXQgc3VwcG9ydCBmb3Igc2NyZWVuIGNvbnRlbnQgY29kaW5nIHRvb2xzIGlzIG5v dCBleHBsaWNpdGx5IG1lbnRpb25lZC4NCkkgYWJzb2x1dGVseSBhZ3JlZSB0aGF0IGFic2VuY2Ug b2YgdGhlc2UgdG9vbHMgd2lsbCBoYXJtIGNvbXByZXNzaW9uIHBlcmZvcm1hbmNlIG9mIGEgY2Fu ZGlkYXRlIGNvZGVjLiBTbywgaWYgdGhlc2UgdG9vbHMgYXJlIG5vdCB1c2VkIGluIGEgY29kZWMs IGl0IGNhbid0IGJlIGNvbXBldGl0aXZlIGFzIGNvbXBhcmVkIHRvIG90aGVyIGNhbmRpZGF0ZSBj b2RlY3MuIEl0IHdpbGwgYmVjb21lIGFwcGFyZW50IHdoaWxlIHRlc3RpbmcgaXQgKGJ5IHRoZSB3 YXksIHRoZSB0ZXN0aW5nIGRyYWZ0IGNvbnRhaW5zIHNjcmVlbiBjb250ZW50IG1hdGVyaWFscyku IE9uIHRoZSBvdGhlciBoYW5kLCB3ZSBzaG91bGRuJ3QgaW5zaXN0IG9uIHN1cHBvcnRpbmcgZm9y IHNwZWNpZmljIHNjcmVlbiBjb250ZW50IHRvb2xzIHRoYXQgd291bGQgcmVzdHJpY3QgdGhlIGZy ZWVkb20gb2YgY29kZWMgZGV2ZWxvcGVycy4NCg0KUGxlYXNlIG5vdGUgdGhhdCB0aGVzZSBhbmQg b3RoZXIgY29uY2VybnMgd2VyZSBhZGRyZXNzZWQgaW4gbXkgcmVzcG9uc2U6DQpodHRwczovL21h aWxhcmNoaXZlLmlldGYub3JnL2FyY2gvbXNnL3Rzdi1hcnQvUnB5cGpxUkpEX05qZW5zLVNZd2Rw UlJRRlUwDQoNCi0tDQpCZXN0IHJlZ2FyZHMsDQpBbGV4ZXkgRmlsaXBwb3YNCg0KLS0tLS1Pcmln aW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEJlbmphbWluIEthZHVrIHZpYSBEYXRhdHJhY2tlciBb bWFpbHRvOm5vcmVwbHlAaWV0Zi5vcmddIA0KU2VudDogVGh1cnNkYXksIEp1bmUgMTMsIDIwMTkg NDoyMyBQTQ0KVG86IFRoZSBJRVNHIDxpZXNnQGlldGYub3JnPg0KQ2M6IGRyYWZ0LWlldGYtbmV0 dmMtcmVxdWlyZW1lbnRzQGlldGYub3JnOyBNbyBaYW5hdHkgPG16YW5hdHlAY2lzY28uY29tPjsg bmV0dmMtY2hhaXJzQGlldGYub3JnOyBtemFuYXR5QGNpc2NvLmNvbTsgdmlkZW8tY29kZWNAaWV0 Zi5vcmcNClN1YmplY3Q6IEJlbmphbWluIEthZHVrJ3MgRGlzY3VzcyBvbiBkcmFmdC1pZXRmLW5l dHZjLXJlcXVpcmVtZW50cy0wOTogKHdpdGggRElTQ1VTUyBhbmQgQ09NTUVOVCkNCg0KQmVuamFt aW4gS2FkdWsgaGFzIGVudGVyZWQgdGhlIGZvbGxvd2luZyBiYWxsb3QgcG9zaXRpb24gZm9yDQpk cmFmdC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy0wOTogRGlzY3Vzcw0KDQpXaGVuIHJlc3BvbmRp bmcsIHBsZWFzZSBrZWVwIHRoZSBzdWJqZWN0IGxpbmUgaW50YWN0IGFuZCByZXBseSB0byBhbGwg ZW1haWwgYWRkcmVzc2VzIGluY2x1ZGVkIGluIHRoZSBUbyBhbmQgQ0MgbGluZXMuIChGZWVsIGZy ZWUgdG8gY3V0IHRoaXMgaW50cm9kdWN0b3J5IHBhcmFncmFwaCwgaG93ZXZlci4pDQoNCg0KUGxl YXNlIHJlZmVyIHRvIGh0dHBzOi8vd3d3LmlldGYub3JnL2llc2cvc3RhdGVtZW50L2Rpc2N1c3Mt Y3JpdGVyaWEuaHRtbA0KZm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgSUVTRyBESVNDVVNTIGFu ZCBDT01NRU5UIHBvc2l0aW9ucy4NCg0KDQpUaGUgZG9jdW1lbnQsIGFsb25nIHdpdGggb3RoZXIg YmFsbG90IHBvc2l0aW9ucywgY2FuIGJlIGZvdW5kIGhlcmU6DQpodHRwczovL2RhdGF0cmFja2Vy LmlldGYub3JnL2RvYy9kcmFmdC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy8NCg0KDQoNCi0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0NCkRJU0NVU1M6DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCkkgc3VwcG9ydCBSb21hbidzIERp c2N1c3MuDQoNCkkgYW0gc3ltcGF0aGV0aWMgdG8gdGhlIHRzdi1hcnQgcmV2aWV3ZXIncyBjb25j ZXJucyB0aGF0IHRoaXMgZG9jdW1lbnQgaXMgZm9jdXNlZCBvbiB2aWRlbyB0ZWNobm9sb2d5IG9m IDUgeWVhcnMgYWdvIGFuZCBtYXkgbGFjayByZWxldmFudCBpbiB0aGUgY3VycmVudCB3b3JsZC4g IEkgZG9uJ3QgaW50ZW5kIHRvIGhvbGQgYSBEaXNjdXNzIHBvaW50IGZvciBhbnkgc3BlY2lmaWMg cmVzb2x1dGlvbiwgYnV0IEkgZG8gdGhpbmsgdGhlIElFU0cgc2hvdWxkIGRpc2N1c3Mgd2hldGhl ciB0aGlzIGNvbmNlcm4gYWZmZWN0cyB0aGUgdmFsdWUgb2YgcHVibGlzaGluZyB0aGlzIGRvY3Vt ZW50IGFzIGFuIFJGQy4NCg0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpDT01NRU5UOg0KLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LQ0KDQpTZWN0aW9uIDIuMQ0KDQpXaGF0IGRvICJQQU0iIGFuZCAiUkEiIG1lYW4/ICBNb3Zpbmcg QXBwZW5kaXggQSBlYXJsaWVyIChiZWZvcmUgU2VjdGlvbg0KMikgb3IgcmVmZXJyaW5nIHRvIGl0 IGZyb20gdGhlIEludHJvZHVjdGlvbiB3b3VsZCBiZSBoZWxwZnVsLiAgTm90ZSB0aGF0IFJGQyBz dHlsZSBpcyB0byBleHBhbmQgb24gZmlyc3QgdXNlLi4uDQoNCiAgICAgICAuIEhpZ2ggRHluYW1p YyBSYW5nZSAoSERSKSwgV2lkZSBDb2xvciBHYW11dCAoV0NHKSwgaGlnaA0KICAgICAgICAgIHJl c29sdXRpb24gKGN1cnJlbnRseSwgdXAgdG8gNEspLCBoaWdoIGZyYW1lLXJhdGUgY29udGVudCBh cmUNCiAgICAgICAgICBpbXBvcnRhbnQgdXNlIGNhc2VzLCB0aGUgY29kZWMgc2hvdWxkIGJlIGFi bGUgdG8gZW5jb2RlIHN1Y2gNCiAgICAgICAgICBjb250ZW50IGVmZmljaWVudGx5Lg0KDQpuaXRz OiBtaXNzaW5nICJhbmQiIGluIHNlcmlhbCBsaXN0LCBhbmQgdGhlIGxhc3QgY29tbWEgaXMgYSBj b21tYSBzcGxpY2UuDQoNClNlY3Rpb24gMi41DQoNCltHb29nbGUgZGlkbid0IGhlbHAgbWUgZmlu ZCByZWZlcmVuY2UgWzldLl0NCg0KU2VjdGlvbiAyLjYNCg0KVGhlIChsb25nKSBsaXN0IGluIFNl Y3Rpb24gMi41IGluY2x1ZGVzICJjbG91ZCBnYW1pbmciOyBob3cgbXVjaCBvdmVybGFwIGRvZXMg dGhhdCBoYXZlIHdpdGggdGhpcyBzZXJ2aWNlPw0KDQpTZWN0aW9uIDMuMSwgMy4yDQoNCldoYXQg aXMgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiAiR2VuZXJhbCBSZXF1aXJlbWVudHMiIGFuZCAiQmFz aWMgUmVxdWlyZW1lbnRzIj8NCg0KZWN0aW9uIDMuMi4xDQoNCklzICJFeGVtcGxhcnkgaW5wdXQg c291cmNlIGZvcm1hdHMiIHN1cHBvc2VkIHRvIGp1c3QgYmUgYW4gZXhhbXBsZSwgb3IgYW4gaW5k aWNhdGlvbiBvZiB0aGUgcGlubmFjbGUgb2YgcG9zc2libGUgdmFsdWVzPw0KDQpTZWN0aW9uIDQu MQ0KDQogICBJbml0aWFsbHksIGZvciB0aGUgY29kZWMgc2VsZWN0ZWQgYXMgYSByZWZlcmVuY2Ug b25lIChlLmcuLCBIRVZDIG9yDQogICBWUDkpLCBhIHNldCBvZiAxMCBRUCAocXVhbnRpemF0aW9u IHBhcmFtZXRlcikgdmFsdWVzIHNob3VsZCBiZQ0KICAgc3BlY2lmaWVkIChpbiBhIHNlcGFyYXRl IGRvY3VtZW50IG9uIEludGVybmV0IHZpZGVvIGNvZGVjIHRlc3RpbmcpDQogICBhbmQgY29ycmVz cG9uZGluZyBxdWFsaXR5IHZhbHVlcyBzaG91bGQgYmUgY2FsY3VsYXRlZC4gWy4uLl0NCg0KVGhp cyBzZWVtcyB0byBzdWdnZXN0ICgiSW5pdGlhbGx5IiwgImZvciB0aGUgY29kZWMgc2VsZWN0ZWQi KSB0aGF0IHRoZSBldmF1bGF0aW9uIHJlcXVpcmVtZW50cyBhcmUgbm90IHlldCBjb21wbGV0ZS4g IEFyZSB0aGV5IGludGVuZGVkIHRvIGJlIGEgc2luZ2xlIHNldCBvZiByZXF1aXJlbWVudHMgZm9y IHRoZSBjb2RlYydzIGRldmVsb3BtZW50LCBvciBjdXN0b21pemVkIHRvIHNvbWUgcGVyLWFwcGxp Y2F0aW9uIHJlcXVpcmVtZW50cz8gIChJcyB0aGVyZSBhIHJlZmVyZW5jZSBuZWVkZWQgdG8gb25n b2luZyB3b3JrIHRvIHNvbGlkaWZ5IHRoZXNlIHJlcXVpcmVtZW50cz8pDQoNCiAgIFFQJ2sgPSBh cmdtaW4geyBhYnMoUSdpKFFQJ2kpIC0gUWsoUVBrKSkgfSwNCiAgICAgICAgICBpIGluIFINCg0K SSB3b3VsZCBzdWdnZXN0IGRlZmluaW5nIHRoZSBhcmdtaW4gZnVuY3Rpb24uDQoNCkl0J3Mgc3Vy cHJpc2luZyB0byBzZWUgbm8gcmVmZXJlbmNlIHRvIGRyYWZ0LWlldGYtbmV0dmMtdGVzdGluZyBm cm9tIHRoaXMgZG9jdW1lbnQuDQoNClNlY3Rpb24gNg0KDQpJIGRvbid0IHNlZSBtdWNoIG5lZWQg Zm9yIGEgIkNvbmNsdXNpb25zIiBzZWN0aW9uIG9mIHRoaXMgbmF0dXJlLCBpbiB0aGlzIGRvY3Vt ZW50Lg0KDQpBcHBlbmRpeCBCDQoNCkRlZmluaW5nIChlLmcuKSAiaGlnaCBkeW5hbWljIHJhbmdl IiBhbmQgIndpZGUgY29sb3IgZ2FtdXQiIHdpdGggcmVzcGVjdCB0byAibm9ybWFsIiBvciAiY29u dmVudGlvbmFsIiBtZWNoYW5pc21zIGRvZXMgbm90IHJlYWxseSBwcm92aWRlIGEgc3RhYmxlIGFu ZCBhcmNoaXZhbCByZWZlcmVuY2UgZm9yIGNvbXBhcmlzb24uDQoNCg0K From nobody Thu Jun 13 10:15:10 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8CF24120120; Thu, 13 Jun 2019 10:15:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WHW58FpU-Cyx; Thu, 13 Jun 2019 10:15:00 -0700 (PDT) Received: from huawei.com (lhrrgout.huawei.com [185.176.76.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8DA8F12011C; Thu, 13 Jun 2019 10:15:00 -0700 (PDT) Received: from lhreml704-cah.china.huawei.com (unknown [172.18.7.106]) by Forcepoint Email with ESMTP id 2219132B076A8A28C085; Thu, 13 Jun 2019 18:14:58 +0100 (IST) Received: from fraeml703-chm.china.huawei.com (10.206.15.52) by lhreml704-cah.china.huawei.com (10.201.108.45) with Microsoft SMTP Server (TLS) id 14.3.408.0; Thu, 13 Jun 2019 18:14:57 +0100 Received: from fraeml705-chm.china.huawei.com (10.206.15.54) by fraeml703-chm.china.huawei.com (10.206.15.52) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Thu, 13 Jun 2019 19:14:57 +0200 Received: from fraeml705-chm.china.huawei.com ([10.206.112.183]) by fraeml705-chm.china.huawei.com ([10.206.112.183]) with mapi id 15.01.1713.004; Thu, 13 Jun 2019 19:14:57 +0200 From: Filippov Alexey To: Roman Danyliw , Adam Roach , The IESG CC: "draft-ietf-netvc-requirements@ietf.org" , Mo Zanaty , "netvc-chairs@ietf.org" , "video-codec@ietf.org" Thread-Topic: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) Thread-Index: AQHVIL2iTyqJ65ndE0u1t8hrSi8/oaaZ1RMw Date: Thu, 13 Jun 2019 17:14:56 +0000 Message-ID: <8820dcb9272f4de7b3ebb7cba68052f4@huawei.com> References: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> In-Reply-To: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.198.51.251] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: Subject: Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jun 2019 17:15:04 -0000 RGVhciBSb21hbiwNCg0KVGhhbmsgeW91IGEgbG90IGZvciB5b3VyIGNvbW1lbnRzIGFuZCByYWlz aW5nIHRoZSBpbXBvcnRhbnQgcXVlc3Rpb25zLg0KDQo+IEhvdyBzaG91bGQgYSBzZXQgb2YgUVBz IGJlIHNwZWNpZmllZD8NCkFzIHlvdSBwcm9iYWJseSBrbm93LCBhIHNldCBvZiBhdmFpbGFibGUg UVAgdmFsdWVzIGNhbiBiZSBjb2RlYyBzcGVjaWZpYy4gVXN1YWxseSwgYSBRUCBzZXQgaXMgc2Vs ZWN0ZWQgdG8gY292ZXIgbWVkaXVtIGJpdC1yYXRlIHJhbmdlIHRoYXQgaXMgY29uc2lkZXJlZCB0 byBiZSB0aGUgbW9zdCBjb21wbGV4IGZvciBjb21wcmVzc2luZy4gV2UgcHJvcG9zZWQgdG8gc2V0 IHVwIHRoZSBhc3Nlc3NtZW50IHByb2Nlc3Mgbm90IG9ubHkgZm9yIHRoaXMgcmFuZ2UgYnV0IGFs c28gZm9yIGxvdyBhbmQgaGlnaCBRUCByYW5nZXMgdG8gY292ZXIgYSB3aWRlciByYW5nZSBvZiBh cHBsaWNhdGlvbnMuIEFjY29yZGluZyB0byBteSB1bmRlcnN0YW5kaW5nLCBjb25jcmV0ZSBRUCB2 YWx1ZXMgc2hvdWxkIGJlIHNwZWNpZmllZCBmb3IgY2FuZGlkYXRlIGNvZGVjcyBpbiBkcmFmdC1p ZXRmLW5ldHZjLXRlc3RpbmcuDQoNCj4gSG93IHNob3VsZCB0aGUgcXVhbGl0eSB2YWx1ZXMgYmUg Y2FsY3VsYXRlZD8NCkl0IGlzIGV4cGxhaW5lZCBpbiBTZWN0aW9uIDQuMSwgbmFtZWx5OiDigJxU byBhc3Nlc3MgdGhlIHF1YWxpdHkgb2Ygb3V0cHV0IChkZWNvZGVkKSBzZXF1ZW5jZXMsIHR3byBp bmRleGVzLCBQU05SIFszXSBhbmQgTVMtU1NJTSBbMywxMV0gYXJlIHNlcGFyYXRlbHkgY29tcHV0 ZWQuIEluIHRoZSBjYXNlIG9mIHRoZSBZQ2JDciBjb2xvciBmb3JtYXQsIFBTTlIgc2hvdWxkIGJl IGNhbGN1bGF0ZWQgZm9yIGVhY2ggY29sb3IgcGxhbmUgd2hlcmVhcyBNUy1TU0lNIGlzIGNhbGN1 bGF0ZWQgZm9yIGx1bWEgY2hhbm5lbCBvbmx5LiBJbiB0aGUgY2FzZSBvZiB0aGUgUkdCIGNvbG9y IGZvcm1hdCwgYm90aCBtZXRyaWNzIGFyZSBjb21wdXRlZCBmb3IgUiwgRyBhbmQgQiBjaGFubmVs cy4gVGh1cywgZm9yIGVhY2ggc2VxdWVuY2UsIDMwIFJELXBvaW50cyBmb3IgUFNOUiAoaS5lLiB0 aHJlZSBSRC1jdXJ2ZXMsIG9uZSBmb3IgZWFjaCBjaGFubmVsKSBhbmQgMTAgUkQtcG9pbnRzIGZv ciBNUy1TU0lNIChpLmUuIG9uZSBSRC1jdXJ2ZSwgZm9yIGx1bWEgY2hhbm5lbCBvbmx5KSBzaG91 bGQgYmUgY2FsY3VsYXRlZCBpbiB0aGUgY2FzZSBvZiBZQ2JDci4gSWYgY29udGVudCBpcyBlbmNv ZGVkIGFzIFJHQiwgNjAgUkQtcG9pbnRzICgzMCBmb3IgUFNOUiBhbmQgMzAgZm9yIE1TLVNTSU0p IHNob3VsZCBiZSBjYWxjdWxhdGVkLCBpLmUuIHRocmVlIFJELWN1cnZlcyAob25lIGZvciBlYWNo IGNoYW5uZWwpIGFyZSBjb21wdXRlZCBmb3IgUFNOUiBhcyB3ZWxsIGFzIHRocmVlIFJELWN1cnZl cyAob25lIGZvciBlYWNoIGNoYW5uZWwpIGZvciBNUy1TU0lNLuKAnSBJbiByZWZlcmVuY2VzIFsz LCAxMV0sIHRoZXNlIDIgcXVhbGl0eSBhc3Nlc3NtZW50IG1ldHJpY3MgYW5kIHRoZSB3YXlzIG9m IGhvdyB0byBjYWxjdWxhdGUgdGhlbSBhcmUgZGVzY3JpYmVkIGluIGRldGFpbC4NCg0KPi0tIFdo YXQgZG9lcyB0aGUgdGV4dCDigJwoaW4gYSBzZXBhcmF0ZSBkb2N1bWVudCBvbiBJbnRlcm5ldCB2 aWRlbyBjb2RlYyB0ZXN0aW5nKeKAnSBtZWFuPw0KPiBQZXIg4oCcQSBsaXN0IG9mIHZpZGVvIHNl cXVlbmNlcyB0aGF0IHNob3VsZCBiZSB1c2VkIGZvciB0ZXN0aW5nIGFzIHdlbGwgYXMgdGhlIDEw IFFQIHZhbHVlcyBmb3IgdGhlIHJlZmVyZW5jZSBjb2RlYyBhcmUgZGVmaW5lZCBpbiBhIHNlcGFy YXRlIGRvY3VtZW50ICIsIHdoYXQgZG9jdW1lbnQgaXMgdGhhdD8gIElzIGl0IGRyYWZ0LWlldGYt bmV0dmMtdGVzdGluZz8NCkFjY29yZGluZyB0byBteSB1bmRlcnN0YW5kaW5nLCBpdCBpcyBkcmFm dC1pZXRmLW5ldHZjLXRlc3RpbmcuDQoNCj5XaGF0IGRvZXMg4oCcY29kZWMgaW1wbGVtZW50YXRp b24gKGZvciBib3RoIGFuIGVuY29kZXIgYW5kIGEgZGVjb2Rlcikgc2hvdWxkIGNvdmVyIHRoZSB3 b3JzdCBjYXNlIG9mIGNvbXB1dGF0aW9uYWwgY29tcGxleGl0eSwgbWVtb3J5IGJhbmR3aWR0aCwg YW5kIHBoeXNpY2FsIG1lbW9yeSBzaXpl4oCdIG1lYW4/DQpJ4oCZZCBsaWtlIHRvIHRoYW5rIEFk YW0gZm9yIGhpcyBjb21tZW50LiBJIGFsc28gdGhpbmsgdGhhdCBoaXMgZm9ybXVsYXRpb24gKCIu Li5zaG91bGQgdGFrZSBpbnRvIGNvbnNpZGVyYXRpb24gdGhlIHdvcnN0LWNhc2UuLi4iKSBpcyBj bGVhcmVyLiANCkEgbW9yZSBkZXRhaWxlZCBleHBsYW5hdGlvbiBvbiB0aGlzIFNlY3Rpb24gY2Fu IGJlIGZvdW5kIGluIG15IHJlc3BvbnNlIHRvIHRoZSBzZWNkaXIgcmV2aWV3ZXI6IGh0dHBzOi8v bWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvc2VjZGlyL1FZeC1obDA3YWVjNVhoZ3FrbGRx Z1dGRzNJcyANCg0KPlBsZWFzZSBhZGQgYWRkaXRpb25hbCBsYW5ndWFnZSB0aGF0IGNvZGVjIHNo b3VsZCBiZSB3cml0dGVuIGluIGEgZGVmZW5zaXZlIHN0eWxlIGFzIHRoZXkgd2lsbCBiZSBwcm9j ZXNzaW5nIHVudHJ1c3RlZCBpbnB1dC4NCkdvb2QgcG9pbnQuIFRoYW5rIHlvdS4gSeKAmWxsIGFk ZCBpdC4NCg0KLS0NCkJlc3QgcmVnYXJkcywNCkFsZXhleSBGaWxpcHBvdg0KDQotLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogUm9tYW4gRGFueWxpdyB2aWEgRGF0YXRyYWNrZXIgW21h aWx0bzpub3JlcGx5QGlldGYub3JnXSANClNlbnQ6IFdlZG5lc2RheSwgSnVuZSAxMiwgMjAxOSA0 OjI1IEFNDQpUbzogVGhlIElFU0cgPGllc2dAaWV0Zi5vcmc+DQpDYzogZHJhZnQtaWV0Zi1uZXR2 Yy1yZXF1aXJlbWVudHNAaWV0Zi5vcmc7IE1vIFphbmF0eSA8bXphbmF0eUBjaXNjby5jb20+OyBu ZXR2Yy1jaGFpcnNAaWV0Zi5vcmc7IG16YW5hdHlAY2lzY28uY29tOyB2aWRlby1jb2RlY0BpZXRm Lm9yZw0KU3ViamVjdDogUm9tYW4gRGFueWxpdydzIERpc2N1c3Mgb24gZHJhZnQtaWV0Zi1uZXR2 Yy1yZXF1aXJlbWVudHMtMDk6ICh3aXRoIERJU0NVU1MgYW5kIENPTU1FTlQpDQoNClJvbWFuIERh bnlsaXcgaGFzIGVudGVyZWQgdGhlIGZvbGxvd2luZyBiYWxsb3QgcG9zaXRpb24gZm9yDQpkcmFm dC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy0wOTogRGlzY3Vzcw0KDQpXaGVuIHJlc3BvbmRpbmcs IHBsZWFzZSBrZWVwIHRoZSBzdWJqZWN0IGxpbmUgaW50YWN0IGFuZCByZXBseSB0byBhbGwgZW1h aWwgYWRkcmVzc2VzIGluY2x1ZGVkIGluIHRoZSBUbyBhbmQgQ0MgbGluZXMuIChGZWVsIGZyZWUg dG8gY3V0IHRoaXMgaW50cm9kdWN0b3J5IHBhcmFncmFwaCwgaG93ZXZlci4pDQoNCg0KUGxlYXNl IHJlZmVyIHRvIGh0dHBzOi8vd3d3LmlldGYub3JnL2llc2cvc3RhdGVtZW50L2Rpc2N1c3MtY3Jp dGVyaWEuaHRtbA0KZm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgSUVTRyBESVNDVVNTIGFuZCBD T01NRU5UIHBvc2l0aW9ucy4NCg0KDQpUaGUgZG9jdW1lbnQsIGFsb25nIHdpdGggb3RoZXIgYmFs bG90IHBvc2l0aW9ucywgY2FuIGJlIGZvdW5kIGhlcmU6DQpodHRwczovL2RhdGF0cmFja2VyLmll dGYub3JnL2RvYy9kcmFmdC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy8NCg0KDQoNCi0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0NCkRJU0NVU1M6DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCigxKSBJIHdvcnJ5IHRoYXQgdGhlIGxl dmVsIG9mIGRldGFpbCBpbiB0aGUgaW4gdGhlIENvbXByZXNzaW9uIFBlcmZvcm1hbmNlIEV2YWx1 YXRpb24gKFNlY3Rpb24gNC4xKSBpcyBpbnN1ZmZpY2llbnQgZm9yIGltcGxlbWVudGF0aW9uLiAg U3BlY2lmaWNhbGx5Og0KDQooYSkgUGVyIOKAnEluaXRpYWxseSwgZm9yIHRoZSBjb2RlYyBzZWxl Y3RlZCBhcyBhIHJlZmVyZW5jZSBvbmUgKGUuZy4sIEhFVkMgb3IgVlA5KSwgYSBzZXQgb2YgMTAg UVAgcXVhbnRpemF0aW9uIHBhcmFtZXRlcikgdmFsdWVzIHNob3VsZCBiZSBzcGVjaWZpZWQgKGlu IGEgc2VwYXJhdGUgZG9jdW1lbnQgb24gSW50ZXJuZXQgdmlkZW8gY29kZWMgdGVzdGluZykgYW5k IGNvcnJlc3BvbmRpbmcgcXVhbGl0eSB2YWx1ZXMgc2hvdWxkIGJlIGNhbGN1bGF0ZWQu4oCdDQoN Ci0tIEhvdyBzaG91bGQgYSBzZXQgb2YgUVBzIGJlIHNwZWNpZmllZD8NCg0KLS1Ib3cgc2hvdWxk IHRoZSBxdWFsaXR5IHZhbHVlcyBiZSBjYWxjdWxhdGVkPw0KDQotLSBXaGF0IGRvZXMgdGhlIHRl eHQg4oCcKGluIGEgc2VwYXJhdGUgZG9jdW1lbnQgb24gSW50ZXJuZXQgdmlkZW8gY29kZWMgdGVz dGluZynigJ0gbWVhbj8NCg0KKGIpIFBlciDigJxBIGxpc3Qgb2YgdmlkZW8gc2VxdWVuY2VzIHRo YXQgc2hvdWxkIGJlIHVzZWQgZm9yIHRlc3RpbmcgYXMgd2VsbCBhcyB0aGUgMTAgUVAgdmFsdWVz IGZvciB0aGUgcmVmZXJlbmNlIGNvZGVjIGFyZSBkZWZpbmVkIGluIGEgc2VwYXJhdGUgZG9jdW1l bnQgIiwgd2hhdCBkb2N1bWVudCBpcyB0aGF0PyAgSXMgaXQgZHJhZnQtaWV0Zi1uZXR2Yy10ZXN0 aW5nPw0KDQooMikgUGVyIHRoZSBTZWN1cml0eSBDb25zaWRlcmF0aW9ucyBTZWN0aW9uIChTZWN0 aW9uIDUpDQoNCi0tIFdoYXQgZG9lcyDigJxjb2RlYyBpbXBsZW1lbnRhdGlvbiAoZm9yIGJvdGgg YW4gZW5jb2RlciBhbmQgYSBkZWNvZGVyKSBzaG91bGQgY292ZXIgdGhlIHdvcnN0IGNhc2Ugb2Yg Y29tcHV0YXRpb25hbCBjb21wbGV4aXR5LCBtZW1vcnkgYmFuZHdpZHRoLCBhbmQgcGh5c2ljYWwg bWVtb3J5IHNpemXigJ0gbWVhbj8NCg0KLS0gUGxlYXNlIGFkZCBhZGRpdGlvbmFsIGxhbmd1YWdl IHRoYXQgY29kZWMgc2hvdWxkIGJlIHdyaXR0ZW4gaW4gYSBkZWZlbnNpdmUgc3R5bGUgYXMgdGhl eSB3aWxsIGJlIHByb2Nlc3NpbmcgdW50cnVzdGVkIGlucHV0Lg0KDQoNCi0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0N CkNPTU1FTlQ6DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCigxKSBUaGVyZSBpcyBzb21ldGhpbmcgb2RkIGFi b3V0IHRoZSBkb2N1bWVudCBmb3JtYXR0aW5nIOKAkyB0aGUgdGl0bGUgYW5kIHRoZSBmaXJzdCBh dXRob3IgbGFzdCBuYW1lIGluIHRoZSBmb290ZXIgaXMgd3JhcHBlZCBpbiDigJw8IOKApiA+4oCd DQoNCigyKSBJdCB3b3VsZCBiZSBoZWxwZnVsIHRvIGZvcndhcmQgcmVmZXJlbmNlIHRoYXQgYWNy b255bXMgYXJlIGV4cGxhaW5lZCBpbiBBcHBlbmRpeCBBLg0KDQooMykgVGhpcyBkcmFmdCB1c2Vz IHRoZSB3b3JkcyBzaG91bGQgYW5kIG11c3QgdG8gcHJlc2NyaWJlIGFjdGlvbi4gIFdoeSB3YXNu 4oCZdA0KUkZDMjExOSBjaXRlZCB0byBleHBsYWluIHRoZXNlIHdvcmRzPw0KDQooNCkgU2VjdGlv biAyLjAuICBBIHJlZmVyZW5jZSB0byBleHBsYWluIOKAnFlDYkNyIDQ6Mjow4oCdIHdvdWxkIGJl IGhlbHBmdWwNCg0KKDUpIFNlY3Rpb24gMi4xLiBQZXIg4oCcaGlnaCBlbmNvZGVyIGNvbXBsZXhp dHnigJ0gYW5kIOKAnGRlY29kaW5nIGNvbXBsZXhpdHnigJ0sIEkgaW5pdGlhdGUgcmVhZCB0aGF0 IGFzIGEgcXVhbGl0YXRpdmUgbWVhc3VyZS4gIEhvd2V2ZXIsIHRoZSB0ZXh0IHNheXMg4oCcdXAg dG8gMTB4IGFuZCBtb3Jl4oCdIHNvIHRoYXQgaW1wbGllcyBzb21lIHF1YW50aXRhdGl2ZSBtZWFz dXJlLiAgV2hhdCBpcyB0aGF0Pw0KDQooNikgU2VjdGlvbiAyLjEuICBFeHBhbmQgUVAgdmFsdWVz IG9uIGZpcnN0IHVzZQ0KDQooNykgU2VjdGlvbiAyLnguICBUaGUgbGFuZ3VhZ2UgYXJvdW5kIGNv bnRlbnQgZG9lc27igJl0IGFwcGVhciB0byBiZSBjb25zaXN0LiANCkZvciBleGFtcGxlOg0KDQot LSBTZWN0aW9uIDIuMSwgSW50ZXJuZXQgVmlkZW8gU3RyZWFtaW5nIHNheXMg4oCcbW92aWVzLCBU Vi1zZXJpZXMgYW5kIHNob3dzLCBhbmQgYW5pbWF0aW9uLuKAnQ0KDQotLSBTZWN0aW9uIDIuMiwg SVBUViBzYXlzIOKAnHRlbGV2aXNpb24gY29udGVudOKAnQ0KDQotLSBTZWN0aW9uIDIuNSwgU2Ny ZWVuIGNhc3Rpbmcgc2F5cyDigJxidXNpbmVzcyBwcmVzZW50YXRpb25zIOKApiwgYW5pbWF0aW9u IChjYXJ0b29ucyksIGdhbWluZyBjb250ZW50LCBkYXRhIHZpc3VhbGl6YXRpb24sIOKApiwgdmly dHVhbCBkZXNrdG9wIGluZnJhc3RydWN0dXJlIChWREkpLCBzY3JlZW4vZGVza3RvcCBzaGFyaW5n IGFuZCBjb2xsYWJvcmF0aW9uLCBzdXBlcnZpc29yeSBjb250cm9sIGFuZCBkYXRhIGFjcXVpc2l0 aW9uIChTQ0FEQSkgZGlzcGxheSwgYXV0b21vdGl2ZS9uYXZpZ2F0aW9uIGRpc3BsYXksIGNsb3Vk IGdhbWluZywgZmFjdG9yeSBhdXRvbWF0aW9uIGRpc3BsYXksIHdpcmVsZXNzIGRpc3BsYXksIGRp c3BsYXkgd2FsbCwgZGlnaXRhbCBvcGVyYXRpbmcgcm9vbSAoRGlPUiksIGV0Yy4NCg0KV2hhdCB0 aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIFNlY3Rpb24gMi4x4oCZcyBhbmltYXRpb24gYW5kIFNlY3Rp b24gMi414oCZcyBjYXJ0b29ucz8NCg0KV2hhdOKAmXMgdGhlIGRpZmZlcmVudCBiZXR3ZWVuIFNl Y3Rpb24gMi4x4oCZcyDigJxtb3ZpZXMsIFRWIHNlcmllcyDigKbigJ0gYW5kIFNlY3Rpb24gMi4y 4oCZcyDigJx0ZWxldmlzaW9uIGNvbnRlbnTigJ0/DQoNCig4KSBTZWN0aW9uIDIuNS4gIFRoZSBz ZW50ZW5jZSDigJxDdXJyZW50bHksIOKApuKAnSBpcyB2ZXJ5IGNoYWxsZW5naW5nIHRvIHBhcnNl IGFzIGl0IGluY2x1ZGVzIGlubGluZSDigJxpLmUuLOKAnSBhbmQg4oCcZXRj4oCdLg0KDQooOSkg U2VjdGlvbiAyLjUuICBQZXIg4oCccG93ZXJwb2ludCwgd29yZCBkb2N1bWVudHPigJ0sIHRoZXNl IGFyZSBzcGVjaWZpYyBNaWNyb3NvZnQgcHJvZHVjdHMuICBJIHJlY29tbWVuZCB1c2luZyBtb3Jl IGdlbmVyaWMgbmFtZXMuDQoNCigxMCkgU2VjdGlvbiA0LiAgSSBmb3VuZCBpdCBjb25mdXNpbmcg dGhhdCBhbiBldmFsdWF0aW9uIG1ldGhvZG9sb2d5IHdhcyBpbiBhIHJlcXVpcmVtZW50cyBkb2N1 bWVudC4gIEkgd291bGQgaGF2ZSBleHBlY3RlZCBpdCBpbiB0aGUgZHJhZnQtaWV0Zi1uZXR2Yy10 ZXN0aW5nDQoNCigxMSkgU2VjdGlvbiA0LjEuICBWUDkgbmVlZHMgYSByZWZlcmVuY2UuDQoNCigx MikgU2VjdGlvbiA2LiAgSSBkb27igJl0IHRoaW5rIHRoaXMgZW50aXJlIHNlY3Rpb24gaXMgbmVj ZXNzYXJ5Lg0KDQooMTMpIEVkaXRvcmlhbCBOaXRzOg0KLS0gU2VjdGlvbiAzLiAgU3R5bGUgbml0 LiAgcy9jaGFwdGVyL3NlY3Rpb24vDQoNCi0tIFNlY3Rpb24gNC4xLiBUeXBvLiAgcy9jb21wdXRn ZWQvY29tcHV0ZWQNCg0KDQo= From nobody Fri Jun 14 14:07:23 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7D5871200E5; Fri, 14 Jun 2019 14:07:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.199 X-Spam-Level: X-Spam-Status: No, score=-4.199 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FJdu-OWuH77b; Fri, 14 Jun 2019 14:07:18 -0700 (PDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 40AB11200B1; Fri, 14 Jun 2019 14:07:17 -0700 (PDT) Received: from kduck.mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id x5EL76qK013202 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 14 Jun 2019 17:07:08 -0400 Date: Fri, 14 Jun 2019 16:07:05 -0500 From: Benjamin Kaduk To: Filippov Alexey Cc: "draft-ietf-netvc-requirements@ietf.org" , Mo Zanaty , "netvc-chairs@ietf.org" , "video-codec@ietf.org" Message-ID: <20190614210705.GU52381@kduck.mit.edu> References: <156043220008.12531.8325545739299673939.idtracker@ietfa.amsl.com> <6546942c71054cc884b6cb671cfcd918@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6546942c71054cc884b6cb671cfcd918@huawei.com> User-Agent: Mutt/1.10.1 (2018-07-13) Archived-At: Subject: Re: [video-codec] Benjamin Kaduk's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2019 21:07:21 -0000 On Thu, Jun 13, 2019 at 02:37:27PM +0000, Filippov Alexey wrote: > Dear Benjamin, all, > > Thank you a lot for commenting our draft. > > > I support Roman's Discuss. > Got it and try to answer the raised question as soon as possible. > > > I am sympathetic to the tsv-art reviewer's concerns that this document is focused on video technology of 5 years ago and may lack relevant in the current world. > I'd like to comment the tsv-art reviewer's concerns. According to my understanding, the review is focused on the fact that support for quality and resolution scalability is not mandatory. These types of scalability are important but usually some price in terms of compression performance should be paid for their support. So, if it is efficiently implemented in a candidate codec (i.e. "with compression efficiency penalty up to 5% of BD-rate increase per layer" as pointed out in Section 3.3.2), they will be in the main profile. Otherwise, " a separate profile is needed ...". In my opinion, supporting for quality and resolution scalability without taking into account compression efficiency penalty may prevent successful adoption of NETVC codec by industry. > > Another concern raised by the tsv-art reviewer is that support for screen content coding tools is not explicitly mentioned. > I absolutely agree that absence of these tools will harm compression performance of a candidate codec. So, if these tools are not used in a codec, it can't be competitive as compared to other candidate codecs. It will become apparent while testing it (by the way, the testing draft contains screen content materials). On the other hand, we shouldn't insist on supporting for specific screen content tools that would restrict the freedom of codec developers. > > Please note that these and other concerns were addressed in my response: > https://mailarchive.ietf.org/arch/msg/tsv-art/RpypjqRJD_Njens-SYwdpRRQFU0 I did not your previous response, which is much appreciated. Unfortunately, this is far from my area of expertise, so my main goal here was to raise the question to the IESG at a whole, and having done that, I have changed my ballot position to No Objection. It would be great to hear more from Bernard when he has time, and I think Adam is going to work on getting that to happen. -Ben > > -----Original Message----- > From: Benjamin Kaduk via Datatracker [mailto:noreply@ietf.org] > Sent: Thursday, June 13, 2019 4:23 PM > To: The IESG > Cc: draft-ietf-netvc-requirements@ietf.org; Mo Zanaty ; netvc-chairs@ietf.org; mzanaty@cisco.com; video-codec@ietf.org > Subject: Benjamin Kaduk's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) > > Benjamin Kaduk has entered the following ballot position for > draft-ietf-netvc-requirements-09: Discuss > > When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) > > > Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html > for more information about IESG DISCUSS and COMMENT positions. > > > The document, along with other ballot positions, can be found here: > https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/ > > > > ---------------------------------------------------------------------- > DISCUSS: > ---------------------------------------------------------------------- > > I support Roman's Discuss. > > I am sympathetic to the tsv-art reviewer's concerns that this document is focused on video technology of 5 years ago and may lack relevant in the current world. I don't intend to hold a Discuss point for any specific resolution, but I do think the IESG should discuss whether this concern affects the value of publishing this document as an RFC. > > > ---------------------------------------------------------------------- > COMMENT: > ---------------------------------------------------------------------- > > Section 2.1 > > What do "PAM" and "RA" mean? Moving Appendix A earlier (before Section > 2) or referring to it from the Introduction would be helpful. Note that RFC style is to expand on first use... > > . High Dynamic Range (HDR), Wide Color Gamut (WCG), high > resolution (currently, up to 4K), high frame-rate content are > important use cases, the codec should be able to encode such > content efficiently. > > nits: missing "and" in serial list, and the last comma is a comma splice. > > Section 2.5 > > [Google didn't help me find reference [9].] > > Section 2.6 > > The (long) list in Section 2.5 includes "cloud gaming"; how much overlap does that have with this service? > > Section 3.1, 3.2 > > What is the difference between "General Requirements" and "Basic Requirements"? > > ection 3.2.1 > > Is "Exemplary input source formats" supposed to just be an example, or an indication of the pinnacle of possible values? > > Section 4.1 > > Initially, for the codec selected as a reference one (e.g., HEVC or > VP9), a set of 10 QP (quantization parameter) values should be > specified (in a separate document on Internet video codec testing) > and corresponding quality values should be calculated. [...] > > This seems to suggest ("Initially", "for the codec selected") that the evaulation requirements are not yet complete. Are they intended to be a single set of requirements for the codec's development, or customized to some per-application requirements? (Is there a reference needed to ongoing work to solidify these requirements?) > > QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) }, > i in R > > I would suggest defining the argmin function. > > It's surprising to see no reference to draft-ietf-netvc-testing from this document. > > Section 6 > > I don't see much need for a "Conclusions" section of this nature, in this document. > > Appendix B > > Defining (e.g.) "high dynamic range" and "wide color gamut" with respect to "normal" or "conventional" mechanisms does not really provide a stable and archival reference for comparison. > > From nobody Fri Jun 14 16:07:58 2019 Return-Path: X-Original-To: video-codec@ietf.org Delivered-To: video-codec@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 19DEC1200C4; Fri, 14 Jun 2019 16:07:49 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Linda Dunbar via Datatracker To: Cc: draft-ietf-netvc-testing.all@ietf.org, video-codec@ietf.org, ietf@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 6.97.1 Auto-Submitted: auto-generated Precedence: bulk Reply-To: Linda Dunbar Message-ID: <156055366901.28347.16629268075901765352@ietfa.amsl.com> Date: Fri, 14 Jun 2019 16:07:49 -0700 Archived-At: Subject: [video-codec] Opsdir telechat review of draft-ietf-netvc-testing-08 X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2019 23:07:49 -0000 Reviewer: Linda Dunbar Review result: Has Nits Reviewer: Linda Dunbar Review result: Ready with Comments & Nits I have reviewed this document as part of the Operational directorate's ongoing effort to review all IETF documents being processed by the IESG. These comments were written with the intent of improving the operational aspects of the IETF drafts. Comments that are not addressed in last call may be included in AD reviews during the IESG review. Document editors and WG chairs should treat these comments just like any other last call comments. This document describes the guidelines and procedures for evaluating images & videos. Even though I am not a video/codec expert, the document is written clear enough for me to follow through. Only one part not clear: There are many image names being listed in the Section 5.2.2 ~5.2.5. Are those image names significant? I see some of those names are listed in the [TESTSEQUENCE] reference. But I don't understand why need to literally list those names. What the reason for listing them? those names occupied many pages, kind of diluting the other procedures being described. Best Regards, Linda Dunbar From nobody Fri Jun 21 07:09:08 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 96FFE12026E; Fri, 21 Jun 2019 07:09:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cert.org Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G0qHWfS75_dJ; Fri, 21 Jun 2019 07:08:57 -0700 (PDT) Received: from taper.sei.cmu.edu (taper.sei.cmu.edu [147.72.252.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5B468120114; Fri, 21 Jun 2019 07:08:57 -0700 (PDT) Received: from korb.sei.cmu.edu (korb.sei.cmu.edu [10.64.21.30]) by taper.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id x5LE8hXV020821; Fri, 21 Jun 2019 10:08:43 -0400 DKIM-Filter: OpenDKIM Filter v2.11.0 taper.sei.cmu.edu x5LE8hXV020821 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cert.org; s=yc2bmwvrj62m; t=1561126123; bh=qyEPNnMPGEeBuLlZ15gPNa/JCGJ6QbPqYRb6yfNY0zc=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=dvSTN7TGnXvsAT5ownYABrrMtGG7GgOJ7Lg6a1Q5HkAdyYNITkmdqEW0OqDmcIfhm v9S7Q197L5rhXgFFzmQ/54jyWQGD6y3aQxjHbn2tE+WcQqllndeWUzxi02m+0oHk3q A+y20Y6Bs6iJV4Fsk/exmaEEbyUYLx8utgckZipE= Received: from CASSINA.ad.sei.cmu.edu (cassina.ad.sei.cmu.edu [10.64.28.249]) by korb.sei.cmu.edu (8.14.7/8.14.7) with ESMTP id x5LE8dbi018908; Fri, 21 Jun 2019 10:08:39 -0400 Received: from MARATHON.ad.sei.cmu.edu ([10.64.28.250]) by CASSINA.ad.sei.cmu.edu ([10.64.28.249]) with mapi id 14.03.0439.000; Fri, 21 Jun 2019 10:08:38 -0400 From: Roman Danyliw To: Filippov Alexey , Adam Roach , The IESG CC: "netvc-chairs@ietf.org" , "video-codec@ietf.org" , "draft-ietf-netvc-requirements@ietf.org" , Mo Zanaty Thread-Topic: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) Thread-Index: AQHVIL2m9AWiZ72osE6epr04IDyWcKaaGJ4AgAwKMSA= Date: Fri, 21 Jun 2019 14:08:38 +0000 Message-ID: <359EC4B99E040048A7131E0F4E113AFC01B33A0CEA@marathon> References: <156030268519.5895.7315446863069831893.idtracker@ietfa.amsl.com> <8820dcb9272f4de7b3ebb7cba68052f4@huawei.com> In-Reply-To: <8820dcb9272f4de7b3ebb7cba68052f4@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.64.22.6] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Archived-At: Subject: Re: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT) X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Jun 2019 14:09:01 -0000 SGkhDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogaWVzZyBbbWFpbHRv Omllc2ctYm91bmNlc0BpZXRmLm9yZ10gT24gQmVoYWxmIE9mIEZpbGlwcG92IEFsZXhleQ0KPiBT ZW50OiBUaHVyc2RheSwgSnVuZSAxMywgMjAxOSAxOjE1IFBNDQo+IFRvOiBSb21hbiBEYW55bGl3 IDxyZGRAY2VydC5vcmc+OyBBZGFtIFJvYWNoIDxhZGFtQG5vc3RydW0uY29tPjsNCj4gVGhlIElF U0cgPGllc2dAaWV0Zi5vcmc+DQo+IENjOiBuZXR2Yy1jaGFpcnNAaWV0Zi5vcmc7IHZpZGVvLWNv ZGVjQGlldGYub3JnOyBkcmFmdC1pZXRmLW5ldHZjLQ0KPiByZXF1aXJlbWVudHNAaWV0Zi5vcmc7 IE1vIFphbmF0eSA8bXphbmF0eUBjaXNjby5jb20+DQo+IFN1YmplY3Q6IFJFOiBSb21hbiBEYW55 bGl3J3MgRGlzY3VzcyBvbiBkcmFmdC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy0wOToNCj4gKHdp dGggRElTQ1VTUyBhbmQgQ09NTUVOVCkNCj4gDQo+IERlYXIgUm9tYW4sDQo+IA0KPiBUaGFuayB5 b3UgYSBsb3QgZm9yIHlvdXIgY29tbWVudHMgYW5kIHJhaXNpbmcgdGhlIGltcG9ydGFudCBxdWVz dGlvbnMuDQo+IA0KPiA+IEhvdyBzaG91bGQgYSBzZXQgb2YgUVBzIGJlIHNwZWNpZmllZD8NCj4g QXMgeW91IHByb2JhYmx5IGtub3csIGEgc2V0IG9mIGF2YWlsYWJsZSBRUCB2YWx1ZXMgY2FuIGJl IGNvZGVjIHNwZWNpZmljLg0KPiBVc3VhbGx5LCBhIFFQIHNldCBpcyBzZWxlY3RlZCB0byBjb3Zl ciBtZWRpdW0gYml0LXJhdGUgcmFuZ2UgdGhhdCBpcw0KPiBjb25zaWRlcmVkIHRvIGJlIHRoZSBt b3N0IGNvbXBsZXggZm9yIGNvbXByZXNzaW5nLiBXZSBwcm9wb3NlZCB0byBzZXQgdXANCj4gdGhl IGFzc2Vzc21lbnQgcHJvY2VzcyBub3Qgb25seSBmb3IgdGhpcyByYW5nZSBidXQgYWxzbyBmb3Ig bG93IGFuZCBoaWdoIFFQDQo+IHJhbmdlcyB0byBjb3ZlciBhIHdpZGVyIHJhbmdlIG9mIGFwcGxp Y2F0aW9ucy4gQWNjb3JkaW5nIHRvIG15DQo+IHVuZGVyc3RhbmRpbmcsIGNvbmNyZXRlIFFQIHZh bHVlcyBzaG91bGQgYmUgc3BlY2lmaWVkIGZvciBjYW5kaWRhdGUgY29kZWNzDQo+IGluIGRyYWZ0 LWlldGYtbmV0dmMtdGVzdGluZw0KDQpTZWUgYmVsb3cuICBDYW4geW91IGhlbHAgd2l0aCBhIHNw ZWNpZmljIHBvaW50ZXIgaW50byBkcmFmdC1pZXRmLW5ldHZjLXRlc3Rpbmcgd2hpY2ggbGlzdHMg Y2FuZGlkYXRlIGNvZGVjcytRUHMuDQoNCj4gDQo+ID4gSG93IHNob3VsZCB0aGUgcXVhbGl0eSB2 YWx1ZXMgYmUgY2FsY3VsYXRlZD8NCj4gSXQgaXMgZXhwbGFpbmVkIGluIFNlY3Rpb24gNC4xLCBu YW1lbHk6IOKAnFRvIGFzc2VzcyB0aGUgcXVhbGl0eSBvZiBvdXRwdXQNCj4gKGRlY29kZWQpIHNl cXVlbmNlcywgdHdvIGluZGV4ZXMsIFBTTlIgWzNdIGFuZCBNUy1TU0lNIFszLDExXSBhcmUNCj4g c2VwYXJhdGVseSBjb21wdXRlZC4gSW4gdGhlIGNhc2Ugb2YgdGhlIFlDYkNyIGNvbG9yIGZvcm1h dCwgUFNOUiBzaG91bGQgYmUNCj4gY2FsY3VsYXRlZCBmb3IgZWFjaCBjb2xvciBwbGFuZSB3aGVy ZWFzIE1TLVNTSU0gaXMgY2FsY3VsYXRlZCBmb3IgbHVtYQ0KPiBjaGFubmVsIG9ubHkuIEluIHRo ZSBjYXNlIG9mIHRoZSBSR0IgY29sb3IgZm9ybWF0LCBib3RoIG1ldHJpY3MgYXJlDQo+IGNvbXB1 dGVkIGZvciBSLCBHIGFuZCBCIGNoYW5uZWxzLiBUaHVzLCBmb3IgZWFjaCBzZXF1ZW5jZSwgMzAg UkQtcG9pbnRzIGZvcg0KPiBQU05SIChpLmUuIHRocmVlIFJELWN1cnZlcywgb25lIGZvciBlYWNo IGNoYW5uZWwpIGFuZCAxMCBSRC1wb2ludHMgZm9yIE1TLQ0KPiBTU0lNIChpLmUuIG9uZSBSRC1j dXJ2ZSwgZm9yIGx1bWEgY2hhbm5lbCBvbmx5KSBzaG91bGQgYmUgY2FsY3VsYXRlZCBpbiB0aGUN Cj4gY2FzZSBvZiBZQ2JDci4gSWYgY29udGVudCBpcyBlbmNvZGVkIGFzIFJHQiwgNjAgUkQtcG9p bnRzICgzMCBmb3IgUFNOUiBhbmQNCj4gMzAgZm9yIE1TLVNTSU0pIHNob3VsZCBiZSBjYWxjdWxh dGVkLCBpLmUuIHRocmVlIFJELWN1cnZlcyAob25lIGZvciBlYWNoDQo+IGNoYW5uZWwpIGFyZSBj b21wdXRlZCBmb3IgUFNOUiBhcyB3ZWxsIGFzIHRocmVlIFJELWN1cnZlcyAob25lIGZvciBlYWNo DQo+IGNoYW5uZWwpIGZvciBNUy1TU0lNLuKAnSBJbiByZWZlcmVuY2VzIFszLCAxMV0sIHRoZXNl IDIgcXVhbGl0eSBhc3Nlc3NtZW50DQo+IG1ldHJpY3MgYW5kIHRoZSB3YXlzIG9mIGhvdyB0byBj YWxjdWxhdGUgdGhlbSBhcmUgZGVzY3JpYmVkIGluIGRldGFpbC4NCg0KR290IGl0LiAgVGhhbmtz Lg0KDQo+ID4tLSBXaGF0IGRvZXMgdGhlIHRleHQg4oCcKGluIGEgc2VwYXJhdGUgZG9jdW1lbnQg b24gSW50ZXJuZXQgdmlkZW8gY29kZWMNCj4gdGVzdGluZynigJ0gbWVhbj8NCj4gPiBQZXIg4oCc QSBsaXN0IG9mIHZpZGVvIHNlcXVlbmNlcyB0aGF0IHNob3VsZCBiZSB1c2VkIGZvciB0ZXN0aW5n IGFzIHdlbGwgYXMgdGhlDQo+IDEwIFFQIHZhbHVlcyBmb3IgdGhlIHJlZmVyZW5jZSBjb2RlYyBh cmUgZGVmaW5lZCBpbiBhIHNlcGFyYXRlIGRvY3VtZW50ICIsDQo+IHdoYXQgZG9jdW1lbnQgaXMg dGhhdD8gIElzIGl0IGRyYWZ0LWlldGYtbmV0dmMtdGVzdGluZz8NCj4gQWNjb3JkaW5nIHRvIG15 IHVuZGVyc3RhbmRpbmcsIGl0IGlzIGRyYWZ0LWlldGYtbmV0dmMtdGVzdGluZy4NCg0KQ2FuIHlv dSBwbGVhc2UgaGVscCBtZSB3aXRoIHRoZSBzcGVjaWZpYyBzZWN0aW9uIHJlZmVyZW5jZXMgaW4g ZHJhZnQtaWV0Zi1uZXR2Yy10ZXN0aW5nIGFzIEknbSBub3Qgc2VlaW5nIGl0IC0tIHdoZXJlIGlz IHRoZSBleHBsaWNpdCByZWZlcmVuY2UgY29kZWMgYmVpbmcgbmFtZWQgYW5kIGl0cyBhc3NvY2lh dGUgMTAgUVAgdmFsdWVzPyAgRm9yIHdoYXQgaXQncyB3b3J0aCBJIHNlZSBTZWN0aW9uIDQuMSBv ZiB0aGlzIGRyYWZ0IHNheWluZyB0aGF0Og0KDQogICBBcw0KICAgdGhlIHJlZmVyZW5jZSBmb3Ig ZXZhbHVhdGlvbiwgc3RhdGUtb2YtdGhlLWFydCB2aWRlbyBjb2RlY3Mgc3VjaCBhcw0KICAgSEVW Qy9ILjI2NSBbNCw1XSBvciBWUDkgIG11c3QgYmUgdXNlZC4gVGhlIHJlZmVyZW5jZSBzb3VyY2Ug Y29kZSBvZg0KICAgdGhlIEhFVkMvSC4yNjUgY29kZWMgY2FuIGJlIGZvdW5kIGF0IFs2XS4gVGhl IEhFVkMvSC4yNjUgY29kZWMgbXVzdA0KICAgYmUgY29uZmlndXJlZCBhY2NvcmRpbmcgdG8gWzEz XSBhbmQgVGFibGUgOS4NCg0KSSdtIGxvb2tpbmcgZm9yIHNvbWV0aGluZyB0aGF0IGVpdGhlciB1 bmFtYmlndW91c2x5IHBvaW50cyB0byB0aGUgcmVmZXJlbmNlIGNvZGVjK1FQczsgb3IgY2xlYXJl ciBsYW5ndWFnZSB0aGF0IHNheXMgdGhhdCBhbGwgb2YgdGhpcyBpcyBvdXQgb2Ygc2NvcGUgaW4g dGhpcyBhbmQvb3IgdGhlIC10ZXN0aW5nIGRyYWZ0LCBidXQgaXMgY2xlYXIgb24gd2hhdCBpbnB1 dCBpbnRvIHRoZSBwcm9jZXNzZXMgb2YgdGhpcyBkcmFmdCBpcyByZXF1aXJlZC4NCg0KPiA+V2hh dCBkb2VzIOKAnGNvZGVjIGltcGxlbWVudGF0aW9uIChmb3IgYm90aCBhbiBlbmNvZGVyIGFuZCBh IGRlY29kZXIpDQo+IHNob3VsZCBjb3ZlciB0aGUgd29yc3QgY2FzZSBvZiBjb21wdXRhdGlvbmFs IGNvbXBsZXhpdHksIG1lbW9yeQ0KPiBiYW5kd2lkdGgsIGFuZCBwaHlzaWNhbCBtZW1vcnkgc2l6 ZeKAnSBtZWFuPw0KPiBJ4oCZZCBsaWtlIHRvIHRoYW5rIEFkYW0gZm9yIGhpcyBjb21tZW50LiBJ IGFsc28gdGhpbmsgdGhhdCBoaXMgZm9ybXVsYXRpb24NCj4gKCIuLi5zaG91bGQgdGFrZSBpbnRv IGNvbnNpZGVyYXRpb24gdGhlIHdvcnN0LWNhc2UuLi4iKSBpcyBjbGVhcmVyLg0KPiBBIG1vcmUg ZGV0YWlsZWQgZXhwbGFuYXRpb24gb24gdGhpcyBTZWN0aW9uIGNhbiBiZSBmb3VuZCBpbiBteSBy ZXNwb25zZSB0bw0KPiB0aGUgc2VjZGlyIHJldmlld2VyOiBodHRwczovL21haWxhcmNoaXZlLmll dGYub3JnL2FyY2gvbXNnL3NlY2Rpci9RWXgtDQo+IGhsMDdhZWM1WGhncWtsZHFnV0ZHM0lzDQoN CkknZCBsaWtlIHRvIG9mZmVyIGEgZnVydGhlciByZWZpbmVtZW50IG9mIHdoYXQgQWRhbSBwcm9w b3NlZDoNCg0KT3JpZ2luYWw6DQpIb3dldmVyLCBpdCBpcyB3b3J0aCBub3RpbmcgdGhhdCBhIGNv ZGVjIGltcGxlbWVudGF0aW9uIChmb3IgYm90aCBhbiBlbmNvZGVyIGFuZCBhIGRlY29kZXIpIHNo b3VsZCBjb3ZlciB0aGUgd29yc3QgY2FzZSBvZiBjb21wdXRhdGlvbmFsIGNvbXBsZXhpdHksIG1l bW9yeSBiYW5kd2lkdGgsIGFuZCBwaHlzaWNhbCBtZW1vcnkgc2l6ZSAoZS5nLiwgZm9yICBkZWNv ZGVkIHBpY3R1cmVzIHVzZWQgYXMgcmVmZXJlbmNlcykuDQoNCkFkYW06DQpIb3dldmVyLCBpdCBp cyB3b3J0aCBub3RpbmcgdGhhdCBhIGNvZGVjIGltcGxlbWVudGF0aW9uIChmb3IgYm90aCBhbiBl bmNvZGVyIGFuZCBhIGRlY29kZXIpIHNob3VsZCB0YWtlIGludG8gY29uc2lkZXJhdGlvbiB0aGUg d29yc3QtY2FzZSBjb21wdXRhdGlvbmFsIGNvbXBsZXhpdHksIG1lbW9yeSBiYW5kd2lkdGgsIGFu ZCBwaHlzaWNhbCBtZW1vcnkgc2l6ZSAoZS5nLiwgZm9yIGRlY29kZWQgcGljdHVyZXMgdXNlZCBh cyByZWZlcmVuY2VzKS4NCg0KUm9tYW4rQWRhbToNCkhvd2V2ZXIsIGl0IGlzIHdvcnRoIG5vdGlu ZyB0aGF0IGEgY29kZWMgaW1wbGVtZW50YXRpb24gKGZvciBib3RoIGFuIGVuY29kZXIgYW5kIGEg ZGVjb2Rlcikgc2hvdWxkIHRha2UgaW50byBjb25zaWRlcmF0aW9uIHRoZSB3b3JzdC1jYXNlIGNv bXB1dGF0aW9uYWwgY29tcGxleGl0eSwgbWVtb3J5IGJhbmR3aWR0aCwgYW5kIHBoeXNpY2FsIG1l bW9yeSBzaXplIG5lZWRlZCB0byBwcm9jZXNzZXMgdGhlIGlucHV0IChlLmcuLCB0aGUgZGVjb2Rl ZCBwaWN0dXJlcyB1c2VkIGFzIHJlZmVyZW5jZXMpLg0KDQo+ID5QbGVhc2UgYWRkIGFkZGl0aW9u YWwgbGFuZ3VhZ2UgdGhhdCBjb2RlYyBzaG91bGQgYmUgd3JpdHRlbiBpbiBhIGRlZmVuc2l2ZQ0K PiBzdHlsZSBhcyB0aGV5IHdpbGwgYmUgcHJvY2Vzc2luZyB1bnRydXN0ZWQgaW5wdXQuDQo+IEdv b2QgcG9pbnQuIFRoYW5rIHlvdS4gSeKAmWxsIGFkZCBpdC4NCg0KUmVnYXJkcywNClJvbWFuDQoN Cj4gDQo+IC0tDQo+IEJlc3QgcmVnYXJkcywNCj4gQWxleGV5IEZpbGlwcG92DQo+IA0KPiAtLS0t LU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBSb21hbiBEYW55bGl3IHZpYSBEYXRhdHJh Y2tlciBbbWFpbHRvOm5vcmVwbHlAaWV0Zi5vcmddDQo+IFNlbnQ6IFdlZG5lc2RheSwgSnVuZSAx MiwgMjAxOSA0OjI1IEFNDQo+IFRvOiBUaGUgSUVTRyA8aWVzZ0BpZXRmLm9yZz4NCj4gQ2M6IGRy YWZ0LWlldGYtbmV0dmMtcmVxdWlyZW1lbnRzQGlldGYub3JnOyBNbyBaYW5hdHkNCj4gPG16YW5h dHlAY2lzY28uY29tPjsgbmV0dmMtY2hhaXJzQGlldGYub3JnOyBtemFuYXR5QGNpc2NvLmNvbTsg dmlkZW8tDQo+IGNvZGVjQGlldGYub3JnDQo+IFN1YmplY3Q6IFJvbWFuIERhbnlsaXcncyBEaXNj dXNzIG9uIGRyYWZ0LWlldGYtbmV0dmMtcmVxdWlyZW1lbnRzLTA5OiAod2l0aA0KPiBESVNDVVNT IGFuZCBDT01NRU5UKQ0KPiANCj4gUm9tYW4gRGFueWxpdyBoYXMgZW50ZXJlZCB0aGUgZm9sbG93 aW5nIGJhbGxvdCBwb3NpdGlvbiBmb3INCj4gZHJhZnQtaWV0Zi1uZXR2Yy1yZXF1aXJlbWVudHMt MDk6IERpc2N1c3MNCj4gDQo+IFdoZW4gcmVzcG9uZGluZywgcGxlYXNlIGtlZXAgdGhlIHN1Ympl Y3QgbGluZSBpbnRhY3QgYW5kIHJlcGx5IHRvIGFsbCBlbWFpbA0KPiBhZGRyZXNzZXMgaW5jbHVk ZWQgaW4gdGhlIFRvIGFuZCBDQyBsaW5lcy4gKEZlZWwgZnJlZSB0byBjdXQgdGhpcyBpbnRyb2R1 Y3RvcnkNCj4gcGFyYWdyYXBoLCBob3dldmVyLikNCj4gDQo+IA0KPiBQbGVhc2UgcmVmZXIgdG8g aHR0cHM6Ly93d3cuaWV0Zi5vcmcvaWVzZy9zdGF0ZW1lbnQvZGlzY3Vzcy1jcml0ZXJpYS5odG1s DQo+IGZvciBtb3JlIGluZm9ybWF0aW9uIGFib3V0IElFU0cgRElTQ1VTUyBhbmQgQ09NTUVOVCBw b3NpdGlvbnMuDQo+IA0KPiANCj4gVGhlIGRvY3VtZW50LCBhbG9uZyB3aXRoIG90aGVyIGJhbGxv dCBwb3NpdGlvbnMsIGNhbiBiZSBmb3VuZCBoZXJlOg0KPiBodHRwczovL2RhdGF0cmFja2VyLmll dGYub3JnL2RvYy9kcmFmdC1pZXRmLW5ldHZjLXJlcXVpcmVtZW50cy8NCj4gDQo+IA0KPiANCj4g LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQ0KPiBESVNDVVNTOg0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IA0KPiAoMSkgSSB3 b3JyeSB0aGF0IHRoZSBsZXZlbCBvZiBkZXRhaWwgaW4gdGhlIGluIHRoZSBDb21wcmVzc2lvbiBQ ZXJmb3JtYW5jZQ0KPiBFdmFsdWF0aW9uIChTZWN0aW9uIDQuMSkgaXMgaW5zdWZmaWNpZW50IGZv ciBpbXBsZW1lbnRhdGlvbi4gIFNwZWNpZmljYWxseToNCj4gDQo+IChhKSBQZXIg4oCcSW5pdGlh bGx5LCBmb3IgdGhlIGNvZGVjIHNlbGVjdGVkIGFzIGEgcmVmZXJlbmNlIG9uZSAoZS5nLiwgSEVW QyBvcg0KPiBWUDkpLCBhIHNldCBvZiAxMCBRUCBxdWFudGl6YXRpb24gcGFyYW1ldGVyKSB2YWx1 ZXMgc2hvdWxkIGJlIHNwZWNpZmllZCAoaW4gYQ0KPiBzZXBhcmF0ZSBkb2N1bWVudCBvbiBJbnRl cm5ldCB2aWRlbyBjb2RlYyB0ZXN0aW5nKSBhbmQgY29ycmVzcG9uZGluZw0KPiBxdWFsaXR5IHZh bHVlcyBzaG91bGQgYmUgY2FsY3VsYXRlZC7igJ0NCj4gDQo+IC0tIEhvdyBzaG91bGQgYSBzZXQg b2YgUVBzIGJlIHNwZWNpZmllZD8NCj4gDQo+IC0tSG93IHNob3VsZCB0aGUgcXVhbGl0eSB2YWx1 ZXMgYmUgY2FsY3VsYXRlZD8NCj4gDQo+IC0tIFdoYXQgZG9lcyB0aGUgdGV4dCDigJwoaW4gYSBz ZXBhcmF0ZSBkb2N1bWVudCBvbiBJbnRlcm5ldCB2aWRlbyBjb2RlYw0KPiB0ZXN0aW5nKeKAnSBt ZWFuPw0KPiANCj4gKGIpIFBlciDigJxBIGxpc3Qgb2YgdmlkZW8gc2VxdWVuY2VzIHRoYXQgc2hv dWxkIGJlIHVzZWQgZm9yIHRlc3RpbmcgYXMgd2VsbCBhcw0KPiB0aGUgMTAgUVAgdmFsdWVzIGZv ciB0aGUgcmVmZXJlbmNlIGNvZGVjIGFyZSBkZWZpbmVkIGluIGEgc2VwYXJhdGUgZG9jdW1lbnQN Cj4gIiwgd2hhdCBkb2N1bWVudCBpcyB0aGF0PyAgSXMgaXQgZHJhZnQtaWV0Zi1uZXR2Yy10ZXN0 aW5nPw0KPiANCj4gKDIpIFBlciB0aGUgU2VjdXJpdHkgQ29uc2lkZXJhdGlvbnMgU2VjdGlvbiAo U2VjdGlvbiA1KQ0KPiANCj4gLS0gV2hhdCBkb2VzIOKAnGNvZGVjIGltcGxlbWVudGF0aW9uIChm b3IgYm90aCBhbiBlbmNvZGVyIGFuZCBhIGRlY29kZXIpDQo+IHNob3VsZCBjb3ZlciB0aGUgd29y c3QgY2FzZSBvZiBjb21wdXRhdGlvbmFsIGNvbXBsZXhpdHksIG1lbW9yeQ0KPiBiYW5kd2lkdGgs IGFuZCBwaHlzaWNhbCBtZW1vcnkgc2l6ZeKAnSBtZWFuPw0KPiANCj4gLS0gUGxlYXNlIGFkZCBh ZGRpdGlvbmFsIGxhbmd1YWdlIHRoYXQgY29kZWMgc2hvdWxkIGJlIHdyaXR0ZW4gaW4gYQ0KPiBk ZWZlbnNpdmUgc3R5bGUgYXMgdGhleSB3aWxsIGJlIHByb2Nlc3NpbmcgdW50cnVzdGVkIGlucHV0 Lg0KPiANCj4gDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gQ09NTUVOVDoNCj4gLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K PiANCj4gKDEpIFRoZXJlIGlzIHNvbWV0aGluZyBvZGQgYWJvdXQgdGhlIGRvY3VtZW50IGZvcm1h dHRpbmcg4oCTIHRoZSB0aXRsZSBhbmQNCj4gdGhlIGZpcnN0IGF1dGhvciBsYXN0IG5hbWUgaW4g dGhlIGZvb3RlciBpcyB3cmFwcGVkIGluIOKAnDwg4oCmID7igJ0NCj4gDQo+ICgyKSBJdCB3b3Vs ZCBiZSBoZWxwZnVsIHRvIGZvcndhcmQgcmVmZXJlbmNlIHRoYXQgYWNyb255bXMgYXJlIGV4cGxh aW5lZCBpbg0KPiBBcHBlbmRpeCBBLg0KPiANCj4gKDMpIFRoaXMgZHJhZnQgdXNlcyB0aGUgd29y ZHMgc2hvdWxkIGFuZCBtdXN0IHRvIHByZXNjcmliZSBhY3Rpb24uICBXaHkNCj4gd2FzbuKAmXQN Cj4gUkZDMjExOSBjaXRlZCB0byBleHBsYWluIHRoZXNlIHdvcmRzPw0KPiANCj4gKDQpIFNlY3Rp b24gMi4wLiAgQSByZWZlcmVuY2UgdG8gZXhwbGFpbiDigJxZQ2JDciA0OjI6MOKAnSB3b3VsZCBi ZSBoZWxwZnVsDQo+IA0KPiAoNSkgU2VjdGlvbiAyLjEuIFBlciDigJxoaWdoIGVuY29kZXIgY29t cGxleGl0eeKAnSBhbmQg4oCcZGVjb2RpbmcgY29tcGxleGl0eeKAnSwgSQ0KPiBpbml0aWF0ZSBy ZWFkIHRoYXQgYXMgYSBxdWFsaXRhdGl2ZSBtZWFzdXJlLiAgSG93ZXZlciwgdGhlIHRleHQgc2F5 cyDigJx1cCB0bw0KPiAxMHggYW5kIG1vcmXigJ0gc28gdGhhdCBpbXBsaWVzIHNvbWUgcXVhbnRp dGF0aXZlIG1lYXN1cmUuICBXaGF0IGlzIHRoYXQ/DQo+IA0KPiAoNikgU2VjdGlvbiAyLjEuICBF eHBhbmQgUVAgdmFsdWVzIG9uIGZpcnN0IHVzZQ0KPiANCj4gKDcpIFNlY3Rpb24gMi54LiAgVGhl IGxhbmd1YWdlIGFyb3VuZCBjb250ZW50IGRvZXNu4oCZdCBhcHBlYXIgdG8gYmUgY29uc2lzdC4N Cj4gRm9yIGV4YW1wbGU6DQo+IA0KPiAtLSBTZWN0aW9uIDIuMSwgSW50ZXJuZXQgVmlkZW8gU3Ry ZWFtaW5nIHNheXMg4oCcbW92aWVzLCBUVi1zZXJpZXMgYW5kIHNob3dzLA0KPiBhbmQgYW5pbWF0 aW9uLuKAnQ0KPiANCj4gLS0gU2VjdGlvbiAyLjIsIElQVFYgc2F5cyDigJx0ZWxldmlzaW9uIGNv bnRlbnTigJ0NCj4gDQo+IC0tIFNlY3Rpb24gMi41LCBTY3JlZW4gY2FzdGluZyBzYXlzIOKAnGJ1 c2luZXNzIHByZXNlbnRhdGlvbnMg4oCmLCBhbmltYXRpb24NCj4gKGNhcnRvb25zKSwgZ2FtaW5n IGNvbnRlbnQsIGRhdGEgdmlzdWFsaXphdGlvbiwg4oCmLCB2aXJ0dWFsIGRlc2t0b3ANCj4gaW5m cmFzdHJ1Y3R1cmUgKFZESSksIHNjcmVlbi9kZXNrdG9wIHNoYXJpbmcgYW5kIGNvbGxhYm9yYXRp b24sIHN1cGVydmlzb3J5DQo+IGNvbnRyb2wgYW5kIGRhdGEgYWNxdWlzaXRpb24gKFNDQURBKSBk aXNwbGF5LCBhdXRvbW90aXZlL25hdmlnYXRpb24NCj4gZGlzcGxheSwgY2xvdWQgZ2FtaW5nLCBm YWN0b3J5IGF1dG9tYXRpb24gZGlzcGxheSwgd2lyZWxlc3MgZGlzcGxheSwgZGlzcGxheQ0KPiB3 YWxsLCBkaWdpdGFsIG9wZXJhdGluZyByb29tIChEaU9SKSwgZXRjLg0KPiANCj4gV2hhdCB0aGUg ZGlmZmVyZW5jZSBiZXR3ZWVuIFNlY3Rpb24gMi4x4oCZcyBhbmltYXRpb24gYW5kIFNlY3Rpb24g Mi414oCZcw0KPiBjYXJ0b29ucz8NCj4gDQo+IFdoYXTigJlzIHRoZSBkaWZmZXJlbnQgYmV0d2Vl biBTZWN0aW9uIDIuMeKAmXMg4oCcbW92aWVzLCBUViBzZXJpZXMg4oCm4oCdIGFuZCBTZWN0aW9u DQo+IDIuMuKAmXMg4oCcdGVsZXZpc2lvbiBjb250ZW504oCdPw0KPiANCj4gKDgpIFNlY3Rpb24g Mi41LiAgVGhlIHNlbnRlbmNlIOKAnEN1cnJlbnRseSwg4oCm4oCdIGlzIHZlcnkgY2hhbGxlbmdp bmcgdG8gcGFyc2UgYXMgaXQNCj4gaW5jbHVkZXMgaW5saW5lIOKAnGkuZS4s4oCdIGFuZCDigJxl dGPigJ0uDQo+IA0KPiAoOSkgU2VjdGlvbiAyLjUuICBQZXIg4oCccG93ZXJwb2ludCwgd29yZCBk b2N1bWVudHPigJ0sIHRoZXNlIGFyZSBzcGVjaWZpYw0KPiBNaWNyb3NvZnQgcHJvZHVjdHMuICBJ IHJlY29tbWVuZCB1c2luZyBtb3JlIGdlbmVyaWMgbmFtZXMuDQo+IA0KPiAoMTApIFNlY3Rpb24g NC4gIEkgZm91bmQgaXQgY29uZnVzaW5nIHRoYXQgYW4gZXZhbHVhdGlvbiBtZXRob2RvbG9neSB3 YXMgaW4gYQ0KPiByZXF1aXJlbWVudHMgZG9jdW1lbnQuICBJIHdvdWxkIGhhdmUgZXhwZWN0ZWQg aXQgaW4gdGhlIGRyYWZ0LWlldGYtbmV0dmMtDQo+IHRlc3RpbmcNCj4gDQo+ICgxMSkgU2VjdGlv biA0LjEuICBWUDkgbmVlZHMgYSByZWZlcmVuY2UuDQo+IA0KPiAoMTIpIFNlY3Rpb24gNi4gIEkg ZG9u4oCZdCB0aGluayB0aGlzIGVudGlyZSBzZWN0aW9uIGlzIG5lY2Vzc2FyeS4NCj4gDQo+ICgx MykgRWRpdG9yaWFsIE5pdHM6DQo+IC0tIFNlY3Rpb24gMy4gIFN0eWxlIG5pdC4gIHMvY2hhcHRl ci9zZWN0aW9uLw0KPiANCj4gLS0gU2VjdGlvbiA0LjEuIFR5cG8uICBzL2NvbXB1dGdlZC9jb21w dXRlZA0KPiANCg0K From nobody Wed Jun 26 11:22:16 2019 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 33EA21203B4 for ; Wed, 26 Jun 2019 11:22:15 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.998 X-Spam-Level: X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UGcmJzzUKXNh for ; Wed, 26 Jun 2019 11:22:13 -0700 (PDT) Received: from mail-lj1-x22f.google.com (mail-lj1-x22f.google.com [IPv6:2a00:1450:4864:20::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A963C120306 for ; Wed, 26 Jun 2019 11:22:12 -0700 (PDT) Received: by mail-lj1-x22f.google.com with SMTP id 205so3206450ljj.8 for ; Wed, 26 Jun 2019 11:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=UKVbD5a1XoY4p9u09UDrJDmmrvQ+OcKy+gzuHPfinw8=; b=J0P2pyrn1THyFbjQfchfc5p6mfRBqxCm8oxRF1R/oJGxscio8DCNf26sR59UBXh53M a2mlozj6sapHHJCcok1s93oEXuiyMWs62j4eH7fExKgKXzzifISh+kFJEhLfVcDjFghy DTZuzQvBHhH8Zeu8pecKBq0fSL835toIzLyymjWo4kVh6x5akEEJ7pqW6X2mit9fDNSs AAalOLU8eKdBV3FAPG051XVGXGYhF6/kwpkya1YanAu2QGwJ/JRAJoSapAUswNXcm59P xdoDKBqneazfYY9qN+6UrljFHgBMU3bKFySuoFC8Dq8vQQisqwpSPfWj/iNSYvmSSfRv 65hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=UKVbD5a1XoY4p9u09UDrJDmmrvQ+OcKy+gzuHPfinw8=; b=QCneV0H1x19xyerXVltr4syw4iYwUaiZMaCw5UeaPPjkk/rUURBsN9mNgrNjxV2Nrl trxAqKoZNTJPDfbhc/tJMF4Uejxf/JsaGnYbCZbZC7hpB/RwIcE86ObsyglrQt97lUOu Bl2BzVO/zVsERoXDX/9BMSTGa8i2pIrb1ZvIIij4Zt7EdlgyNpY/VWXC1t5TWi/GVMIk y6XNfBldJHCXSv/MKshJaJ9sExOH2Db23N6Yz/HvpZ+4m11sAZCMdoiYTRY1wxRZ47Bt oBTiFGvgIgiPjkhV1okv3yUyOtKQigSmVLTnQI+/swjyk7jmdpQU4vZJe4glEbldQ/4W KtMQ== X-Gm-Message-State: APjAAAV2ymtuHclfDpNSYROWT/PuWEq7jNu8XWsVGHz48L47hUPocwVp sAqcEyUeZ5LnRF6OjseltxUWPbAQ/FIgB2onpwwzd/z0+VU= X-Google-Smtp-Source: APXvYqyLdiNBnCJcao3d42fhsQ4X/myw7FsWgZ34jbR2aFXZ1pB7bSiSjXN061cDgO7aDee2VaksiE918wH8O5PIQ/4= X-Received: by 2002:a2e:94cb:: with SMTP id r11mr3624740ljh.212.1561573330401; Wed, 26 Jun 2019 11:22:10 -0700 (PDT) MIME-Version: 1.0 From: Raphael Canut Date: Wed, 26 Jun 2019 20:21:58 +0200 Message-ID: To: video-codec@ietf.org Content-Type: multipart/alternative; boundary="000000000000a63d3b058c3e1fb0" Archived-At: Subject: [video-codec] NHW Project - state-of-the-art for high quality X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Jun 2019 18:22:15 -0000 --000000000000a63d3b058c3e1fb0 Content-Type: text/plain; charset="UTF-8" Hi, I contacted you quickly about the NHW Project as I wanted to reach the professionals in image/video compression. This last month I maybe re-oriented the NHW Project to high quality (up to -l9 quality setting), and I made quite large visual tests on many images (at -l9, -l5, -l3 and -h3 quality settings), and my conclusion is that clearly the NHW Project is visually better than AOM AV1 (AVIF) and HEVC at high quality, -because it has more neatness-. I have also worked these last months with a swiss engineer and have adapted the NHW Project to power of 2 square tiles, and so now we can decompose any image size into square tiles.Don't have coded a deblocking filter for now but I suspect that the major drawback of this approach will be artifacts at tile boundaries, notably at high compression... The ideal would be to adapt directly the NHW Project to the whole image dimensions (and not decompose into tiles), but this is a bigger task... and in fact I contacted you because as it seems confirmed that the NHW Project is very good and extremely competitive at high quality/mid compression, would there be companies/organizations that could sponsorize me to adapt the NHW Project to any image size (would require a cheap 3 to 6 months contract) and maybe want to push a little further the NHW Project? The other advantages of the NHW Project are that it is new royalty-free technology, and it is very fast to compute (encode/decode), a lot faster than AVIF and HEVC for example. Again, if you find there is some interest in the NHW Project, I am really looking for a company to back the NHW Project, which will help a lot to give the project weight in organizations and comitties like JPEG and Alliance for Open Media. Looking forward some further development, Cheers, Raphael Canut --000000000000a63d3b058c3e1fb0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I contacted you quickly = about the NHW Project as I wanted to reach the professionals in image/video= compression.

This last month I maybe re-oriented = the NHW Project to high quality (up to -l9 quality setting), and I made qui= te large visual tests on many images (at -l9, -l5, -l3 and -h3 quality sett= ings), and my conclusion is that clearly the NHW Project is visually better= than AOM AV1 (AVIF) and HEVC at high quality, -because it has more neatnes= s-.

I have also worked these last months with = a swiss engineer and have adapted the NHW Project to power of 2 square tile= s, and so now we can decompose any image size into square tiles.Don't h= ave coded a deblocking filter for now but I suspect that the major drawback= of this approach will be artifacts at tile boundaries, notably at high com= pression...

The ideal would be to adapt directly t= he NHW Project to the whole image dimensions (and not decompose into tiles)= , but this is a bigger task... and in fact I contacted you because as it se= ems confirmed that the NHW Project is very good and extremely competitive a= t high quality/mid compression, would there be companies/organizations that= could sponsorize me to adapt the NHW Project to any image size (would requ= ire a cheap 3 to 6 months contract) and maybe want to push a little further= the NHW Project?

The other advantages of the NHW Project are that it is new=20 royalty-free technology, and it is very fast to compute (encode/decode), a lot faster than AVIF and HEVC for example.

Again, if you find there is some interest in the = NHW Project, I am really looking for a company to back the NHW Project, whi= ch will help a lot to give the project weight in organizations and comittie= s like JPEG and Alliance for Open Media.

Looking f= orward some further development,

Cheers,
Raphael Canut



<= br>
--000000000000a63d3b058c3e1fb0--