From nobody Fri Jan 9 06:20:48 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B8A01A89FB for ; Fri, 9 Jan 2015 06:20:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.909 X-Spam-Level: X-Spam-Status: No, score=-1.909 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, T_RP_MATCHES_RCVD=-0.01, URI_TRY_3LD=0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 15-nnn7KQ7x4 for ; Fri, 9 Jan 2015 06:20:29 -0800 (PST) Received: from mork.alvestrand.no (mork.alvestrand.no [158.38.152.117]) by ietfa.amsl.com (Postfix) with ESMTP id 9E4EB1A8987 for ; Fri, 9 Jan 2015 06:20:29 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by mork.alvestrand.no (Postfix) with ESMTP id 89C637C0D88 for ; Fri, 9 Jan 2015 15:20:28 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at alvestrand.no Received: from mork.alvestrand.no ([127.0.0.1]) by localhost (mork.alvestrand.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6B2PHqJP3MhJ for ; Fri, 9 Jan 2015 15:20:27 +0100 (CET) Received: from [IPv6:2001:470:de0a:27:5018:5754:3883:e84c] (unknown [IPv6:2001:470:de0a:27:5018:5754:3883:e84c]) by mork.alvestrand.no (Postfix) with ESMTPSA id 22B657C0BD8 for ; Fri, 9 Jan 2015 15:20:27 +0100 (CET) Message-ID: <54AFE3AA.8080601@alvestrand.no> Date: Fri, 09 Jan 2015 15:20:26 +0100 From: Harald Alvestrand User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: video-codec@ietf.org Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Archived-At: Subject: [video-codec] An opensource project for codec comparision X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 14:20:34 -0000 Part of the problem of figuring out what video codecs are fit for what purpose is measurements. Unless we can measure something, it's very hard to know if we have achieved it. We know that the problem of measuring something that relates to the quality of video encoding is hard. But we think that it's worth the effort to make an open-source platform that lets us make the measurements we can, and - by virtue of being open source - allows us to at least agree on what we are measuring. Such a platform can never be anything but a work-in-progress, but this seems as good a time as any for calling attention to the fact that it's been made public. The platform we've made is available here: https://github.com/google/compare-codecs The results of the test runs generated by the platform are available here= : http://compare-codecs.appspot.com/ Comments are most welcome! Harald --=20 Surveillance is pervasive. Go Dark. From nobody Fri Jan 9 06:35:12 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0B3B01A1B63 for ; Fri, 9 Jan 2015 06:35:09 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.277 X-Spam-Level: X-Spam-Status: No, score=-3.277 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_MISMATCH_ORG=0.611, HOST_MISMATCH_COM=0.311, RCVD_IN_DNSWL_MED=-2.3, SPF_FAIL=0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1HhdZQmll_tb for ; Fri, 9 Jan 2015 06:35:05 -0800 (PST) Received: from smtp.mozilla.org (mx1.corp.phx1.mozilla.com [63.245.216.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DCB0B1A914B for ; Fri, 9 Jan 2015 06:35:04 -0800 (PST) Received: from [172.17.0.48] (50-78-100-113-static.hfc.comcastbusiness.net [50.78.100.113]) (Authenticated sender: tterriberry@mozilla.com) by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPSA id 108F3F273D for ; Fri, 9 Jan 2015 06:35:04 -0800 (PST) Message-ID: <54AFE717.50207@xiph.org> Date: Fri, 09 Jan 2015 06:35:03 -0800 From: "Timothy B. Terriberry" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 SeaMonkey/2.26 MIME-Version: 1.0 To: video-codec@ietf.org References: <54AFE3AA.8080601@alvestrand.no> In-Reply-To: <54AFE3AA.8080601@alvestrand.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [video-codec] An opensource project for codec comparision X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 14:35:09 -0000 Harald Alvestrand wrote: > Comments are most welcome! I'm very glad to see something like this. A couple of questions: 1. Are you planning on adding any objective quality metrics besides PSNR? 2. The Bjontegaard calculation appears to fit a single cubic to an arbitrary number of points. When comparing small changes using many sample points, this will be highly sensitive to noise at very low and very high qualities (i.e., exactly the least interesting places). The following JCTVC document discusses the issue: http://phenix.it-sudparis.eu/jct/doc_end_user/documents/6_Torino/wg11/JCTVC-F270-v1.zip One proposal in that document is to use Piecewise Cubic Hermite Polynomial Interpolation (PCHIP)... there's Matlab code at that can be used for validation. Although, if there are many sample points in a curve, even piecewise linear interpolation can provide a more accurate result than a single cubic. From nobody Fri Jan 9 07:12:43 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EC19E1A6F11 for ; Fri, 9 Jan 2015 07:12:41 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.91 X-Spam-Level: X-Spam-Status: No, score=-1.91 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SGAdMK-eLPnc for ; Fri, 9 Jan 2015 07:12:40 -0800 (PST) Received: from mork.alvestrand.no (mork.alvestrand.no [IPv6:2001:700:1:2::117]) by ietfa.amsl.com (Postfix) with ESMTP id DE7971A8972 for ; Fri, 9 Jan 2015 07:12:39 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by mork.alvestrand.no (Postfix) with ESMTP id 0D74A7C365F; Fri, 9 Jan 2015 16:12:38 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at alvestrand.no Received: from mork.alvestrand.no ([127.0.0.1]) by localhost (mork.alvestrand.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T5xqc7vqmuEU; Fri, 9 Jan 2015 16:12:36 +0100 (CET) Received: from [IPv6:2001:470:de0a:27:10b:593e:5741:7f05] (unknown [IPv6:2001:470:de0a:27:10b:593e:5741:7f05]) by mork.alvestrand.no (Postfix) with ESMTPSA id A29267C09E2; Fri, 9 Jan 2015 16:12:36 +0100 (CET) Message-ID: <54AFEFE3.5090101@alvestrand.no> Date: Fri, 09 Jan 2015 16:12:35 +0100 From: Harald Alvestrand User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: "Timothy B. Terriberry" , video-codec@ietf.org References: <54AFE3AA.8080601@alvestrand.no> <54AFE717.50207@xiph.org> In-Reply-To: <54AFE717.50207@xiph.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Archived-At: Subject: Re: [video-codec] An opensource project for codec comparision X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 15:12:42 -0000 Den 09. jan. 2015 15:35, skrev Timothy B. Terriberry: > Harald Alvestrand wrote: >> Comments are most welcome! > > I'm very glad to see something like this. A couple of questions: > > 1. Are you planning on adding any objective quality metrics besides PSNR? Yes, definitely. The requirement is that the measurement can be made by an opensource tool that we can integrate (compatible license). Your SSIM tool should fit the bill :-) Even with PSNR, there are a number of tweaks I'd like to make visible: per-Y/U/V PSNR, median per-frame PSNR vs overall clip PSNR, for instance ... these things tend to point out "odd" results. That's why the results component is represented as a key/value set :-) > 2. The Bjontegaard calculation appears to fit a single cubic to an > arbitrary number of points. When comparing small changes using many > sample points, this will be highly sensitive to noise at very low and > very high qualities (i.e., exactly the least interesting places). The > following JCTVC document discusses the issue: > http://phenix.it-sudparis.eu/jct/doc_end_user/documents/6_Torino/wg11/JCTVC-F270-v1.zip > > > One proposal in that document is to use Piecewise Cubic Hermite > Polynomial Interpolation (PCHIP)... there's Matlab code at > > that can be used for validation. Although, if there are many sample > points in a curve, even piecewise linear interpolation can provide a > more accurate result than a single cubic. The "Size AVG" column is in fact piecewise linear interpolation; the reason for giving both is that it gives some place to check if BD-PSNR has gone haywire (really BD-Rate, which is the metric given). I've wondered about the choice of cubic for BD-PSNR; the datasets at the moment are all a maximum of 4 points, so cubic (dimension = points - 1) seems appropriate, but with more points, cubic seems like a more random choice. The paper actually suggests going with 8 points and a degree-6 polynomial as one possible approach. Another thing I would like to do is to include a side-by-side clip player into the site, so that we could actually show the clips; this could be the basis for adding some subjective evaluation into the mix - never a really bad thing. But of course that bloats the storage requirements by quite a large factor! > > _______________________________________________ > video-codec mailing list > video-codec@ietf.org > https://www.ietf.org/mailman/listinfo/video-codec From nobody Fri Jan 9 07:23:57 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 795C81A017D for ; Fri, 9 Jan 2015 07:23:56 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.277 X-Spam-Level: X-Spam-Status: No, score=-3.277 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_MISMATCH_ORG=0.611, HOST_MISMATCH_COM=0.311, RCVD_IN_DNSWL_MED=-2.3, SPF_FAIL=0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U4fNYsTN-baL for ; Fri, 9 Jan 2015 07:23:54 -0800 (PST) Received: from smtp.mozilla.org (mx1.corp.phx1.mozilla.com [63.245.216.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A21171A009E for ; Fri, 9 Jan 2015 07:23:54 -0800 (PST) Received: from [172.17.0.48] (50-78-100-113-static.hfc.comcastbusiness.net [50.78.100.113]) (Authenticated sender: tterriberry@mozilla.com) by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPSA id 0DACEF2749 for ; Fri, 9 Jan 2015 07:23:54 -0800 (PST) Message-ID: <54AFF289.2040607@xiph.org> Date: Fri, 09 Jan 2015 07:23:53 -0800 From: "Timothy B. Terriberry" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 SeaMonkey/2.26 MIME-Version: 1.0 To: video-codec@ietf.org References: <54AFE3AA.8080601@alvestrand.no> <54AFE717.50207@xiph.org> <54AFEFE3.5090101@alvestrand.no> In-Reply-To: <54AFEFE3.5090101@alvestrand.no> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Archived-At: Subject: Re: [video-codec] An opensource project for codec comparision X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jan 2015 15:23:56 -0000 Harald Alvestrand wrote: > The "Size AVG" column is in fact piecewise linear interpolation; the > reason for giving both is that it gives some place to check if BD-PSNR > has gone haywire (really BD-Rate, which is the metric given). Ah, I'd missed that, thanks. > seems appropriate, but with more points, cubic seems like a more random > choice. The paper actually suggests going with 8 points and a degree-6 > polynomial as one possible approach. Well, the problem with higher degree polynomials is that you run into precision problems. Degree 6 is about the highest you can fit into 64-bit doubles with any usefulness, but even that has issues sometimes. If you need more degrees of freedom to describe something, piecewise interpolation seems the safer route. > never a really bad thing. But of course that bloats the storage > requirements by quite a large factor! Perhaps combined with the median per-frame idea, you could just show an image or two. Not as good as video, of course, but a lot smaller. From nobody Fri Jan 23 13:45:25 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 196E41A8775 for ; Fri, 23 Jan 2015 13:45:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.7 X-Spam-Level: X-Spam-Status: No, score=0.7 tagged_above=-999 required=5 tests=[BAYES_50=0.8, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OsaH_PPS03JV for ; Fri, 23 Jan 2015 13:45:04 -0800 (PST) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C49EE1A871A for ; Fri, 23 Jan 2015 13:45:03 -0800 (PST) Received: by mail-lb0-f169.google.com with SMTP id f15so9279023lbj.0 for ; Fri, 23 Jan 2015 13:45:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=+ZYn1GgmODVbtR1vWpSihptSogT8/Mbul0cwhbuAHFQ=; b=LqKHuKNFBjsdLruYNhu9k4AD6eCPuJSYkJN99pXs8lXndRcqjj5quV8MFIJ2BaGbwV FL709SdEpCVACzQHl2h6kX4UEeDSClWx9RS18gFOJRPxkYU7ppPab8ZnE1hZOx8m8Fw+ /QNT6zQl4uQVWfWWz3BLIWB8WaWhUmoIxWqSkB8AG36qoqvSE7srbWs30rvWzUE5MZ8C rqf/SHlHwig39XstfkjguUkfIWxC0vFCKAEkWgVw0xKmXhBzfa0b79xNjmX8whExQ3Xj wHuD0Dx+S8ZdXP7vaMsUAwQlLrsntu9z5vkMkaJPIYJFVS2bKm4eEpcOA0Tz0NPiMh/H /SNw== MIME-Version: 1.0 X-Received: by 10.112.13.103 with SMTP id g7mr9692113lbc.29.1422049502301; Fri, 23 Jan 2015 13:45:02 -0800 (PST) Received: by 10.112.31.44 with HTTP; Fri, 23 Jan 2015 13:45:02 -0800 (PST) Date: Fri, 23 Jan 2015 13:45:02 -0800 Message-ID: From: Monty Montgomery To: "video-codec@ietf.org" Content-Type: text/plain; charset=UTF-8 Archived-At: Subject: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jan 2015 21:45:07 -0000 Hi everyone, Testing discussions are already happening in private and elsewhere. The basic points are usually pretty similar and I expect they're going to keep coming up, so I'm going to try centralizing it here. The Daala team is using a small number of fairly short test sequences in an automated testing system ('Are We compressed Yet?) mostly as a way of doing regression testing. It's also used to sanity check small improvements that are unlikely to be apparent in visual testing. The various graphs we've shown in demos and the like are coming from AWCY. One point that's been raised several times is that we're not testing reasonable or practical bitrates. Or, rather, that our range includes the typical useful rates, but we also go way up to insanely high rates no one would ever use. That's mainly because AWCY is meant as a sanity/regression check, and many subtle bugs we've had in the past have only popped out at near-lossless quality levels. Automated testing should report these rates, but we don't intend to use them in evaluation of relative practical performance. Another point that keeps coming up is that we should come up with a standard set of sequences and rates for trading and evaluating first- and probably second-line evaluations. Full agreement there, though our own frontline tetsing right now is only using a handful of clips of a few seconds each (though our available library is much larger). Monty From nobody Sun Jan 25 12:58:26 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 74F5C1A1A95 for ; Sun, 25 Jan 2015 12:58:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.078 X-Spam-Level: X-Spam-Status: No, score=-0.078 tagged_above=-999 required=5 tests=[BAYES_40=-0.001, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9aKR1T5JobBe for ; Sun, 25 Jan 2015 12:58:21 -0800 (PST) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C03541A1AD2 for ; Sun, 25 Jan 2015 12:58:20 -0800 (PST) Received: by mail-wi0-f176.google.com with SMTP id em10so6378885wid.3 for ; Sun, 25 Jan 2015 12:58:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=Nq5jC7vmKW2+E1mYpa8cBA9jQMRMuO4zApqtWVguSPk=; b=YOIutLYxAAUmfG5OTr0ZA7o/xQ6kA0sqRBX6LMIAM5o5SOSaTB9W61roWWv9EwteRf BGrRrIRg9EQHjjIfv7nwXDZgfPjXUZH0TcCd6YZFn2RR2xVNxOqspBMOj7JxDfG5UOox XHF++dxzzTFD2HmLyoNQWWqL04h9fzNzklWN5DvA4bcMXbmZqhL9O6uWRIgcnecFG5wr 3MwTIXFHZ4/1CibSeGedFyp3LSdHSEONAPbd8mE18cF0hqTH/m6VmyDKHhBHHI7sVx0o tX+z4JgtED7X04J1mVF1XtFsz7C4GQ0IAn23wYdhXj0dWrulPHJphD94geiXCl2D5zFS MTkA== X-Gm-Message-State: ALoCoQmxzAArjBD2H2lzXn4GzrEzttzhwRkIL2Po4WXC7CQy4z1WjY5EKYFJNrOZnpCsVgIbeA+l X-Received: by 10.180.73.239 with SMTP id o15mr25341874wiv.14.1422219499583; Sun, 25 Jan 2015 12:58:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.27.142.215 with HTTP; Sun, 25 Jan 2015 12:57:39 -0800 (PST) In-Reply-To: References: From: Eric Rescorla Date: Sun, 25 Jan 2015 12:57:39 -0800 Message-ID: To: Monty Montgomery Content-Type: multipart/alternative; boundary=f46d0435c00810d162050d804877 Archived-At: Cc: "video-codec@ietf.org" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jan 2015 20:58:23 -0000 --f46d0435c00810d162050d804877 Content-Type: text/plain; charset=UTF-8 On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery wrote: > Hi everyone, > > Testing discussions are already happening in private and elsewhere. > The basic points are usually pretty similar and I expect they're going > to keep coming up, so I'm going to try centralizing it here. > > The Daala team is using a small number of fairly short test sequences > in an automated testing system ('Are We compressed Yet?) mostly as a > way of doing regression testing. It's also used to sanity check small > improvements that are unlikely to be apparent in visual testing. The > various graphs we've shown in demos and the like are coming from AWCY. > > One point that's been raised several times is that we're not testing > reasonable or practical bitrates. Or, rather, that our range includes > the typical useful rates, but we also go way up to insanely high rates > no one would ever use. That's mainly because AWCY is meant as a > sanity/regression check, and many subtle bugs we've had in the past > have only popped out at near-lossless quality levels. Automated > testing should report these rates, but we don't intend to use them in > evaluation of relative practical performance. > To sharpen this point a little bit, it's probably useful to distinguish between testing for performance evaluation and testing for regression, continuous integration etc. For the purpose of continuous integration, we should, as you suggest, test all sorts of parameter sets, including insane ones, as well as how the decoder handles bogus encodings, etc. For the purpose of performance evaluation/comparison, I would suggest that it would be useful to try to define a set of strawman scenarios which we would use. I'm sure others have different opinions, but for me the scenarios of most interest are basically videoconferencing, so roughly speaking: - Some set of sizes like: - QCIF - VGA - 1080p - 30fps. - Bit rates on the order of 200kps - 5ish Mbps, depending on the size (I could be a bit off here). Presumably we want some streaming use cases, but I have a less good handle on what typical streaming is... -Ekr Another point that keeps coming up is that we should come up with a > standard set of sequences and rates for trading and evaluating first- > and probably second-line evaluations. Full agreement there, though > our own frontline tetsing right now is only using a handful of clips > of a few seconds each (though our available library is much larger). > > Monty > > _______________________________________________ > video-codec mailing list > video-codec@ietf.org > https://www.ietf.org/mailman/listinfo/video-codec > --f46d0435c00810d162050d804877 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery <xiphmont@gm= ail.com> wrote:
Hi everyone= ,

Testing discussions are already happening in private and elsewhere.
The basic points are usually pretty similar and I expect they're going<= br> to keep coming up, so I'm going to try centralizing it here.

The Daala team is using a small number of fairly short test sequences
in an automated testing system ('Are We compressed Yet?) mostly as a way of doing regression testing.=C2=A0 It's also used to sanity check s= mall
improvements that are unlikely to be apparent in visual testing.=C2=A0 The<= br> various graphs we've shown in demos and the like are coming from AWCY.<= br>
One point that's been raised several times is that we're not testin= g
reasonable or practical bitrates.=C2=A0 Or, rather, that our range includes=
the typical useful rates, but we also go way up to insanely high rates
no one would ever use.=C2=A0 That's mainly because AWCY is meant as a sanity/regression check, and many subtle bugs we've had in the past
have only popped out at near-lossless quality levels.=C2=A0 Automated
testing should report these rates, but we don't intend to use them in evaluation of relative practical performance.

To sharpen this point a little bit, it's probably useful to dist= inguish
between testing for performance evaluation and testing fo= r regression,
continuous integration etc.

For the purpose of continuous integration, we should, as you
su= ggest, test all sorts of parameter sets, including insane ones,
a= s well as how the decoder handles bogus encodings, etc. For
the p= urpose of performance evaluation/comparison, I would suggest
that= it would be useful to try to define a set of strawman scenarios
= which we would use. I'm sure others have different opinions, but
<= div>for me the scenarios of most interest are basically videoconferencing,<= /div>
so roughly speaking:

- Some set of sizes= like:
=C2=A0 - QCIF
=C2=A0 - VGA
=C2=A0 - 10= 80p

- 30fps.
- Bit rates on the order of= 200kps - 5ish Mbps, depending on the size
(I could be a bit off = here).


Presumably we want some stre= aming use cases, but I have a less good
handle on what typical st= reaming is...

-Ekr






Another point that keeps coming up is that we should come up with a
standard set of sequences and rates for trading and evaluating first-
and probably second-line evaluations.=C2=A0 Full agreement there, though our own frontline tetsing right now is only using a handful of clips
of a few seconds each (though our available library is much larger).

Monty

_______________________________________________
video-codec mailing list
video-codec@ietf.org
https://www.ietf.org/mailman/listinfo/video-codec

--f46d0435c00810d162050d804877-- From nobody Sun Jan 25 13:15:13 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 84BC21A0025 for ; Sun, 25 Jan 2015 13:15:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.977 X-Spam-Level: X-Spam-Status: No, score=-1.977 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ggUifwyjNiqF for ; Sun, 25 Jan 2015 13:15:05 -0800 (PST) Received: from mail-wg0-f47.google.com (mail-wg0-f47.google.com [74.125.82.47]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 413201A001C for ; Sun, 25 Jan 2015 13:15:05 -0800 (PST) Received: by mail-wg0-f47.google.com with SMTP id n12so6009716wgh.6 for ; Sun, 25 Jan 2015 13:15:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=4/cHFnlP4K0YIcuMmHxeeJA1m1y1Kf4yYoezRDTWVkM=; b=HRFqTHQ8RmJfvPKZ3j6Z0czpXWLpzhHljPxJscxs5xkOwycw/82k7nHYd0UZG1JarJ 1QB9pf4SfphhlUMtmC0HJIzASoMvNXo8MWKsyf3nOETpM1TRBjPKrKUfy1cc1PRCdcuH xp4HUSzxqcsZ+tBZo7POnNhusOH7P67OBXDq8V12Yi8m08eUHsaPbw9/Z7gjfZJY4YHu OIwX7DGdOtY8ybkH3rcK/Zu6rDLMBWNVLRsnz8oDvpfpbd4EuGfefUO5tsj9RpxyvuMg p9yqBs9LaCOOUFx4iVdN1JHXvoBGYTr9fz9LpFh77gug+PCqSoawY0PCOxgP/pPxdeNv vR9w== X-Gm-Message-State: ALoCoQm69adUIysascEguXG+/ysB0Fd9edES5Sekb9AV7Nz1O5yvNxJBHxiM2mSu4EHEHNgRu7Bi X-Received: by 10.194.108.9 with SMTP id hg9mr37008928wjb.68.1422220503941; Sun, 25 Jan 2015 13:15:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.27.142.215 with HTTP; Sun, 25 Jan 2015 13:14:23 -0800 (PST) In-Reply-To: References: From: Eric Rescorla Date: Sun, 25 Jan 2015 13:14:23 -0800 Message-ID: To: Monty Montgomery Content-Type: multipart/alternative; boundary=047d7bf10b1cee443e050d8083c4 Archived-At: Cc: "video-codec@ietf.org" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jan 2015 21:15:08 -0000 --047d7bf10b1cee443e050d8083c4 Content-Type: text/plain; charset=UTF-8 Following up to myself: what i'm hoping here is to get a consensus list of the domains of interest so that we can then evaluate our progress against them. -Ekr On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla wrote: > > On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery > wrote: > >> Hi everyone, >> >> Testing discussions are already happening in private and elsewhere. >> The basic points are usually pretty similar and I expect they're going >> to keep coming up, so I'm going to try centralizing it here. >> >> The Daala team is using a small number of fairly short test sequences >> in an automated testing system ('Are We compressed Yet?) mostly as a >> way of doing regression testing. It's also used to sanity check small >> improvements that are unlikely to be apparent in visual testing. The >> various graphs we've shown in demos and the like are coming from AWCY. >> >> One point that's been raised several times is that we're not testing >> reasonable or practical bitrates. Or, rather, that our range includes >> the typical useful rates, but we also go way up to insanely high rates >> no one would ever use. That's mainly because AWCY is meant as a >> sanity/regression check, and many subtle bugs we've had in the past >> have only popped out at near-lossless quality levels. Automated >> testing should report these rates, but we don't intend to use them in >> evaluation of relative practical performance. >> > > To sharpen this point a little bit, it's probably useful to distinguish > between testing for performance evaluation and testing for regression, > continuous integration etc. > > For the purpose of continuous integration, we should, as you > suggest, test all sorts of parameter sets, including insane ones, > as well as how the decoder handles bogus encodings, etc. For > the purpose of performance evaluation/comparison, I would suggest > that it would be useful to try to define a set of strawman scenarios > which we would use. I'm sure others have different opinions, but > for me the scenarios of most interest are basically videoconferencing, > so roughly speaking: > > - Some set of sizes like: > - QCIF > - VGA > - 1080p > > - 30fps. > - Bit rates on the order of 200kps - 5ish Mbps, depending on the size > (I could be a bit off here). > > > Presumably we want some streaming use cases, but I have a less good > handle on what typical streaming is... > > -Ekr > > > > > > > Another point that keeps coming up is that we should come up with a >> standard set of sequences and rates for trading and evaluating first- >> and probably second-line evaluations. Full agreement there, though >> our own frontline tetsing right now is only using a handful of clips >> of a few seconds each (though our available library is much larger). >> >> Monty >> >> _______________________________________________ >> video-codec mailing list >> video-codec@ietf.org >> https://www.ietf.org/mailman/listinfo/video-codec >> > > --047d7bf10b1cee443e050d8083c4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Following up to myself: what i'm hoping here is to get= a consensus list of
the domains of interest so that we can then evalua= te our progress against
them.

-Ekr
=


On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla <<= a href=3D"mailto:ekr@rtfm.com" target=3D"_blank">ekr@rtfm.com> wrote:

=
On F= ri, Jan 23, 2015 at 1:45 PM, Monty Montgomery <xiphmont@gmail.com>= wrote:
Hi everyone,

Testing discussions are already happening in private and elsewhere.
The basic points are usually pretty similar and I expect they're going<= br> to keep coming up, so I'm going to try centralizing it here.

The Daala team is using a small number of fairly short test sequences
in an automated testing system ('Are We compressed Yet?) mostly as a way of doing regression testing.=C2=A0 It's also used to sanity check s= mall
improvements that are unlikely to be apparent in visual testing.=C2=A0 The<= br> various graphs we've shown in demos and the like are coming from AWCY.<= br>
One point that's been raised several times is that we're not testin= g
reasonable or practical bitrates.=C2=A0 Or, rather, that our range includes=
the typical useful rates, but we also go way up to insanely high rates
no one would ever use.=C2=A0 That's mainly because AWCY is meant as a sanity/regression check, and many subtle bugs we've had in the past
have only popped out at near-lossless quality levels.=C2=A0 Automated
testing should report these rates, but we don't intend to use them in evaluation of relative practical performance.

To sharpen this point a little bit, it's probably useful = to distinguish
between testing for performance evaluation and tes= ting for regression,
continuous integration etc.

For the purpose of continuous integration, we should, as you
=
suggest, test all sorts of parameter sets, including insane ones,
as well as how the decoder handles bogus encodings, etc. For
the purpose of performance evaluation/comparison, I would suggest
that it would be useful to try to define a set of strawman scenarios
which we would use. I'm sure others have different opinions, but=
for me the scenarios of most interest are basically videoconfere= ncing,
so roughly speaking:

- Some set o= f sizes like:
=C2=A0 - QCIF
=C2=A0 - VGA
=C2= =A0 - 1080p

- 30fps.
- Bit rates on the = order of 200kps - 5ish Mbps, depending on the size
(I could be a = bit off here).


Presumably we want s= ome streaming use cases, but I have a less good
handle on what ty= pical streaming is...

-Ekr
<= div>




<= div>
Another point that keeps coming up is that we should come up with a
standard set of sequences and rates for trading and evaluating first-
and probably second-line evaluations.=C2=A0 Full agreement there, though our own frontline tetsing right now is only using a handful of clips
of a few seconds each (though our available library is much larger).

Monty

_______________________________________________
video-codec mailing list
video-codec@ietf.= org
https://www.ietf.org/mailman/listinfo/video-codec


--047d7bf10b1cee443e050d8083c4-- From nobody Mon Jan 26 17:10:04 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 701651A0264 for ; Mon, 26 Jan 2015 17:10:02 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.101 X-Spam-Level: X-Spam-Status: No, score=-0.101 tagged_above=-999 required=5 tests=[BAYES_20=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l9vS9UUs_v7Q for ; Mon, 26 Jan 2015 17:10:00 -0800 (PST) Received: from mail-lb0-x234.google.com (mail-lb0-x234.google.com [IPv6:2a00:1450:4010:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D43E21A1AE2 for ; Mon, 26 Jan 2015 17:09:52 -0800 (PST) Received: by mail-lb0-f180.google.com with SMTP id b6so10527081lbj.11 for ; Mon, 26 Jan 2015 17:09:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=zGHe9gff97PN+pvSzh5z+bMIDpwlU1ymIr6ngmyHLkM=; b=UbBF3JT0HCa6cLPbyTdOmJOcCho6BK6HUvqk7eZY0VqQRJuCKWomiLKE8qNU32Mumv r9ik/7rmSNxAWxIFrlalz8VnV3XQdUGJIMKrzc7AAgOeyxlHeQnnxT9tY8zR/2S5EaAJ JgHDiHt56FYZjAlRVv7IaUOWlkPvIaxNAkrIe8GWzZ1UE09Ef3Abv0fyELiVNejanR/X UjnH/qIVVZoBzoewrYMHZD56z1kReWUx1w+T83MfB9ou2DLfzF2jbFCnfMeq7kKvgDdI +1QiuJc5jxfo4Jq12FVXQNVc6pKgts7vI+swCLB0jvemx05uOQELLhnFqC4Zt845/aQO MLyg== MIME-Version: 1.0 X-Received: by 10.152.88.44 with SMTP id bd12mr1154490lab.86.1422320991302; Mon, 26 Jan 2015 17:09:51 -0800 (PST) Received: by 10.112.31.44 with HTTP; Mon, 26 Jan 2015 17:09:51 -0800 (PST) In-Reply-To: References: Date: Mon, 26 Jan 2015 17:09:51 -0800 Message-ID: From: Monty Montgomery To: Eric Rescorla Content-Type: text/plain; charset=UTF-8 Archived-At: Cc: "video-codec@ietf.org" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2015 01:10:03 -0000 Right, and our use of AWCY and test clips being mostly regression testing at the moment (and specific to our particular glass box), I wasn't proposing the particulars what we're doing right now as the starting point. Rather, it could be, but used a different way, not as we're using it. Monty On Sun, Jan 25, 2015 at 1:14 PM, Eric Rescorla wrote: > Following up to myself: what i'm hoping here is to get a consensus list of > the domains of interest so that we can then evaluate our progress against > them. > > -Ekr > > > On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla wrote: >> >> >> On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery >> wrote: >>> >>> Hi everyone, >>> >>> Testing discussions are already happening in private and elsewhere. >>> The basic points are usually pretty similar and I expect they're going >>> to keep coming up, so I'm going to try centralizing it here. >>> >>> The Daala team is using a small number of fairly short test sequences >>> in an automated testing system ('Are We compressed Yet?) mostly as a >>> way of doing regression testing. It's also used to sanity check small >>> improvements that are unlikely to be apparent in visual testing. The >>> various graphs we've shown in demos and the like are coming from AWCY. >>> >>> One point that's been raised several times is that we're not testing >>> reasonable or practical bitrates. Or, rather, that our range includes >>> the typical useful rates, but we also go way up to insanely high rates >>> no one would ever use. That's mainly because AWCY is meant as a >>> sanity/regression check, and many subtle bugs we've had in the past >>> have only popped out at near-lossless quality levels. Automated >>> testing should report these rates, but we don't intend to use them in >>> evaluation of relative practical performance. >> >> >> To sharpen this point a little bit, it's probably useful to distinguish >> between testing for performance evaluation and testing for regression, >> continuous integration etc. >> >> For the purpose of continuous integration, we should, as you >> suggest, test all sorts of parameter sets, including insane ones, >> as well as how the decoder handles bogus encodings, etc. For >> the purpose of performance evaluation/comparison, I would suggest >> that it would be useful to try to define a set of strawman scenarios >> which we would use. I'm sure others have different opinions, but >> for me the scenarios of most interest are basically videoconferencing, >> so roughly speaking: >> >> - Some set of sizes like: >> - QCIF >> - VGA >> - 1080p >> >> - 30fps. >> - Bit rates on the order of 200kps - 5ish Mbps, depending on the size >> (I could be a bit off here). >> >> >> Presumably we want some streaming use cases, but I have a less good >> handle on what typical streaming is... >> >> -Ekr >> >> >> >> >> >> >>> Another point that keeps coming up is that we should come up with a >>> standard set of sequences and rates for trading and evaluating first- >>> and probably second-line evaluations. Full agreement there, though >>> our own frontline tetsing right now is only using a handful of clips >>> of a few seconds each (though our available library is much larger). >>> >>> Monty >>> >>> _______________________________________________ >>> video-codec mailing list >>> video-codec@ietf.org >>> https://www.ietf.org/mailman/listinfo/video-codec >> >> > From nobody Tue Jan 27 01:17:29 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 69D6B1A8763 for ; Tue, 27 Jan 2015 01:17:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -14.511 X-Spam-Level: X-Spam-Status: No, score=-14.511 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s1HyCyWxPsbb for ; Tue, 27 Jan 2015 01:17:24 -0800 (PST) Received: from alln-iport-1.cisco.com (alln-iport-1.cisco.com [173.37.142.88]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AEB211A8740 for ; Tue, 27 Jan 2015 01:17:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=4550; q=dns/txt; s=iport; t=1422350243; x=1423559843; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=RGT9ssgz3PDnUmkl69eD8hT63s9opUb+7FKtYc9MKuk=; b=NWgLJf7hs2LC65w75ymNzWkym0uL8I3VXc3N1T7Pe3iltFBakuWM5Gv9 G95vpTmaR4s2p9TLgCdJyPzoJ3hpGwWtd/eOZYOo2aKaxB/oZ2QsTQEve K2dgb0n/f0AYi+75b1R0eW7wjonfjgpA6+NevIXhWtt4H1S/Xrij4Yoef I=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0B4BQDtVsdU/4YNJK1agwZSWQTGSAqFcQKBJUMBAQEBAX2EDQEBBAEBATc0CxACAQgOCh4QIQYLJQIEAQ0FiBgDEQ3OSw2FFwEBAQEBAQEBAQEBAQEBAQEBAQEBARMEjU6CKgeEKQWOboVYggSBRYxGhXYig25vgUR+AQEB X-IronPort-AV: E=Sophos;i="5.09,473,1418083200"; d="scan'208";a="117813541" Received: from alln-core-12.cisco.com ([173.36.13.134]) by alln-iport-1.cisco.com with ESMTP; 27 Jan 2015 09:17:22 +0000 Received: from xhc-aln-x14.cisco.com (xhc-aln-x14.cisco.com [173.36.12.88]) by alln-core-12.cisco.com (8.14.5/8.14.5) with ESMTP id t0R9HMWD032404 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 27 Jan 2015 09:17:22 GMT Received: from xmb-rcd-x14.cisco.com ([169.254.4.163]) by xhc-aln-x14.cisco.com ([173.36.12.88]) with mapi id 14.03.0195.001; Tue, 27 Jan 2015 03:17:22 -0600 From: "Mo Zanaty (mzanaty)" To: Monty Montgomery , Eric Rescorla Thread-Topic: [video-codec] Test sequences, automated and otherwise Thread-Index: AQHQOhIO9CuEN+46kkW30MEsXbTPuQ== Date: Tue, 27 Jan 2015 09:17:21 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.7.141117 x-originating-ip: [64.100.32.216] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Archived-At: Cc: "video-codec@ietf.org" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2015 09:17:27 -0000 Since a particularly interesting comparison will be HEVC, should we consider some JCT-VC standard sequences and operating points? We can redefine the classes of sequences to be the most relevant for initial experiments. For example: Class 1: Video Conferencing a) 360p 30Hz 0.1-1Mbps b) 720p 30Hz 0.2-2Mbps c) 720p 60Hz 0.3-3Mbps d) 1080p 30Hz 0.4-4Mbps e) 1080p 60Hz 0.6-6Mbps f) 2160p 30Hz 1.2-12Mbps Class 2: Video Streaming a) 360p 30Hz 0.2-2Mbps b) 720p 30Hz 0.3-3Mbps c) 1080p 24Hz 0.6-6Mbps d) 2160p 24Hz 2-20Mbps 1b and 2c may be the most interesting operating points for many. We can also consider other classes like professional applications, screen content / wireless display, etc. with relatively higher rates. Mo On 1/26/15, 8:09 PM, Monty Montgomery wrote: Right, and our use of AWCY and test clips being mostly regression testing at the moment (and specific to our particular glass box), I wasn't proposing the particulars what we're doing right now as the starting point. Rather, it could be, but used a different way, not as we're using it. Monty On Sun, Jan 25, 2015 at 1:14 PM, Eric Rescorla wrote: > Following up to myself: what i'm hoping here is to get a consensus list >of > the domains of interest so that we can then evaluate our progress against > them. > > -Ekr > > > On Sun, Jan 25, 2015 at 12:57 PM, Eric Rescorla wrote: >> >> >> On Fri, Jan 23, 2015 at 1:45 PM, Monty Montgomery >> wrote: >>> >>> Hi everyone, >>> >>> Testing discussions are already happening in private and elsewhere. >>> The basic points are usually pretty similar and I expect they're going >>> to keep coming up, so I'm going to try centralizing it here. >>> >>> The Daala team is using a small number of fairly short test sequences >>> in an automated testing system ('Are We compressed Yet?) mostly as a >>> way of doing regression testing. It's also used to sanity check small >>> improvements that are unlikely to be apparent in visual testing. The >>> various graphs we've shown in demos and the like are coming from AWCY. >>> >>> One point that's been raised several times is that we're not testing >>> reasonable or practical bitrates. Or, rather, that our range includes >>> the typical useful rates, but we also go way up to insanely high rates >>> no one would ever use. That's mainly because AWCY is meant as a >>> sanity/regression check, and many subtle bugs we've had in the past >>> have only popped out at near-lossless quality levels. Automated >>> testing should report these rates, but we don't intend to use them in >>> evaluation of relative practical performance. >> >> >> To sharpen this point a little bit, it's probably useful to distinguish >> between testing for performance evaluation and testing for regression, >> continuous integration etc. >> >> For the purpose of continuous integration, we should, as you >> suggest, test all sorts of parameter sets, including insane ones, >> as well as how the decoder handles bogus encodings, etc. For >> the purpose of performance evaluation/comparison, I would suggest >> that it would be useful to try to define a set of strawman scenarios >> which we would use. I'm sure others have different opinions, but >> for me the scenarios of most interest are basically videoconferencing, >> so roughly speaking: >> >> - Some set of sizes like: >> - QCIF >> - VGA >> - 1080p >> >> - 30fps. >> - Bit rates on the order of 200kps - 5ish Mbps, depending on the size >> (I could be a bit off here). >> >> >> Presumably we want some streaming use cases, but I have a less good >> handle on what typical streaming is... >> >> -Ekr >> >> >> >> >> >> >>> Another point that keeps coming up is that we should come up with a >>> standard set of sequences and rates for trading and evaluating first- >>> and probably second-line evaluations. Full agreement there, though >>> our own frontline tetsing right now is only using a handful of clips >>> of a few seconds each (though our available library is much larger). >>> >>> Monty >>> >>> _______________________________________________ >>> video-codec mailing list >>> video-codec@ietf.org >>> https://www.ietf.org/mailman/listinfo/video-codec >> >> > _______________________________________________ video-codec mailing list video-codec@ietf.org https://www.ietf.org/mailman/listinfo/video-codec From rgiles@mozilla.com Tue Jan 27 10:14:58 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BB6891A88DB for ; Tue, 27 Jan 2015 10:14:58 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.979 X-Spam-Level: X-Spam-Status: No, score=-1.979 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N08DzOToCXK7 for ; Tue, 27 Jan 2015 10:14:55 -0800 (PST) Received: from mail-wg0-f49.google.com (mail-wg0-f49.google.com [74.125.82.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 514E91A88FF for ; Tue, 27 Jan 2015 10:14:54 -0800 (PST) Received: by mail-wg0-f49.google.com with SMTP id k14so16267637wgh.8 for ; Tue, 27 Jan 2015 10:14:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=jXQwmvwC6TaRk2FR0yAiAY6Gei4dZRZjfLOcFt83JR8=; b=i+X/ijJa1jHnsvRZ3HAAzClJouymaG9zSXSEn3h+linriUlvRp1zpf58bEYdUvm6ne 1iztNAFaBZyZU4eWfJy+izF4evdlq673NAJyUoHkPopEvmp3N6qPiK4x1KJKEJWv8CII pkd4MlCK/y3McIOgI6ftJAtouhgkuL3XGgvRcPEb1wLf0lbqbMlQZzwaFLpi8U+LCPMH gHYL+mUCO25qqKD33I7ymJ1FIggtUjW+IauZw6TA8m2jrFirkCDnE23a3ITuO1mq9KBO YLSitBXwrniJ5+QGo/ivBWb1mF+UOTeyn4AQfWCQlCQfQZQRd6WxjjsFjRtwAYXxSsNm pApw== X-Gm-Message-State: ALoCoQmQNwJg1nHasB4UbzAIkE28s+0Hhk8G74M41uoReOf7AHkb1Porb8xqnQJOUQXFbIaZvAZf MIME-Version: 1.0 X-Received: by 10.194.189.138 with SMTP id gi10mr5633667wjc.86.1422382492930; Tue, 27 Jan 2015 10:14:52 -0800 (PST) Received: by 10.216.163.129 with HTTP; Tue, 27 Jan 2015 10:14:52 -0800 (PST) In-Reply-To: References: Date: Tue, 27 Jan 2015 10:14:52 -0800 Message-ID: From: Ralph Giles To: "Mo Zanaty (mzanaty)" Content-Type: text/plain; charset=UTF-8 Archived-At: Cc: "video-codec@ietf.org" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jan 2015 18:20:42 -0000 On Tue, Jan 27, 2015 at 1:17 AM, Mo Zanaty (mzanaty) wrote: > Class 2: Video Streaming > ... > c) 1080p 24Hz 0.6-6Mbps > d) 2160p 24Hz 2-20Mbps > > 1b and 2c may be the most interesting operating points for many. > > We can also consider other classes like professional applications, screen > content / wireless display, etc. with relatively higher rates. I think 1080p 60Hz at least should be on the list as well. This is important for game video and other action/interactive use, as your say, and already deployed by YouTube with current codecs. -r From nobody Sat Jan 31 13:05:56 2015 Return-Path: X-Original-To: video-codec@ietfa.amsl.com Delivered-To: video-codec@ietfa.amsl.com Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 602091A1EED for ; Sat, 31 Jan 2015 13:05:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.8 X-Spam-Level: X-Spam-Status: No, score=0.8 tagged_above=-999 required=5 tests=[BAYES_50=0.8] autolearn=ham Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pnVcljdVBznY for ; Sat, 31 Jan 2015 13:05:52 -0800 (PST) Received: from mail.zaytoon.hidayahonline.net (zaytoon.hidayahonline.net [173.193.202.83]) by ietfa.amsl.com (Postfix) with ESMTP id 67B481A1EF2 for ; Sat, 31 Jan 2015 13:05:52 -0800 (PST) Received: from [192.168.1.100] (d47-69-200-237.try.wideopenwest.com [69.47.237.200]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: basilgohar@librevideo.org) by mail.zaytoon.hidayahonline.net (Postfix) with ESMTPSA id 0B102658C71; Sat, 31 Jan 2015 16:05:51 -0500 (EST) Message-ID: <54CD43AC.5070001@librevideo.org> Date: Sat, 31 Jan 2015 16:05:48 -0500 From: Basil Mohamed Gohar Organization: Libre Video User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Ralph Giles References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Archived-At: Cc: "video-codec@ietf.org" , "Mo Zanaty \(mzanaty\)" Subject: Re: [video-codec] Test sequences, automated and otherwise X-BeenThere: video-codec@ietf.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Video codec BoF discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Jan 2015 21:05:54 -0000 On 01/27/2015 01:14 PM, Ralph Giles wrote: > On Tue, Jan 27, 2015 at 1:17 AM, Mo Zanaty (mzanaty) wrote: > > > Class 2: Video Streaming > > ... > > c) 1080p 24Hz 0.6-6Mbps > > d) 2160p 24Hz 2-20Mbps > > > > 1b and 2c may be the most interesting operating points for many. > > > > We can also consider other classes like professional applications, screen > > content / wireless display, etc. with relatively higher rates. > > I think 1080p 60Hz at least should be on the list as well. This is > important for game video and other action/interactive use, as your > say, and already deployed by YouTube with current codecs. > > -r For what it's worth, in defense of this, I've also begun uploading 1080p60 content onto one of the YouTube channels I maintain due to the existing support (source is 1080i60, but looks good and smooth after running yadif=1 on it). -- Libre Video http://librevideo.org