FAQ

Instead of answering the same question over and over I've decided to put it into an FAQ this time.

Q1: Why did you use setting X for codec Y?

Because the developers told me to. I've been in contact with the developers of each codec featured in this comparison and asked them for their parameter suggestions for each source. I've tried to the best of my ability (triple checking included) to use exactly those settings. If these suggestions don't show a codec in the best possible light, I am not the one to blame.

In case the company / team working on a codec did not reply (in time), I picked the most advanced settings possible. Once again, if you do not agree with those, ask the people in question to reply to my enquiry next time as this is the only way to ensure any optimized settings can be used in the future.

Keep in mind that settings are to reflect a good compromise between quality and speed. Quality is most important for the review, but encoding speed does matter to most people. Hence, you will generally find settings that will result in higher quality (but normally only in terms of mathematically higher quality, the settings that bring an encoder to a crawl are generally the ones whose effect you can hardly spot at all), but their speed detriment is so considerable that they make no sense for everyday use.

Q2: Why did you not use anamorphic encoding?

Because the only difference it makes for PC playback is that for a given playback solution you get a higher bits/pixel ratio. So, if you do this for all the codecs it's no different than adjusting the resolution. And tests in my forum have shown that as soon as resizing enters the picture, the benefit of anamorphically encoding is questionable.

Q3: Why didn't you use movie X for test 1/2/3?

The reason is twofold: First these sources aren't as easy as you might think and have worked well in the past. Second, using the same sources allows you to compare results of different comparisons, something which is not possible when changing sources.

But while I try to reuse sources, I will change them as necessary. Such change will usually be announced in a comparison when I feel that a source has outlived its usefulness and just isn't hard enough anymore.

Q4: Why didn't you use a lower bitrate for the Futurama test? (doesn't apply to any comparisons after 2004)

What's wrong with 90 minutes of 4:3 video on a single CD? That's a normal value and I don't care how a codec does at 200 kbit/s as I'd never use such a low bitrate to mutilate my movies. Also read the previous question.

Q5: Can I get the original uncompressed images?

With the introduction of the PNG screenshots, this is no longer necessary but if you can give me a good enough reason why you absolutely need these screenshots, I'm always archiving the originals.

Q6: Why didn't you test codec X?

If you have ever done a comparison like this you'll know that it is impossible to test everything. I choose the codecs that are actually used in the community and not only by a few people. It is also a waste of effort to test the same codec twice if it has not been improved in between comparisons.

The codecs included in a comparison will always change from year to year, based on their popularity, potential and my personal interest.

Codecs cannot be tested if they cannot conform to the main requirements of the comparison:

A full list of the requirements and all the information available to codec makers will be made available at a later point.

Q7: Your comparison sucks. Why don't you make a real comparison? I'll tell you what to do..

Please tell the guy who is holding you at gunpoint and forces you to read my article to let out his anger somewhere else. If there's nobody behind you, then ask yourself: who forced you to come here?

Q8: How do I use the zooming and image switcher features?

To change to a certain codec, simply click on the appropriately named hyperlink to the right of a screenshot. To zoom in or out, press the zoom in / zoom out buttons. If you want to scroll in the zoomed area, move your mouse over the small preview image, and move the yellow rectangle around. Click your mouse to freeze the position of the zooming window. To move the rectangle again, click on the preview window again.

Q9: Why does your fancy image switcher / zoomer not work in my browser?

These features have been tested in IE6 SP1, Mozilla 1.6b, Firebird 0.7 (now called Firefox) and Opera 7.2.3. We cannot support every browser there is but the mentioned browser represent the most up-to-date version of all major browsers. If you prefer something else, then you'll just have to live with the disadvantages your selection involves.

Q10: Why don't you use numeric quality metrics for your evaluation?

I've been given PSNR optimized settings for 3 codecs, which were different from the settings for optimal visual quality. Does that inspire your confidence in quality metrics? The fact that codec developers supply different settings should make you realize that quality metrics cannot compete with visual evaluation Furthermore, last time I released a codec comparison, the respected German computer magazine c't had a codec test, and they used the JNDMetrix, a commercial and widely used quality metric (and probably the best there is). And what were the results? VP3 came out on top. Now wait a minute, isn't VP3 that codec that looked like s***? Download it and encode your own movies and you shall see. And that codec should be the best? Then I rather trust my own eyes.

Q11: Why did you use postprocessing to review the clips?

If you have to ask this I suspect I'll never be able to convince you otherwise but I'll try anyway:

1) Most people use postprocessing. If you download and install a codec to view a movie, you'll have postprocessing automatically activated.

2) The DEC in codec means decoder. The decoder is an intricate part of a codec and postprocessing is one of the advanced features of a decoder. Good postprocessing can have quite an impact on quality and hence it makes sense to include it in a test.

3) While not specifically having asked for it, many times I have received playback setting tips along with the codec parameters from codec developers and guess what: they all told me to use postprocessing.

4) I don't like a non postprocessed picture. I see this on my DivX capable standalone.. the image is so much more visually pleasing on my PC monitor that I wouldn't even consider turning postprocessing off.

Q12: Why don't you use an anime source instead of Futurama? (deprecated by the 2005 comparison)

Despite what the anime crowd (which is really loud) wants you to believe, more people watch regular animated features than Anime. Look at all the cartoons, and mainstream Hollywood movies (Disney anyone), that attract a much wider audience than Anime (which is a special interest genre).

Q13: Why are the images reloaded from the server each time I zoom in / out?

They shouldn't be. We have tested this quite thoroughly but it seems that in irregular intervals, an image is indeed reloaded. We suspect this occurs when the browser cache is full. As the high bandwidth version contains more than 9 MB of image material, your browser cache should be flushed prior to viewing the comparison and you should consider increasing your browser cache size.

Q14: Why is film grain in the decoder not an allowed feature?

Actually, it is allowed if we have an integrated grain solution, where grain is parametrized and stored on the encoding side, and then decoded on the decoding side. But if we're talking about just adding random noise, I feel this would falsify the results of a comparison.

Q15: How long does a codec comparison take?

Well, here's a quick rundown on the 2004 shoot-out:

It started back in late September when I got the first build of new codecs (Moonlight and HDX4). In the following months, I did some encoding but with a very low priority, to determine if those codecs could be included. Turns out I've had quite a hard time with Moonlight, but I eventually ended up watching some encoded HDX4 content and decided to include the codec in the upcoming comparison.

The real work started in late November, when I started my search for a replacement of the Matrix source. I encoded both Matrix follow-ups in a bunch of codecs, and sample-reviewed them (watching a full movie encoded several time in short order is not something I plan to ever do). I finally settled for Matrix Revolutions, which I considered to be hard enough to encode. I sent out invitations to participate in the comparison on December 5th to all the companies that are now included in the comparison. They had until December 19th to deliver a codec build to be tested, along with settings for each source. There was a lot of going back and forth and explaining certain things in the two weeks.

I got the first encoders in the middle of the 2nd week, and I started encoding on Friday December 17th, using the NeroDigital Main Profile codec. I had two PCs available, my 2.8 GHz P4 notebook, and my Athlon64 3500+ box. Usually, there's some repetition in this process, due to human error, or technical problems. That was no different this time: I had to re-encode SPR in RV10 because my notebook ran out of space. Similarly, when I though I was done encoding on Tuesday evening and was multiplexing video with audio, I discovered that all Futurama clips encoded on my notebook were interlaced, so they had to be redone (I accidentally forgot to include the deinterlacer in the script): RealVideo, 3ivX and VP6.2. I also redid all 3ivX clips even prior to that (on Tuesday morning and evening), because I only got partial settings from 3ivX until this point (somebody got sick and couldn't answer his email), and even with those I managed to pick the wrong settings. Futurama was re-encoded on Tuesday evening/night, and it was at that time that the High Profile encoder finally arrived (I allowed that late entry because it was not certain until that point if a working encoder could actually be delivered, so they had to submit a fully working encoder prior to the deadline which would certainly be tested, and with the option to also include a working high profile encoder), and finally on Wednesday I got a new HDX4 build that I also included (newcomer slack.. obviously once it was review time that slack was used up ;).

So we have 10 codecs, 3 sources, that's 30 encoding sessions. Plus a complete redo of 3ivX and HDX4, that's 36, SPR RV10 was done twice, same for Futurama RV10, 3ivX and VP6.2, so that's 40 encoding sessions in total. I guess I can say that both my machines are rock stable since there was not a single crash during that time of very intense usage.

The review started on the 23rd, with about 4 hours of SPR review. I started reviewing Futurama on the 24th, but being Christmas even I could not finish. Then I installed HalfLife2 and was busy with that until the afternoon of the 26th. I spent the entire evening of the 26th reviewing the rest of Futurama and Matrix.

On the 27th, I spent a whopping 13h30 taking screenshots (4h15) and writing the review. Especially the first page takes long because of all the info you have to assemble, calculate everything, etc. Then at about 1am on the 28th I discovered that switching to the VMR9 renderer to take screenshots was not a good idea, as it degraded colors for almost all codecs (interestingly enough AviSynth was not concerned, and NeroDigital used another renderer anyway), so I had to redo 8 out of 11 shots for each scene again on the 28th (another 2 - 2.5h), and finally there was proofreading, converting the images, and update the javascript so that you can switch between the codecs and zoom in and out in the new screenshots, not the ones from last year.

So that's about that. I guess I just stopped you from ever making your own codec comparison, have I not? It doesn't sound like fun, and it isn't, it's just a lot of hard work. In the end I have 146MB of uncompressed screenshots, 70 MB of software (codecs and applications that I didn't already have), 33.4 GB of temporary files (VOBs, audio files, video without audio), and 22.2 GB review files. In addition there are 143 incoming mails, about the same number of outgoing mails and a long MSN Messenger log. Over the next few days, it's cleanup time ;)

Q16: Why the 2.5 MB cutoff to decide if an encoded clip is oversized?

My size target is 700MB and 1400MB respectively. Using standard burning, without overburning and tricks like shorter lead-outs, you can burn about 702.5 MB on a 700 MB CD. Anything else and you have a problem. I do realize that it's possible to overburn, and I suppose that's what you'd do if you end up with a movie that's 3 MB larger than desired, but if you want to be on the real safe side you should not overburn, hence the strict limitation to 2.5 MB. You know how much oversize you get with each codec, so if you do not mind overburning, your red and yellow markers might be different, but that's up to you to decide. And, assuming you're making full use of overburning and short lead outs, you can coax more than 10 MB out of a CD. But in that case you can probably set your size target higher as well.. then why settle for a 700 MB if you could get use another 10 MB and aim for 710MB? And if the red really disturbs you, you can save the page, and edit the table to make it correspond to what you feel is an acceptable cutoff ;)

Q17: Why do you compare the final sizes and not size of encoded video?

First of all, the encoded video will be in a container as well, so it'll have some container overhead as well. You'll never get an actual video stream size unless you derive it (knowing what over head a container has, and the length of the movie in number of frames will allow you to do that). Plus, using the same audio file, the same multiplexing settings, the multiplexing overhead you get for different codecs is the same. For the 2004 codec comparison, the difference in multiplex overhead for all the codecs using the AVI container was no more than 2KB and 3 bytes (for SPR, there was 0 byte difference for Futurama), and that's negligible. Hence, with the same overhead, it does not matter whether I compare the final filesize or the size of the video, but using the former gives you a quicker overview if you'll hit your target size or not, whereas in the latter case, you'd have to perform some calculations to get the results. So, to simplify things, I present the final filesizes.

Q17: Why do you use different containers that have different overheads and thus result in different video bitrates?

What is the alternative? At the point of writing this, there is no container that could contain every of the 2004 codec comparison contestants. And even if it could, such a solution would open a whole can of worms:

1) Blame shifting 1: If the final product ends up oversized or undersized, the codec maker could try to blame this onto the container and his codec acting different in that particular container.
2) Blame shifting 2: If there are any playback problems, the codec maker could blame those on the additional 3rd party filters involved for playback.
3) Overhead calculation: Who provides the actual video bitrate? Codec makers will most likely to refuse to play that game, leaving me stuck with having to derive them on my own and possibly without a proven method that has been around for a while and is known to be working (for the AVI container, GKnot would be such a solution).
4) Codec makers can insist on containers and refuse to participate if another codec container is used.

If I were a codec maker, I'd definitely try to resort to the above if there are problems with my codec. But, if I use their codecs in the environment they are made for, they have no excuses. The overhead calculations for AVI are well known and I can reliably say that if a file is oversized, it's the codecs fault. Similarly, if there's a playback problem, since I'm using the standard components, the error cannot be my setup. And for those codecs that do not use AVI, I can require codec makers to provide a formula to derive the video bitrate, and in absence of any calculation error on my part, I can put the blame for any size problems squarely on the codec.

Last but not least, it is not possible to put any codec into any container directly, which means more work for me, and more calculations as well. Instead of having everything done automatically, imagine having your final size and audio size, subtract audio and overhead from final size to get the actual video size, add the temporary container overhead and derive the video bitrate from that. Sure, it's all very easy as long as you don't have to do it on your own ;)

I realize that this is not an ideal situation, and there might be changed in this in the future (it mostly depends on the cooperation of codec makers and a tool that does all the calculations automatically) but after all, as stated in Q7, feel free to make your own comparison ;)

Q18: Why do you bother with containers and audio at all?

Well, I'd actually prefer not to have to, it just makes my life a lot more complicated. For starters I need to encode audio and then calculate overheads for various containers. Not every audio codec sticks perfectly to the required bitrate, so we have a small discrepancy beginning right there, so compromise one is that I have to live with not perfectly equal audio streams, which introduce a tiny and unnoticeable skew, but still a skew.

Then there's the problem video without audio makes no sense to the end user. Unless you're going to watch a movie that never had audio, who's going to watch without audio? So imho audio needs to be there, even though it introduces imperfections and complications. Consider the alternative: You'd need tools to extract the raw video bitstream from a variety of containers (not every codec supports output to the same container, and most don't support output of the raw bitstreams) to compare the video stream sizes. For AVI and MP4 this shouldn't pose a problem, but what about OGG, ASF and RMVB? So even if that were possible, what about if there's no adequate solution to handle both video and audio together (see Dirac in the 2005 qualification)? If I don't look at audio, I wouldn't have picked that up, and that's a huge liability for a codec.

In the end, once again I'd really like to not have to bother with audio, and container overheads at all (though extracting raw streams from a container is just dealing with overhead in another way), but I feel while this would make my life easier, and while it would be more pure in the sense of a pure codec comparison, it would also be artificial. If you're going to make a DVD backup, it will be put in a certain container and with a certain type of audio.

Q19: Why was the audio changed to CBR in the 2005 comparison?

I struggled with that decision for a long time. It is an uncomfortable decision, but let's consider the options:

All three options have their advantages and disadvantages, each involves a compromise of some sort. In chosing CBR, I tried to ensure the least amount of audio size difference, create the least amount of unfairness between groups of codecs using a certain container, and this choice also migitates container overhead differences to a certain extent, making another imperfection fairer. You can turn it any way you like, container and audio choices will always be bad compromises you have to make. If you can come up with a scenario that involves no compromises and reflects reality as closely as possible, feel free to share it, but if you would just pick another of the compromises outlined here, please turn to Q7 now.

Q20: How do you judge video quality?

Purely on watching the video in full screen mode, sitting at the regular working distance from the screen (not the distance you'd pick to normally watch a video). I divide review scenes in smaller parts and look at the various scenes one after another, going back and forth between codecs to find differences, and intermittently looking at the source as well, to get a feeling of how codecs rate against the source (the AviSynth script, thus having the exact same resolution and size as the encoded video).

The review phase is concluded well before screenshots are even taken. Screenshots are published for two reasons: 1) I can often use them to outline a certain property a codec has (but I mention if a screenshot does not do a codec justice, be it in the positive or negative sense), 2) Who would read a text only comparison?

Q21: Why isn't my favorite codec X not in the 2005 comparison while it was in the 2004 comparison?

Wait a minute, only the qualification round has been done (that's the state by December 18th). As the section on what's new in 2005 says, many codecs are directly qualified for the main round. In total 7 codecs from the 2004 codec comparison are directly qualified. In addition, those codecs that will not participate in 2005 will be mentioned when the main round results are released, along with the reasons those codecs are not included. So relax, just because you don't see a codec in the qualification, if it's a good codec you'll see it in the main round.