Codec shoot-out 2003 - 2nd installment

Table of content:

1: Introduction
2: Test setup
3: Test 1: The Matrix (normal bandwidth) -- ultra high bandwidth
4: Test 2: Saving Private Ryan (normal bandwidth) -- ultra high bandwidth
5: Test 3: Futurama (normal bandwidth) -- ultra high bandwidth
6: Conclusion
7: Besides the codecs
8: Future outlook
9: FAQ

Welcome to yet another codec comparison. It's been a while since the last one and a lot has happened, not all of it good (taking screenshots was a major pain in the ass.. I should charge 200 bucks an hour to do them in the future). Once more, the current codec comparison includes even more contestants than the last time, even though I could not include every codec I wanted. I still don't have any H.264 codecs for testing since the existing open source implementations are pretty much unusable, and Vanguard Software decided that it was still too early for their H.264 simple profile implementation to be tested. But we're getting closer to a testable codec, and with more and more companies investing in H.264 development, chances are better than ever that we'll see a H.264 codec in the next codec comparison.

The test is the exact same as last time so that the results can be directly compared. I have only included codecs that have been improved since the last time (so no WMV9 this time), and the reference codec SBC.

Now a quick overview of what has happened since the last comparison:

• 3ivX has recently released a new version of their codec. It now supports 2 pass encoding and MPEG-4 Advanced Simple Profile features like QPel motion estimation and MPEG quantizer. 3ivX isn't just a codec, it's a suite of MPEG-4 tools, with an AAC audio encoder, and MP4 multiplexer and demultiplexer.
• DivXNetworks has released two new versions of their DivX5 codec since the last comparison, both aimed to increase quality.
• For the first time, I have included ffvfw - the Windows implementation of the *nix based libavcodec, a versatile video encoding / decoding library that supports more formats that you can possibly imagine - in a test. For reasons beyond my control, I could not use the latest builds but settled for athos' October 28th build.
• Nero Digital is Ahead's complete MPEG-4 solution, including an MPEG-4 Advanced Simple Profile video codec, a high efficiency AAC encoder, and a one click DVD backup tool (Recode 2), that turns a DVD into an MP4 file (including subtitles and chapters).
• Realvideo is still under development. A few months back, a high quality mode (EHQ) was added, which is supposed to not smooth out all the details anymore.
• On2 has released their VP6 codec to the public back in October. For the first time since VP3 was released to the open source community, it is now possible to evaluate On2's offering.
• XviD 1.0 beta builds were released only a few weeks ago, but they are a clear sign that we're getting close to v1.0. In fact, no new features should be added to the 1.0 branch.

Codecs that were not included are the following: Windows Media Video 9. Since the codec has not been changed since the last test, it would only be a waste of time to re-test it. Vanguard Software's H.264 simple profile encoder was not submitted for testing. Vanguard feels that at its current state, the codec still needs improvement (like 2 pass encoding for instance).

Last but not least: In order to present each codec the best way possible, I contacted each codec maker 2 weeks prior to the start of encoding to give them a chance to suggest the best settings for the movies used in this test. So, if you think my testing methods and settings are flawed, you're welcome to take it up with the developers because I stuck to their recommendations and if they gave sub-optimal settings it's their loss. At this point I'd like to thank all people who've worked with me in this area. This time, every company I contacted was willing to work with me on this. Thank you.

Now let me present our contestants:

3ivX D4 4.5.1a3 (not publicly released yet)
DivX3 in the form of SBC (Smart Bitrate Control) as done in Nandub (hereafter referred to as SBC)
DivX5.1.1 Pro
ffvfw build dated 10/28
NeroDigital 4.1.4.11 (special VFW build that is not publicly available, but you can encode to AVI using the NeroDigital DS encoding filter that ships as part of Recode 2). This codec is hereafter referred to as ND.
RealNetworks RealVideo9 based on HelixProducer 9.2 and an optimized DLL. This codec is hereafter referred to as RV9
VP6 build 6.0.8.387 (not publicly available, but you can get build 6.0.9.2)
XviD 1.0 beta 3 (the build used was pre-beta but all the changes made since were bugfixes in areas that do not concern the comparison)

All codecs were tested in a 2 pass setup using the settings suggested by the developers (in case of ffvfw the settings come from the lead developer of libavcodec and ffmpeg, the ffvfw developer only suggested the 2 pass encoding method to duplicate ffmpeg's behavior).

Test setup:

All testing was done on the following hardware:

AMD Athlon XP 2800+
Shuttle SN41 mainboard (part of the Shuttle SN41G2 barebone system)
2x256MB PC2700 Apacer CL2.5 DDR RAM
Creative 3d Blaster 4 Titanium4600 (NVidia Geforce4 TI4600)
Samsung SyncMaster 191T-P Display connected via DVI

To encode I used the following software:

DVD2AVIdg 1.0
mpeg2dec3.dll
AviSynth 2.53 for frameserving
VirtualDub 1.5.10 for encoding (except for RV9)
Nandub 1.0 RC2 lumafix ecffix for SBC
Helix Producer 9.2 (build number 9.2.0.384)

To simplify things I used GordianKnot for the DivX3 encoding and AutoRV9 for the RealVideo encoding.

Movies I encoded:

Matrix - Region1, NTSC, length: 2h16
Saving Private Ryan - Region1, NTSC, length: 2h49. I will call this movie SPR from here on
Futurama Season 2 Episode 1 - Region2, PAL, length: 0h21

Encoding parameters:

I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-preset 128 setting) for all movies, then tried to put Matrix onto 1 700MB CD, SPR onto 2 700MB CDs and 4 Futurama Episodes onto one 700 MB CD (resulting in 175MB for an episode). The size of the audio track was 124221KB, 157455KB and 20582KB respectively. This resulted in an effective video bitrate of 581kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit, DivX3 uses kbit = 1024bits so you have to divide that bitrate by 1.024 to get the effective DivX3 bitrate, all other codecs use k = 1000). Please note that multiplexing a VBR MP3 with an AVI gives more overhead than a regular CBR MP3 file. I used GKnot to calculate the bitrate as it's known to be accurate in such situations.

As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got

Note that 700MB equals 716'800KB, 1400MB equals 1'433'600KB and 175 MB equals 179'200 KB.

As you can see, SBC, DivX5, ND and XviD are getting really close to the target size. The codecs that use KB/s for bitrate control rather than kbit/s (3ivX and VP6) have a problem as they get a too high bitrate for Matrix (73 KB/s) and one that's too low for Futurama (123 KB/s), but that alone is not enough to explain the size discrepancy as even when the bitrate is too low (Futurama) the files were still oversized. Ffvfw is also pretty accurate but I had a major problem encoding: The codec crashed once when encoding Matrix and twice encoding SPR, so I gave up on SPR. Now you'll argue that the codec has never crashed for you, but then again you'll be using XviD's 2 pass encoding mechanism, not the ffmpeg one (and we know XviD's 2 pass mechanism is good so there's no reason to test it twice).

@Update (3/17/2004): It seems both 3ivX and VP6.2 assume 1k = 1024 which would explain the size discrepancy.

RV9 is also still not accurate though at least this time we didn't get any oversized files.

Codec settings (applied w.r.t. the default codec settings)

• 3ivX: For Matrix I used H.263 quantizer and QPel motion estimation. For the other two I've used MPEG quantizer.
• SBC: I've used the GKnot defaults (no anti-shit).
• DivX5: Slow mode, slow psy settings and no profile (b-frames are enabled though)
• ffvfw: Macroblock decision rate: distorted, 4 motion vectors enabled, trellis quantization enabled, subpel ME compare: SATD, fullpel ME compare: SATD, and one consecutive B-frame. For two pass encoding I did the following: constant quantizer (q=2) for the first pass writing the logfile, and CBR during the 2nd pass (selecting Use logfile in the output configuration)
• ND: Best quality 2 pass encoding, MV range: -128 to 127.5, psychovisual high, H.263 quantization, halfpel, spatial prediction, PVOP, unrestricted MV and BVOP (B-frames)
• RV9: Very High EQH and a max bitrate that is 5 times the average bitrate. I used 5.1ch audio at 128kbit/s to be comparable to the other codecs.
• VP6: VP61 Advanced Profile, force keyframe every 99999, Local File playback, Undershoot 90%, Adjust quantizer checked, Min quantizer: 2, Max quantizer: 62. For the 2nd pass: Variability: 70, Min section: 40, Max section: 400.
• XviD: SPR: QPel, Trellis quantization & MPEG quantization. For Futurama I additionally used the cartoon mode and disabled B-frames. Matrix: QPel, 1 b-frame, VHQ1, chroma motion, Trellis quantization and the HVS Good Picture matrix.

Encoding speed:

Here I take Matrix as an example. The resolution of SPR was bigger, hence the performance also dropped. Note that you should take these values with a grain of salt. Only the first 10'000 frames were used for speed measures so the overall average FPS might be a bit different. Furthermore, I did not reset the PC before each measurement and I did not perform several measurements per codec. For good absolute values, more testing would be required, but these values should nevertheless give you a good impression of the relative speed difference between codecs. Note that the values are averaged over the two passes where applicable.

If you've seen the last comparison these numbers might surprise you. On the average, encoding speed has dropped considerably. The slowest codec last time (WMV9, 17.43 fps) would now be in the faster half. RV9 obviously got slower because of the EHQ mode, but DivX5's speed drop is beyond me. I even tested it twice because I wouldn't believe my results. I'm waiting to hear from DXN on the speed subject (encoding was faster on my 2.8 GHz P4 notebook, but still slower than every other codec). I was positively surprised by XviD, which once again is very presentable in the speed area. ND blew everything else out of the water. Considering that the codec has been designed for integrated devices (standalones recording to Nero Digital) that is not so surprising, but we'll see how much quality has been sacrificed for speed.

Other important stuff:

Here's the AviSynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DVD2AVI in Matrix and SPR. Futurama was interlaced so I used Decomb's FieldDeinterlace with no blending to get a progressive image. Note that I used neutral bicubic resizing for every clip. I don't like the blurryness of the bilinear resize filter and the neutral setting is a good compromise between crispness and better compressibility but it doesn't blur.

Matrix script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
mpeg2source("D:\DVD\Projects\THE_MATRIX_16X9LB_N_AMERICA_VTS_02_PGC1\THE_MATRIX_16X9LB_N_AMERICA_ForceFilm.d2v")
crop(0,60,718,356)
BicubicResize(640,272,0,0.5)

SPR script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
mpeg2source("D:\DVD\Projects\SAVING_PRIVATE_RYAN_VTS_01_PGC1\SAVING_PRIVATE_RYAN_ForceFilm.d2v")
crop(2,8,714,464)
BicubicResize(640,352,0,0.5)

Futurama script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\decomb.dll")
mpeg2source("D:\DVD\Projects\FUTURAMA_SEASON_2_DISC_1_VTS_01_PGC1\FUTURAMA_SEASON_2_DISC_1.d2v")
FieldDeinterlace(blend=false)
crop(10,4,702,568)
BicubicResize(608,464,0,0.5)

Playback:

I've used the default playback filter for 3ivX, NeroDigital and VP6, using automatic postprocessing strength selection. Where applicable I disabled the film grain effect.

For SBC and DivX5 I used the DivX5 filter at strength 4. I have been using the DivX5 filter to play back DivX3 movies a lot in the past and whenever I thought that something was odd I switched back to the DivX3 filter only to find out that the problem was the source, not the filter. I have also done the same during the comparison but I'm confident to say that the results would not be any different had I used the DivX3 filter for SBC playback (except for being more of a hassle to compare and make screenshots).

To decode ffvfw I've used ffdshow build dated October 28th.

To play XviD clips I used the XviD DS filter with both deblocking and deringing enabled (and film effect disabled).

All files were reviewed using Media Player Classic as it can play back AVIs and RV9 files (also QT by the way). Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player! As I have a flatscreen monitor, I used the monitor's native resolution of 1280x1024 to review all clips. Flatscreens are not good at displaying lower resolutions full screen, plus zooming the video helps to accentuate any problems a codec might have (and which I would've certainly missed when reviewing at a low quality 640x480 resolution). Note that the screen used has a reaction time lower than 25ms which allows for flawless movie playback (and 3D shooters are okay, too).

A note about the screenshots: I did my best to present shots from the exact same frame. Some codecs exhibited delays (for instance in codec A you take frame X, in codec B you have to take frame X+1), and RV9 behaved rather erratically. In the end I managed to get every shot right, but I assure you that taking that many screenshots from that many codecs is not something you like to do.

Now proceed to the first test (low bandwidth jpeg version. This version loads 200KB of images for the beginning, and the total image size for the matrix test is 1.75 MB). If you have a lot of bandwidth and / or don't mind waiting, there's a high bandwidth PNG version (loads 1.13 MB of images initially and if you want to see all the images, you have to download 9.2 MB). Also note that the high bandwidth version requires that you have enough browser cache left or the images will be reloaded each time you zoom in or out.

This document was last updated on March 17, 2004