Codec shoot-out 2004

Table of content:

1: Introduction
2: Test setup
3: Test 1: Matrix Revolutions (normal bandwidth) -- ultra high bandwidth
4: Test 2: Saving Private Ryan (normal bandwidth) -- ultra high bandwidth
5: Test 3: Futurama (normal bandwidth) -- ultra high bandwidth
6: Conclusion
7: Besides the codecs
8: Future outlook
9: FAQ

Welcome ladies and gentleman, to yet another installment of my (in)famous codec comparisons.

It has been a full year since I last looked at a handful of video codecs and how they are suited for typical DVD backup scenarios. We've seen a lot of activity in the codec market in 2004: For one, there are now several H.264 / MPEG-4 AVC (referred to as AVC henceforth) encoders available. With the adoption of AVC as codec for the upcoming high definition DVD formats (be it Blu-ray or HD DVD), as well as its adoption into digital TV standards, I expect to see a lot of movement in that market in the years to come. Microsoft's WMV9 has also been approved for the upcoming high definition DVD formats, and Microsoft is making some inroads into the television business as well, but so far those are proprietary systems and tied to specific geographical areas.

In the MPEG-4 Advanced Simple Profile (referred to as ASP henceforth) area, we have seen major consumer electronic manufacturers release DVD players that can play ASP content (all with the exception of 3 point GMC which rules out the use of XviD's and NeroDigital's GMC for standalone playback). And improvement to existing codecs has been ongoing throughout the year.

Now a quick overview of what has happened since the last comparison with respect to this year's contestants:

• 3ivX's progress has been rather slow throughout the year. Just in time for the comparison, I got a prerelease of the upcoming 3ivX 5.0 release. It now has a quality / speed slider that allows you a tradeoff between quality and speed, and B-frames are also supported. In addition, 3ivX has been working on an AVC codec, but which was not ready to be included in the comparison.
• DivXNetworks has adopted a policy where alpha and beta codecs will be released to the public for testing. We've seen a bunch of such tests, leading up to the latest official release DivX 5.2.1. There have also been two public alphas of DivX 5.2.2 (I presume this will be the version number used). The focus of DivX' development was higher quality through improvements in B-frames (amongst other things it's now possible to have multiple B-frames) and quantizer matrix selection. The Fusion codec I will be testing has new performance modes that improve quality at the expense of encoding time, rate distortion optimized H.263 quantizers, new aspect ratio based resizing, support for 4MB and improved deblocking.
• Nero Digital, last year's winner in the speed department, has had some work done in the quality department. In addition, Nero's codec partner ateme has been working on an AVC encoder, which was recently released to the public in form of the complete DVD backup solution Nero Recode 2.2. In addition to DVD transcoding, Recode can now convert DVDs to ASP, or AVC, both using (HE) AAC audio, chapters and subtitles using the MP4 container.
• RealVideo 10 has been released in 2004, along with several codec updates. The high quality mode EHQ is now a standard feature, and the development focus has been mostly on quality.
• On2 has released their VP6 codec to the public back in October 03. Since then, the codec has undergone many revisions until March 04, but since then there has been hardly any movement. Just recently, On2 started beta testing an integrated conversion application, which lead to additional work on the codec, but I suppose the development focus has been mostly on VP7. But, VP7 was not ready for testing by the submission deadline (December 19th), and it'll also no longer be free of charge once it will be available.
• XviD 1.0 was released in early 2004, and development has since started on the 1.1 branch, which is now nearing its official release. The 1.1 branch adds VHQ for B-frames, aspect ratio support and deringing in the decoder, VBV compliant two pass rate control (important for standalone playback), and also supports DivXNetworks' hardware profiles for perfect standalone playback (though those profiles don't really make full use of all the features an ASP codec can offer).
• I've re-included Microsoft's WMV9, especially since it is part of the specification of HD DVD and Blu-ray. There have been some improvements in WMV9, though it mostly concerns interlaced encoding and inclusion of WMV9 streams into transport streams for digital TV. There also have been some DVDs that include a high definition version of the feature encoded in WMV9.
• Vanguard Software's codec division has been separated from the mother company and is now known as Videosoft Inc. They offer three codecs, an AVC Simple Profile codec, an AVC Main Profile Codec, and an AVC Main Profile codec for professional use, along with an SDK.
• Moonlight software, maker of well known and widely used MPEG-2 decoder filters (and audio decoding filters), have entered the AVC business and are now offering their encoder in form of a one click AVI -> DVD / AVC tool.
• Last but not least, I got an ASP codec from a previously unknown codec maker Jomigo. Their HDX4 codec should be released shortly, in the form of a integrated conversion tool (AVI/DVD -> ASP).

As I have announced last time, I will no longer include DivX3 SBC in codec comparisons. I feel by now we have solutions that are superior to SBC, plus then there's the fact that DivX3 is a hack of Microsoft's first MPEG-4 (though not quite specs compliant) codec and thus stands on legally shaky ground.

Unfortunately, I also had to exclude Moonlight's AVC codec, even though I've been doing some tests prior to the submission deadline. One of the reasons was that their decoding filters interfered with playback of other content (even AviSynth scripts), plus they currently only write their AVC streams into TS (transport streams), which are suited for streaming. TS is not a suitable container for DVD backups though, and adds a significant overhead that would unfairly disadvantage this codec, because due to the additional overhead, the video bitrate would be much smaller than the corresponding video bitrate of every other contestant. Moonlight is working on an MP4 multiplexer for their content, so they might be included in another comparison in the future (hopefully by then also with a 2 pass rate control suited for local playback).

As in previous comparisons, codec makers were contacted two weeks prior to the start of encoding, and asked to provide a codec build to be tested, along with the suggested settings for my three movie sources. I have gotten suggested settings for every contestant, so if you think other settings would've yielded a more favorable result, you are more than welcome to take it up with the codec makers (and not with me.. I only used what I was told to). I'd like to thank all the people involved in this process, it has been a pretty intense two weeks (and I'm still in contact with most companies as the test proceeds).

The contestants

• 3ivX D4 4.5.2fc8 (not publicly released yet, will be released as 3ivX 5.0)
• DivX Fusion 5.9 (not publicly released yet)
• HDX4 1.4 (not publicly released yet)
• Microsoft WMV9 VCM
• NeroDigital AVC Main Profile (core encoder version 1.0.3.4, will be released with the next NeroVision pack). This codec is hereafter referred to as ND MP.
• NeroDigital AVC High Profile (core encoder version 1.1.0.1, far from any release at this point). This codec is hereafter referred to as ND HP.
• On2 VP6 version 6.2.7.0 (not publicly released yet, contains some improvements from the upcoming On2 encoding application).
• RealNetworks RealVideo 10 (based on Helix DNA Producer 10.1.0.390 and optimized DLL). This codec is hereafter referred to as RV10.
• Videosoft Codec Pro 2.2. This codec is hereafter referred to as VSS.
  
• XviD 1.1 (CSV build dated 12/19)

All codecs were tested in a 2 pass setup using the settings suggested by the developers.

Test setup:

All testing was done on the following hardware:

AMD Athlon 64 3500+
Shuttle FN95 mainboard (part of the Shuttle SN95F5 barebone system)
2x512MB PC3200 Kingston HyperX DDR RAM CL2
HIS Excalibur Radeon 9600 XT
Philips 230W5B2 Display connected via DVI

To encode I used the following software:

DGIndex 1.0.12 & DGDecode.dll
AviSynth 2.55 for frameserving
VirtualDub 1.5.10 for encoding (see exceptions below))
Nandub 1.0 RC2 lumafix to multiplex VBR MP3 audio
Helix DNA Producer 10.1 (build number 10.1.0.390)

To encode RV10 I used AutoRV10. To encode to NeroDigital I got a special commandline encoder from ateme, which supports AviSynth input (beta testers of the Main Profile codec will know the tool).

Movies I encoded:

Matrix Revolutions - Region1, NTSC, length: 2h09, 185917 frames (@23.976 fps). I will call this movie Matrix3 from here on
Saving Private Ryan - Region1, NTSC, length: 2h49, 243770 frames (@23.976fps). I will call this movie SPR from here on
Futurama Season 2 Episode 1 - Region2, PAL, length: 0h21, 41271 frames (@25.00 fps)

Encoding parameters:

I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-preset 128 setting) for all movies, then tried to put Matrix3 onto 1 700MB CD, SPR onto 2 700MB CDs and 4 Futurama Episodes onto one 700 MB CD (resulting in 175MB for an episode). The size of the audio track was 117'263 KB, 157'455 KB and 20'582 KB respectively. This resulted in an effective video bitrate of 621kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit). I used GordianKnot for the bitrate calculations (it can calculate the effective overhead of using VBR MP3 audio).

Since there's no VFW codec for RV10 and the ND codecs, I had to use different audio streams for those. In case of NeroDigital, I used Nero's own AAC encoder to create a 128kbit/s CBR audio track (122'481 KB, 160'441 KB and 20'489KB). Since NeroDigital uses the MP4 container, I had to perform special bitrate calculations for that codec. Ateme told me to expect a 10.4 byte overhead per frame which, assuming the audio was 128kbit/s (turns out later that the audio was a bit oversized which resulted in a slightly oversized video as well - the Matrix3 audio bitrate was 129.40 kbit/s and the SPR audio bitrate was 129.27 kbit/s), I got an effective bitrate of 627kbit/s for Matrix3, 1025kbit/s for SPR and 1000kbit/s for Futurama. In case of RV10, AutoRV10 took care of the bitrate calculations, and I just settled for a 128kbit/s AAC audio track, so I expect the RV10 bitrate to be in line with NeroDigital.

I also got different bitrates from Microsoft: 595 kbit/s for Matrix3, 1000 kbit/s for SPR and 900 kbit/s for Futurama. Obviously I asked why I shouldn't take my usual calculated bitrates and was told to use those bitrates to reach my target size. Could it be that Microsoft doesn't trust its own rate control?

As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got:

Note that 700MB equals 716'800KB, 1400MB equals 1'433'600KB and 175 MB equals 179'200 KB. Please note that I assumed that VP6 uses kbit = 1000 bit, so my bitrates for Matrix3 and SPR were too high (I had to re-encode Futurama so I could correct that mistake). The effective final size for the VP6 codec with proper bitrate should be: 728'289 KB for Matrix3 and 1'452'626 KB (but as you can see, both are still severely oversized). I've marked every size in red if it is more than 2.5 MB above target size (2.5 MB is about the oversize a CD can have and you can still burn it without having to activate overburning). The ones marked in bright yellow are undersized more than 2.5 MB (that points to a rate control issue as well but is less problematic than oversized files).

As you can see, 3ivX, DivX, HDX4, both ND codecs and XviD reach the target size. RV10 got a bit undersized, but it'll definitely fit onto CDs. VP6 got severely oversized (except for Futurama which interestingly was on target the 2nd time I encoded it - the first time it was off by more than 5 MB), and thus at that point is not very well suited for DVD backups until On2 revisits the rate control. WMV9 was undersized twice, in line with the lowered suggested bitrate, but still got oversized beyond the allowable limit once. So, for backup purposes I also have to put a question mark right there. VSS also has some serious rate control issues and delivered two oversized and one undersized file.

Codec settings (applied w.r.t. the default codec settings)

• 3ivX: B-frames on, best quality for bitrate, MPEG quantizer for SPR (H.263 for the other two).
• DivX: Home theater profile, Balanced mode with 1-bframe for the first pass, Insane Quality mode for the 2nd pass, optimized H.263 quantizers and psychovisual: Shaping.
• HDX4: Unrestricted profile, MPEG quantization, QPel, 2 consecutive B-frames , Quality: highest, Optimize for film source for Matrix3 and SPR, 1 B-frame and Quality:highest for Futurama.
• ND MP: Cabac entropy coding, B-frames and weighted prediction on, deblocking filter strength -2, max number of reference frames 2, psychovisual level 2, cartoon mode (chroma improvement) on and max 2 consecutive B-frames for Matrix3/SPR. For Futurama, the deblocking strength was increased to +2 and weighted prediction was turned off (the rest of the settings were the same).
• ND HP: 8x8 transform active, off, 2 reference frames, phsychovisual level 2, max 2 consecutive B-frames for Matrix3/SPR, and the same for Futurama, except that there weighted prediction was turned off and the deblock strength was set to 2.
• RV10: Default settings in AutoRV10 except for rate control scaling mode (set to linear) and Auto. Select Best Quantizer for the InLoop filter was turned off.
• VP6: VP62 Heightened Sharpness Profile with all the default settings. For the 2nd pass: Variability 70%, Min Section 60, Max Section 800 and Two Pass Mode: Best quality.
• VSS: I got a long config file to be used so I can't be sure which are the defaults, but here's what I found noteworthy: QPel was on, the deblock filter was used, motion estimation looks at one previous frame and Cabac was used.
• WMV9: Standard settings all around, except for the maximum keyframe distance which was set to 20 seconds
• XviD: Matrix3/SPR: MPEG quantization, Trellis quantization, QPel, VHQ for B-frames & Turbo. For Futurama: no B-frames, Cartoon mode on and Trellis on.

Encoding speed:

Here I took Matrix3 as an example. The resolution of SPR was bigger, hence the performance also dropped. Note that you should take these values with a grain of salt. Only the first 10'000 frames were used for speed measures so the overall average FPS might be a bit different. Furthermore, I did not reset the PC before each measurement and I did not perform several measurements per codec. For good absolute values, more testing would be required, but these values should nevertheless give you a good impression of the relative speed difference between codecs. Note that the values are averaged over the two passes where applicable.

I've taken the liberty to split the speed into three categories: above 2x real-time speed (green), in between 1x and 2x real-time speed (white) and below 1x real-time speed (red). Even though quality is more important than speed, some encoders really severely taxed my nerves. In the end I'll also talk about quality per FPS, and as it turns out, a high FPS is not necessarily an indicator of bad quality, and likewise, a low FPS is not necessarily an indicator of good quality.

A few noteworthy things: DivX's Insane mode during 2nd pass contributed to the low FPS. I was told to use that mode, but perhaps a faster mode could give a more balanced picture without considerable loss of quality. Likewise, for VP6, the best quality mode could be replaced by the standard mode and this would most likely not lead to a significant quality reduction. Also note that the two values in brackets for the NeroDigital codecs: In fact, the values in brackets are the actual framerates from the full movie encode (which took place during the night without any manual intervention). It turns out, ND speeds up during the first pass over time and the full firstpass speed can only be reached if the full movie is encoded (and as you can see this has quite an effect on the average framerate). I can't tell you why this is (but it's a neat trick and they reach more than an average of 100 fps during a full movie first pass) because it's obviously a well kept secret.

Other important stuff

Here's the AviSynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DGIndex in Matrix and SPR. Futurama was interlaced so I used Decomb's FieldDeinterlace with no blending to get a progressive image.

Matrix script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\dgdecode.dll")
mpeg2source("D:\THE_MATRIX_REVOLUTIONS_D1\VIDEO_TS\matrix3.d2v")
crop(8,60,704,356)
LanczosResize(640,272)

SPR script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\dgdecode.dll")
mpeg2source("D:\SAVING_PRIVATE_RYAN\VIDEO_TS\spr.d2v")
crop(2,8,714,464)
LanczosResize(640,352)

Futurama script:

LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\dgdecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\decomb.dll")
mpeg2source("D:\FUTURAMA_SEASON_2_DISC_1\VIDEO_TS\futurama.d2v")
crop(10,4,702,568)
FieldDeinterlace(blend=false)
LanczosResize(640,480)

Playback

I've used the default playback filter for every codec, using automatic postprocessing strength selection where this was possible. In XviD I turned on both deblocking and deringing, in HDX4 I turned on chroma and luma deblocking. I used a special Ateme DS filter for the NeroDigital playback (since the Nero filter only decodes Main Profile content - beta testers might also know an earlier version of this filter). I disabled the film effect where applicable. A note on this: HDX4's optimize for film mode actually filters out film grain during encoding, stores a parametric representation of the grain and applies it again during decoding. Thus, we have grain in HDX4 content. In future comparisons, I expect to see more of this as this method has been added to the AVC High Profile.

All files were reviewed using Media Player Classic as it can play back pretty much everything you can throw at it. Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player!

A note about screenshots: Unlike what you might think hearing about the go-to function in MPC, it's not easy to take screenshots. The following codecs behaved properly (insofar they they either exhibited no delay and using frame X from the AviSynth script meant I could go to frame X in the encoded video and get to see the same frame, or they'd have a constant delay with respect to the AviSynth source): DivX, VP6, XviD. Both NeroDigital codecs exhibited an constant delay as well, but still have some problems in the filter (stepping back frame by frame is not possible). ateme has promised to work on that. Now for the hall of shame: 3ivX and RealVideo were usually ±1 frame off, but since that kept changing, I had to make a screenshot, compare it with the reference frame (switch very fast between the two frames in the picture viewer), and go back to make another one if it didn't match. It got even worse for HDX4, VSS and WMV9. First HDX4: according to the creators, the problem is created by a feature they've introduced to handle corrupt MPEG-4 streams and it should be possible to turn this off in a future version (there'll be a next codec comparison where they can prove that ;). And all three codecs had problem to jump to the proper frame. If you're playing a movie encoded with them, and you use the position slider to move to another place in the movie, you either have black screen until the next I-frame (HDX4), or the video fast forwards (WMV9, VSS) to the next I-frame. When it comes to taking screenshots, that means I have to find the I-frame previous to the frame I want to make my screenshot, then advance frame by frame until I find the proper frame (and if there's no visually very distinctive mark in a screen that's not visible or at a different place in frame X-1 or frame X+1 that's really hard to do). So, let it be said that 3ivX, RV10, HDX4, WMV9 and VSS require work in the filter department and have cost me hours upon hours that are basically wasted because the filters don't implement all the seeking functionality that DirectShow forsees (you'd expect screenshots to take you perhaps 40 minutes.. but the way things are all the screenshots took me 4h15m).

And as an addendum: I had to redo a lot of screenshots due to a color conversion issue. Turns out this time the VMR9 renderer, which I used last time to make screenshots, would lead to discoloration. In order to make screenshots from NeroDigital, I had to install the DirectX SDK to get a new default renderer that exposes a snapshot button in the filter properties. But, while this new default renderer would no longer lead to discoloration, it effectively made making screenshots even more complex: The codecs that don't support frame accurate seeking would now require that I use the snapshot function of the renderer, rather than MPC's built-in one (using the MPC one would return me the previous keyframe). The same also applies to RealVideo.

Now proceed to the first test (low bandwidth JPEG version. This version loads 533KB of images for the beginning, and the total image size for the matrix test is 3.85 MB). If you have a lot of bandwidth and / or don't mind waiting, there's a high bandwidth PNG version (loads 1.98 MB of images initially and if you want to see all the images, you have to download 16.6 MB). Also note that the high bandwidth version requires that you have enough browser cache left or the images will be reloaded each time you zoom in or out.

This document was last updated on December 26, 2005