MKV Mania: Header Compression And The Side Effects

by Damian on August 9, 2010 · 31 comments

in Guides

I generally like to update to the latest versions of software as they come out (whether beta or “stable”). There is always the chance that the new update may break something that was once working, but for the most part to date I have had very few issues.

A few weeks ago I updated to the latest mkvtoolnix, which I believe is currently at v4.2.0. I unfortunately noticed a bad side effect of this, as mkvs created with the latest version were not playing properly on some of my hardware. It turns out that starting with v4.1.0 Mosu, the developer of mkvtoolnix, implemented “header removal compression”. Per Mosu’s faq, this allows a muxer to keep a certain number of bytes that are identical for each frame in the track headers removing them from the individual frames. This reduces the size of the tracks significantly without altering the content as a demuxer can add the bytes found in the track headers to each frame during demuxing. Header compression will apply to a variety of tracks suck as AC3, DTS and MP3 audio tracks as well as Dirac and MPEG-4 part 2 (aka. XviD/DivX) video tracks. Starting with v4.1.0 header compression is enabled by default for these tracks. If you look at the screenshot below the compression settings are under the “Extra Options” tab

So what exactly is the problem? Well, it turns out that several hardware/software players don’t support header compression in mkvs. For example I tested out an mkv with header compression on my PCH C-200 and I got audio but no video. To add fuel to the fire, header compression is set by default as enabled with v4.1.0 or greater, so if you have hardware/software that does not support this feature you must manually set Compression to “none” every single time you use mkvmerge (in my opinion this feature should just be set to disabled by default, or at a minimum allow the user to set it to disabled once and be done with it). Unfortunately the benefit from header compression is minimal at best. I created two identical mkvs, one with header compression and one without. The mkv without header compression weighed in at 20,908,403 KB whereas the mkv with header compression weighed in at 20,906,165 KB, a mere 2,000 KB in savings.

If you are affected by header compression what can you do? Well, there are a few options:

  • Use mkvtoolnix v4.1.0 or greater but press the developers of any hardware/software players having issues to add support for this
  • Revert back to an older version of mkvtoolnix (pre v4.1.0)
  • Set compression to “none” each time you use mkvtoolnix v4.1.0 or greater

I have confirmed that the latest MPC HC beta, Dune players, and PCH C-200 (using the just released RC2 fw) support header compression, and hopefully we will start seeing universal support soon.





Article by

Hi, my name is Damian, and I'm tech gadget addict! Although I always had some interest in technology, it wasn't until I got my EX470 and more importantly found Mediasmartserver.net, that my interest became an addiction. My goal, aside from world domination and to see the Mets/Broncos win another championship, is to set up the perfect digital home where all my media is available at the click of a button. When I am not writing for Mediasmartserver.net you can find me over at my blog at http://www.adigitalhomeblog.com or follow me on twitter


{ 31 comments }

Brajesh August 9, 2010 at 1:44 pm

Thanks Damian, this is good to be aware of. Thankfully, with the C-200 and Dune (which I’m mulling also getting to try out) supporting header compression w/the latest firmwares, I can continue to use the latest MKVMerge w/o worry :).

Damian August 9, 2010 at 1:51 pm

Plus the latest mkvmerge is what you need to get PGS subtitle support (I think possibly somewhere between 4.0 and 4.1 PGS support was added so if you can find the version in between that doesn’t have compression you are set!!!)

Jason August 10, 2010 at 9:00 am

The guy who authors MKVMerge clearly did this as a stunt to get attention. There’s no other reason to do it other than flaunting that it’s in the spec.

Damian August 10, 2010 at 9:09 am

Making the option available but simply leaving as disabled by default would have been the easy solution. Unfortunately a lot of people use mkvmerge through other programs (such as Ripbot, Another EAC3To GUI, etc…) and have no clue about these settings, now wondering why their mkvs won’t play.

Atamido August 12, 2010 at 9:45 pm

The option has been available and disabled in mkvmerge by default for years. Header Removal has been in the specifications for over 5 years. Mosu (the author of mkvmerge) always adds new features as disabled initially to allow others time to add support. He has also found that hardware makers never add support for features that aren’t enabled by default. So having let the feature sit there implemented and available to be used (and in the specification) is now enabling it by default.

This is all perfectly normal. The issue here is the hardware makers.

(In truth, it is often possible to remove more of the header to gain more space savings, but this requires two passes, which mkvmerge doesn’t do.)

Jonason August 23, 2010 at 6:22 pm

No, this issue is the programmer which is an ass – nobody cares about this shitty “feature” which is of no use at all. There was no reason to add it other than him being a jerk.

Atamido August 23, 2010 at 8:22 pm

Did you even read the comments? That “feature” has already been over there for OVER 5 YEARS. It is extremely useful for very low bitrate streaming, so it is critical that hardware supports it. I really don’t know where you get off being such a prick.

Jason August 24, 2010 at 8:03 am

No, it was a stunt to get attention. An esoteric feature with very limited practical use gets set as DEFAULT with no simple options setting to to make it disabled most of the time?

Come on. This was intentional and was done to draw attention to the ‘issue’ by breaking compatibility with dozens of existing media players.

That’s a stunt done to get attention as I see it.

Atamido August 24, 2010 at 8:24 am

The issue wasn’t discovered until after it was enabled by default. As mentioned before, features that improve files by default are added disabled by default, and changed to enabled by default after several years to give time for demuxers to be updated with support. There are multiple open source libraries which are quickly updated with support (which incidentally are what hardware players use).

Remove the tinfoil hat, there was no conspiracy.

Jason August 24, 2010 at 8:32 am

If everything is so innocent then why not include a simple radio checkbox or option selection to make it disabled by default if that is the users preference? Why jam it down the user’s throat?

There are plenty of media players out there that do not get regular firmware updates or for which support has ceased. Those users are stuck re-encoding a huge chunk of their files if they get caught out by this ‘friendly change’.

The good news is that we aren’t held hostage by this crap, we can always go use MakeMKV instead.

Atamido August 24, 2010 at 9:00 am

The programmer feels the GUI is already too complex and resists adding anything else to it. Although, I honestly can’t see much reason to. If you are already using the GUI and you know you want to change a feature, it’s easy to change. If you’re batch processing using the command-line, then it’s easy to set whatever feature from the command-line.

Also, this has nothing to do with re-encoding, it is only re-muxing. If you accidentally created some files with a feature your player can’t support, it’s trivial to batch process files using the command-line, with the limiting factor being the speed of your hard drive.

Jason August 24, 2010 at 9:09 am

:slaps forehead in dismay:

You really don’t get it, do you? The entire point of the GUI is so that people don’t have to learn or use the command line options in the first place!!

The GUI defaults to some poorly implemented and poorly supported header encryption that saves less than 1% of filesize so we should all just start to use the command line options to ‘easily avoid’ this problem? Re-mux our last 10 or 20 encodes (or more) with yet more command line BS? Tell me how I’m going to batch out my last 20 encodes when I have 300 other ones in the same source directory? I write Perl scripts and I still find this a giant PITA and don’t even want to think about dealing with it.

Again, this is an attention seeking stunt, and as I said, the author has clearly showed he cares little for his users, so those users should absolutely be looking into alternatives.

The fact that a small but vocal minority of people have jumped to the authors defense is simply sad. Maybe the author should give more attention to what his users want instead of engaging in an ego trip? It’s like an author with a series of best selling books who decides to write his next one in 20 point Impact font and then rails against users who are upset about it… telling them that Impact is a fully supported font and they can suck eggs if they don’t like it.

Say it with me now… EGO TRIP.

Atamido August 24, 2010 at 10:10 am

You just don’t get it do you? The entire point of the GUI is it’s simple to change the settings. Of course, if someone wanted to see what the command line is that MMG uses, you can set things up and then select the option to view the command-line. It’s that easy.

Of course, MKVmerge/MMG doesn’t support encryption, but I’ll assume you’re just throwing around words now.

If you want to just process new files, then the easiest thing to do would be to create a folder, sort the files by date, and move the new ones to the folder. Then use a simple DOS command line such as FOR to process them all. This isn’t exactly rocket science buddy.

The fact that a small but vocal minority of people have jumped to the hardware maker’s defense is simply sad. It’s like people getting upset at Microsoft when Windows 7 doesn’t install on 5 year old hardware because the hardware maker refuses to update the BIOS.

Say it with me now… DELUSIONAL.

Jason August 24, 2010 at 10:21 am

Funny (and inaccurate) comparison considering that many of the media players now broken by this are less than six months old.

Atamido August 24, 2010 at 10:29 am

Which is all the more interesting as it would be like hardware makers coming out with hardware 6 months before Vista came out that died in Vista because IPv6 is enabled by default. It’s not like IPv6 came out of nowhere.

Jason August 25, 2010 at 10:12 pm

Fortunately MS isn’t stupid enough to make IPV6 the default protocol in their OS.

The proper analogy would be that MS made IPV6 the default on boot up, and you had to manually go in each time and turn IPV4 on.

And then MS argument is that because IPV6 has been a standard for over 10 years they’ve decided it needs to be turned on by default, whether or not other vendors or ISPs are ready for it.

Atamido August 26, 2010 at 6:31 am

Ah, but they did enable it by default, which caused issues for a small number of people. Sounds pretty similar.

Or perhaps you’d prefer UAC which they enabled by default in Vista and affected a much larger number of people? It’s not like UAC was a surprise either. It was in the builds of Longhorn for years, and programmers should have been testing and updating their software to make sure it worked.

XEQ October 28, 2010 at 6:00 am

When you write a highly popular utility, and feels the need to be a bitch at imposing a long forgotten specification, especially if your utility has never implemented it before, it is considered polite to add that feature in, BUT turned off by default. THEN, you give notice that the next version will have this feature turned on by default if you want to be a real bitch.

Any new feature addition will be prone to bugs. Even God introduces bugs when creating man… such as programmers that that wants to behave like a prima donna.

I have stopped updating since 3.4. I’ve seen this change in behavior and suspected something amiss.

The “H Media Splitter” originator is a potential prima donna in the making.

This is why we need to support Open Sourced alternatives. Boy, I really miss Gabest.

SVN August 11, 2010 at 1:31 am
martmeister August 12, 2010 at 9:07 pm

This isn’t supported by the WDTV Live which is a pain. I am assuming it’s the same for the WDTV and WDTV Live Plus.

Jonason August 23, 2010 at 6:23 pm

WD has said they plan to fix it (eventually)

Meins321 October 12, 2010 at 2:25 pm

No honestly it is just an annoyance!
Why default at all? just to get attention is an good argument thank you!
i haven’t thought of that but it might be like mnow mkv is becoming more and more common…

we will see what happens, but most times the reference frames are shittier

i don’t get how people encode their videos with 5.1 and 16 reference frames ^^

Brajesh November 24, 2010 at 9:05 am

Just to be clear, do I just need to select ‘None’ for header compression for audio only (as shown in Damian’s screenshot above)? I’ve been setting it to ‘None’ for video and subs as well. Hope doing so wasn’t a bad thing. In any case, still want to make sure I should be setting it to ‘None’ for audio only for maximum compatibility with media players.

Damian November 24, 2010 at 9:06 am

Yes, set to none for video as well (I haven’t been doing for subs but cannot hurt)

Brajesh November 24, 2010 at 10:07 am

Thanks. Maybe I should just get v4.0.0 from here: http://www.videohelp.com/tools/MKVtoolnix/old-versions#download

What would I lose by going back to v4.0.0?

Damian November 24, 2010 at 10:13 am

I think possibly PGS subtitle support. Also, the very latest mkvmerge looks to have a header fix if you use Haali

fred July 11, 2011 at 8:21 am

Enabling the option has virtually no effect other than to make files unplayable as many of us have found. I have had to remove the header compression from many MKVs that are now being produced by tv\film release groups and in EVERY case they difference in filesize as a result of the ‘marvelous’ file header-compression technology resulted in a decrease of less than 1mb!

So why enable a feature that clearly has no real-world benefit? Well as others have suspected, it is either the author of the GUI etc having a hissy fit and enabling it because ‘they want to’ or that they just screwed up the release of the software and decided to not bother reversing the state of the option back to ‘none’.

I found that the compression option has started to be enabled so frequently now that I took to creating some right-click context-menu command entries for MKVs such as ‘MediaInfo’ (which opens a file in SourceForge’s MediaInfo tool’, ‘MKToolnix: Open with MKMerge GUI’, ‘MKToolnix: Remove Track Compression’ and ‘MKToolnix: Show MKV Information’ all so that I could quickly see if a file had it’s header compressed etc, remove header compression with one command and perform some more advanced options with the MKMerge GUI if things needed tweaking (.reg text below – if you use it you might have to change the paths to your executables etc. – taken from by x64 system with default MKVToonNix paths)

COPY AND PASTE INTO A TEXT FILE SAVE AS A .REG FILE AND RUN;

Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.avi]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.avi\Shell]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.avi\Shell\MKToolnix: Open with MKMerge GUI]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.avi\Shell\MKToolnix: Open with MKMerge GUI\Command]
@=”C:\\Program Files (x86)\\MKVtoolnix\\mmg.exe %1″
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Open with MKMerge GUI]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Open with MKMerge GUI\Command]
@=”C:\\Program Files (x86)\\MKVtoolnix\\mmg.exe %1″
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Remove Track Compression]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Remove Track Compression\Command]
@=”cmd.exe /k \”\”C:\\Program Files (x86)\\MKVtoolnix\\mkvmerge.exe\” -o \”%1_NoComp.mkv\” –engage keep_bitstream_ar_info -A -S –compression -1:none \”%1\” -D -S –compression -1:none \”%1\” -A -D –compression -1:none \”%1\”\”"
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Show MKV Information]
[HKEY_CURRENT_USER\Software\Classes\SystemFileAssociations\.mkv\Shell\MKToolnix: Show MKV Information\Command]
@=”C:\\Program Files (x86)\\MKVtoolnix\\mkvinfo.exe -g %1″

swamibob September 15, 2011 at 9:06 pm

Header stripping makes no sense at all. I just saw in a 6.55GB MKV file a difference of 100KB. Who cares if you save say 300KB in a 20GB mkv file. These files are so big they are not going to make or break you with a 300KB savings. My player will play it right either way, but why do this if it does screw it up for some people. It makes no sense to do it of to have it enabled by default in MKVmerge.
Swami

hellz October 3, 2011 at 11:52 pm

I’m using now mmg 5.0.0 and in Files, Options, mmg tab you can check an option to set compression to none by default for audio/video, I just see it, hope it will work.

Rob July 9, 2012 at 10:04 pm

how does asus o!play hd2 player , react to the bz2 compression of mkv , just wondering would it save GB in size or not for say a 1080 mkv file?
not sure which compression would be best say for video or audio? and how does it affect the quality of playback using mkvmerge v5.6.0 39 mins. to create the mkv file, seems a bit long, but only because I am seeing what it does the the final file size, since this media player almost plays anything, if its bugy of course I am not going to use it option, also it take at least 1 hour to do one file, also which is slow compaired to 7 zip compression , which as far as I know it can’t play compressed zips yet using a hardware video player

Saeed December 16, 2012 at 12:59 am
Comments are closed, visit the forums to continue the discussion.

Previous post:

Next post: