Select provider of libav* libraries

[Andreas Cadhalpun]

Hi Reinhard,

thanks for explaining your point of view here.

On 15.05.2015 09:23, Reinhard Tartler wrote:
> Thanks for this insightful post, Dmitry,
> 
> On Mon, May 11, 2015 at 5:44 AM, Dmitry Smirnov <onlyjob at debian.org <mailto:onlyjob at debian.org>> wrote:
>> On Sat, 9 May 2015 16:00:49 Andreas Cadhalpun wrote:
>>> What would you count as very compelling reasons if more features, less bugs
>>> and better security support are not sufficient?
>>
>> More features is not necessary means less maintenance burden;
>> Less bugs is not always means better software (it is a matter of how upstream
>> manages bugs);
>> Quality of security support is something that remains to be seen.
> 
> I think security is not a decisive topic where either project cannot
> clearly claim of having a clearly better handle.

I disagree: FFmpeg is clearly better at fixing security issues.
To take a random example, an out of bounds read in the bink decoder was fixed
in FFmpeg three years ago [1], while Libav git master is still vulnerable
today.

One can find more such examples, but I have better things to do with my time.

> These days, FFmpeg for
> sure asks for most (if not all) CVE numbers recently assigned, and claims
> to provide patches for them.

FFmpeg not only claims to provide patches, but actually does provide them:
most CVEs link to the corresponding patch.

> What is less visible is what happens behind the scenes:
> 
> Most of these issues originate from Google Guys that work on security and
> conduct fuzzing tests on libavcodec/libavformat. Their main target is of
> course their own libavcodec/libavformat fork that they ship in Chrome, but

As far as I'm aware they use a git snapshot of FFmpeg that they update
from time to time. They only change the build process to produce one single
libffmpegsumo library instead of libavcodec/libavformat/libavutil.

> they (of course) also share their samples with both Michael (FFmpeg) and
> Luca/Anton (Libav). Michael seems to have much more capacity and time, and
> thus is usually faster with pushing patches for such crashers.

Yes, Michael does an awesome job at fixing those [2].

> Libav takes
> the time to investigate, reproduce and understand those patches.

That's a commendable idea, but if the result is that these patches
don't get applied in a reasonable time frame, it's rather bad.

> Unfortunately, in the majority of cases, this is not trivial at all, often
> because of terse (or even wrong) commit messages, or the fact that there
> are better places to fix a particular issue in the code.

In that case it would probably help to ask the author of the patch.
Did the Libav developers do that?

> "Better" usually
> means that more than a single instance of the issue is fixed.

Can you give examples for this?

> Also, given that Libav supports significantly less codecs and formats (and
> in some cases specific variants or features of codecs), many security
> issues simply don't apply.

While it's true that some security issues only affect code not present in
Libav, the majority of issues affect both projects (until they are fixed in
FFmpeg).
Much of the additional code in FFmpeg poses nearly no security problem.
For example, even if there was a potentially security relevant bug in one
of the more than 100 additional filters in FFmpeg, it would be very hard to
exploit, since the filters generally have to be invoked manually.
Only decoders/demuxers can be easily triggered automatically,
e.g. by visiting a web site or opening a folder with images, for which
thumbnails are created.
Additionally it would be much easier to disable some rarely used features
in Debian's FFmpeg than to enable desired features in Debian's Libav,
when they are not present upstream.

> Libav could for sure do a much better job here by releasing faster. In the
> past, I looked at doing new point releases about every 2-3 months, but for
> family reasons, I wasn't able to keep that pace. Release frequency is
> clearly something that needs improvement. As far as for the release
> content, I am not aware of critical fixes being missed, so quality wise I
> think we are good.

You are wrong about the quality. Have a look at the bink example I gave above.

>> I have a feeling that Debian already became life support for libav.
>> Ever since Debian chosen libav, ffmpeg remained alive and apparently doing
>> well without our help. I'm not too sure if libav would be able to stay alive
>> without Debian.
> 
> That's an interesting take on the matter. I don't think it is accurate; for
> instance OpenEmbedded or Gentoo also ship Libav (in favor of FFmpeg), so
> Debian&Ubuntu are clearly not alone.

Interestingly Gentoo recently switched to FFmpeg by default [3] after conducting
a survey [4]. About 300 people participated in that survey and the outcome
was rather clear:
62%    [ 189 ]    "I prefer ffmpeg, and it should be the default."
 4%     [ 15 ]    "I prefer libav, and it should be the default."

> It is true that Libav owes the largest
> portion of its userbase to Debian&Ubuntu, however even if Debian would stop
> shipping Libav, then I doubt that this would threat Libav development in
> any way.

I'm not going to speculate about this.

>> From maintenance prospective libav seems to be a liability. We have to carry
>> patches for packages where upstreams are not too concerned about supporting
>> libav.
> 
> This statement is a bit surprising to me.

Have you looked at the xbmc-libav-hacks [5]?
I can understand that they dropped them when releasing Kodi.

> At the time Debian started to
> ship Libav, there was absolutely *no* difference to FFmpeg. Since that,
> many things have changed, and the two code-bases have diverged. While
> FFmpeg started to merge as much functionality as possible, Libav focused on
> maintainability, cleaned and straightened-up APIs and removed fringe codecs
> and formats.

You forgot to mention that FFmpeg also fixed many bugs in the existing code
base and most of those fixes didn't make it to Libav.

>> Also in the light of past libav transitions and deprecations that required
>> multiple changes in Debian and upstream I know no upstream who is happy to
>> support libav.
> 
> In practice, maintainability means something different for an upstream
> project and a distribution package. While upstream developers extend, fix
> and refactor the code-base,as package maintainers we make sure that
> the libavcodec package works fine with all applications that make use of it.
> 
> To be totally blunt: The libavcodec/libavformat API is horrible. This is
> partly caused because of the aim to have a single library that covers every
> thinkable container and codec flavor using a common API. Libav has
> identified the development process as the culprit, where developers
> submit unfinished and unreviewed work to the main codebase, which is then
> used by applications and can basically never be fixed.
> 
> In an attempt to provide applications a better interface to
> libavcodec/libavformat, Libav has in the past aggressively devised new
> and deprecated old APIs for maintenance reasons, i.e., helping the
> maintenance of the libav codebase.
> For Debian with a large two digit number of reverse dependencies, this
> caused a lot of fall-out that needed to be fixed. Libav developers were
> instrumental with providing patches to make them build again. FFmpeg tries
> hard to keep old APIs, which for sure lowers the effort for Linux
> distributions, but is it really better to keep applications that use
> obsolete and broken APIs? I'm not sure. In the long run, having
> applications use less horrible APIs is for sure a good thing.

FFmpeg merged those new APIs, while sometimes retaining the old APIs
longer to give applications more time to adapt.
When not many applications using the old APIs remain, they are dropped.

> I think the debate on the development methodology is the biggest
> disagreement between the two projects: FFmpeg insists on keeping its
> development velocity and allowing developers to "get work done",
> compromising on maintainability and common understanding of the code base
> in favor of more features.

I don't think getting work done is bad, because if work (like bug fixes)
doesn't get done, it is worse for maintainability.

> Libav on the other hand rather focuses on clean
> implementation and let's say better designed APIs.

Let's say Libav focuses more on cosmetics like consistent indentation,
use of spaces around brackets, a linear commit history and such.

> This requires more work,
> which in effect significantly lowers the development velocity. For a system
> integration job on the scale of Debian, less velocity seems more appealing
> to me.

Less velocity in fixing (security) issues seems everything but appealing.

> I'm having a very hard time to find an answer to the question which project
> is better for Debian. One way of looking at it is which causes less effort
> for the package maintainers. Andreas' and Dmity's mails make it sound that
> Debian could save a lot of effort by switching to FFmpeg. I'm honestly not
> sure if that is true. For instance, chromium upstream does not support
> linking against a system libavcodec library. Andreas had to provide a
> pretty invasive patch in #763632 for this, and I doubt that chromium
> upstream is interested in merging it.

The patch [6] is not invasive. The largest part is splitting a list of FFmpeg
functions used by chromium in three lists for functions from libavcodec,
libavformat and libavutil. (Chromium builds those into only one library,
libffmpegsumo.) Other than that it's about two dozen changed lines in
chromium upstream code to adapt to using three instead of one library
and to remove configuration #defines incompatible with the FFmpeg libraries
in Debian.
The reason for this is that chromium upstream basically contains support
for building against system FFmpeg libraries, it seems just rather unmaintained
and thus quite broken.

> My biggest concern with the list that Andreas provided in
> http://lists.alioth.debian.org/pipermail/pkg-multimedia-maintainers/2015-May/043980.html
> is that the different upstream require specific upstream versions of
> FFmpeg, and will break with other versions, which was pretty much the state
> 2009-2010 when I more or less took over of the package: Many upstream just
> bundled some internal copy of libavcodec that they have developed against
> and were updated with a very inconsistent pace. I'm happy to see that
> FFmpeg finally no longer discourages the use of release tarballs in favor
> for "svn trunk" or nowadays "git master" on the download page.

I'm only aware of two projects that use sufficiently outdated FFmpeg versions
that they probably won't work with current FFmpeg: Avidemux and MythTV [7].
Neither of those are in Debian, precisely due to this fact.

> Nevertheless, the release frequency is just insanely fast,

It is about four major releases per year. There's nothing insane about that.
Libav now tries to make two major releases per year, because users requested
more frequent releases [8].

> and long-term
> support is practically non-existent: the oldest release branch was cut in
> 2014 -- for Debian/Ubuntu, we require significantly longer cycles.

This is plain wrong. FFmpeg generally supports release branches that are
still widely in use. If they aren't there isn't much point in supporting them.
For example the latest release of the FFmpeg 0.5 series was 0.5.15 in November
2014, while the latest release of the Libav 0.5 series was 0.5.10 in February
2013, despite being used in Debian squeeze LTS.

> Long-term support is not the only problem - I remain unconvinced that
> switching from Libav to FFmpeg will solve the fragmentation problem. I fear
> that it would be replaced by fragmentation across different FFmpeg upstream
> versions. This may or may not be true in practice, but this is
> what both my experience and intuition on this topic tells me.

I don't believe there would be more fragmentation than there is currently.
There could even be the contrary effect: If FFmpeg libraries are available
in Debian, there is less reason to embed a copy of them.

> I'm sure that this issue could be addressed in FFmpeg, I'm just not sure
> how insistent (and helpful) FFmpeg is here, because this for sure will
> compromise development velocity, which as written above, is the main
> disagreement between FFmpeg and Libav in my eyes. Maybe Andreas could
> comment on this point?

How do you think FFmpeg could help in this regard?
After all, FFmpeg developers can't decide which version Avidemux/MythTV
uses.

>> I am not qualified for technical comparison between ffmpeg and libav.
> 
> I think nobody really is, because there is no technical metric or similar
> criteria that could help us here.

I disagree. For example, one metric would be to look at fixed issues in
FFmpeg's bug tracker (e.g. [9]) and open issues in Libav's bug tracker
(e.g. [10]) and compare.

> What are the right questions for assessing FFmpeg vs Libav? Let me try to
> formulate questions that focus on maintainability:
> 
> What project is less effort to package?
> - I'd think Libav, because of the less frequent release cycle. Both
> projects have rather accessible upstreams.

If release frequency really would be a problem, one could simply skip
some releases.

> What project is doing a better job with handling defect reports?
> - I'd think FFmpeg, because the Libav bug tracking system is currently
> dysfunctional. AFAIUI this is being worked on. I've worked around this for
> quite some time by talking to individuals on IRC directly, but this clearly
> doesn't scale.

On the other hand bugs not fixed in Libav, but in FFmpeg, can increase the
maintenance effort due to more bug reports in the Debian bug tracker.

> What project causes less effort for applications?
> - A significant number of upstream prefer FFmpeg. OTOH, FFmpeg makes great
> effort to provide source-code compatibility, but obviously a number of
> projects only develop and test against FFmpeg, so this doesn't help.

I'm sure the situation would be far worse if FFmpeg wouldn't make such an
effort to be compatible.

> What project is less effort for the security team?
> - I'd say Libav, the security patches have significantly better commit
> messages and descriptions, and generally make much more sense at least to
> me. Maybe Moritz can elaborate on this.

It seems he already did [11]:
"I think ffmpeg is doing better in terms of handling security issues; when
I contacted Michael Niedermeyer in private we has always quick to reply,
while libav-security@ seems understaffed: Several queries in the past needed
additional poking, some were left unaddressed until today. Also, the Google 
fuzzer guys stated that more samples are unfixed in libav compared to ffmpeg."

> My personal impression is that In FFmpeg, security support is handled by a
> single person whose work is not exactly easy to follow.

Michael is not the only one who cares about FFmpeg's security. For example,
I do as well and I recently fixed a number of issues that I found by fuzzing
myself. Not all of them are fixed in Libav yet, even though I sent my patches
also to libav-devel at libav.org.

> For me personally, this looks like a tie.

For me it looks like a clear choice.

> I would like to think that a
> high-profile, high-performance and universal multimedia codec and format
> library deserves a clean code-base, clean documentation, excellent test
> coverage and a steady and thorough release process. Neither Libav nor
> FFmpeg achieve that fully. Libav had so much potential, but struggles with
> achieving it for IMO mostly manpower reasons - working in or with FFmpeg
> for some reason seems to be more attractive.

My main reason for preferring to work on FFmpeg is that it already works better.

> Clearly, Libav has lost a lot of its drive when it took off. This drive has
> significantly diminished - significant contributors from its inception are
> no longer active. I remain convinced that for the Jessie release, Libav was
> the better choice.

Unfortunately FFmpeg was stuck in the NEW queue until after the transition freeze,
so for Jessie the choice was between Libav and Libav+FFmpeg, which the security
team didn't want.

> For Stretch, it is less clear. Libav seems to me like
> the prudent choice, FFmpeg like the more aggressive one.

For stretch I think FFmpeg would be the sensible choice, while staying with
Libav would just be the 'status quo' choice.

> I'd rather
> encourage people to help with porting missing features and bugs to Libav,
> because it is the cleaner and more maintainable codebase, but given that we
> are all volunteers, this may or may not happen.

I don't think Libav has a much cleaner or more maintainable codebase, it's
just smaller. Fixing issues in Libav that are already fixed in FFmpeg is
just duplicated work, which is not very rewarding to do.

> Wow, that was a much longer email than I hoped. And given that I'm still
> traveling and doing many family visits, it also took me much longer to
> write this than I wished. If we really had to vote right now which way to
> go, I'd probably abstain, because I also have to face the fact that I no
> longer have the capacity to support a high-profile package like libavcodec
> with the same time and devotion as I did for the last 5 years. How ever we
> as a team decide to move forward, I think libavcodec (and related
> libraries) are better team maintained (pkg-multimedia is just fine for
> this), and I'd be happy to help out with the packaging either way.

I'm not opposed to pkg-multimedia team maintenance for FFmpeg.
If anyone wants to help maintaining it, feel free to add yourself as uploader.

Best regards,
Andreas


1: https://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=b3675f890abee0bc446495711223a5c790234672
2: http://j00ru.vexillium.org/?p=2211
3: http://thread.gmane.org/gmane.linux.gentoo.devel/95339/focus=95585
4: https://forums.gentoo.org/viewtopic-t-1010096.html
5: https://sources.debian.net/src/xbmc/2:13.2%2Bdfsg1-4/lib/xbmc-libav-hacks
6: https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=5;filename=chromium-browser_system-ffmpeg.patch;att=1;bug=763632
7: https://trac.ffmpeg.org/wiki/Downstreams
8: https://sources.debian.net/src/libav/6:11.3-3/doc/RELEASE_NOTES/?hl=9#L9
9: https://trac.ffmpeg.org/ticket/4431
10: https://bugzilla.libav.org/show_bug.cgi?id=840
11: https://lists.debian.org/debian-devel/2014/08/msg00060.html