[buildd-tools-devel] re buildd's resolver and package's build deps

Wouter Verhelst wouter at debian.org
Tue Mar 1 21:52:39 UTC 2011


On Mon, Feb 28, 2011 at 07:12:00PM +0000, Roger Leigh wrote:
> On Mon, Feb 28, 2011 at 03:36:47PM +0100, Wouter Verhelst wrote:
> > On Tue, Feb 22, 2011 at 05:08:18PM +0000, Roger Leigh wrote:
> > > This is correct.  I was thinking about drafting a patch for Policy
> > > about this.  Current Policy defines the allowed syntax for
> > > Build-Depends.  It does not however, make any mention of existing
> > > conventions and best practices, which I feel should be addressed.
> > > 
> > > The current "internal" build dependency resolver does not make
> > > any use of the alternative dependencies.  It always picks the first.
> > 
> > It was my understanding that there are some corner cases in which it
> > would select the second:
> > 
> > - where one build-dependency (indirectly) depends on a secondly-listed
> >   alternative, it could install the second.
> > - sbuild will not worry about installing the first alternative if the
> >   second has already been installed.
> > 
> > but I might be mistaken about one or both of the above.
> 
> I'm not totally sure either, to be honest.  The internal resolver code
> is horrible.

Yeah, I know -- I've written a patch for experimental support once, and
it took me several days just to figure out where to add two characters,
or some such...

> It looks like it still has relics of manual source-deps entangled in
> there as well (but I don't dare to touch it since it will probably
> break horribly if I delete it).  The alternatives processing is in
> Sbuild::InternalResolver::parse_one_srcdep and filter_dependencies,
> and it looks like it could well be OK with second alternatives if
> installed.

Right, so I probably wasn't dreaming that up, then.

Not that it matters all that much.

> > At any rate, if a maintainer would wish to support backports easily
> > (ever more relevant now that it's on debian.org systems), having
> > alternative build-dependencies would make perfect sense; in those cases,
> > sbuild would need to pick one set of dependencies for unstable, and the
> > second set for backports.
> 
> Agreed.  Note that we now support strict 'first-only' alternatives
> handling with the 'apt' and 'aptitude' resolvers.  See the notes for
> 0.60.0 and 0.60.1 pertaining to resolvers here:
> 
> http://git.debian.org/?p=buildd-tools/sbuild.git;a=blob;f=NEWS;h=f4576f0e3b3c0fcd660000abcd93898697246b9a;hb=HEAD
> 
> We have made the alternative resolving configurable, so that you can
> use strict 'first only' or all alternatives.  It defaults to
> 'first only', which will give resolution almost entirely like the
> internal resolver.

Good.

> > True, but the right solution for this problem is not to remove support
> > for alternative build-dependencies; instead, policy should make it clear
> > that when a package is built in unstable, no differences in alternatives
> > should cause a change in functionality or support for file formats.
> 
> Please see #614807: debian-policy: Please document autobuilder-imposed
> build-dependency alternative restrictions, which is the current
> proposal for documenting the existing behaviour (and it does preserve
> the ability for maintainers to use alternatives).  Comments would be
> appreciated.

I'll have a look.

> > > I would recommend keeping backports on a separate VCS branch rather than
> > > having a single unified package.
> > 
> > I vehemently disagree with this recommendation. It works now, properly;
> > what you suggest requires merging branches every time you wish to update
> > the package. That introduces an extra maintenance burden for, IMHO, no
> > good reason.
> 
> I've been doing this on e.g. a 'lenny-backports' branch, and it's been
> a model of simplicity.  I just 'git merge <release-tag>' and it's done
> except for updating the changelog (and fixing up the rare conflict).
> 
> http://git.debian.org/?p=buildd-tools/schroot.git;a=shortlog;h=refs/heads/schroot-lenny-backport
> 
> Certainly more reliable and convenient than doing things by hand.

Sure. I'm not saying you shouldn't work on a separate branch if that
works for you. But the mere fact that it works for you does not have to
mean that it will work for everyone, and enforcing that for no
particularly good reason does not sound like a good option to me.

(anyway, that's moot, given your solution)

[...]
> > > The need for concrete|virtual is a fundamental deficiency in our
> > > package management that's been unaddressed for years (see old -devel
> > > discussions).
> > 
> > Pointer?
> 
> The one I could find:
> http://lists.debian.org/debian-devel/2006/08/msg01281.html
> 
> An ancient proposed solution (probably suboptimal):
> http://people.debian.org/~rleigh/virtual-policy_1.dsc
> 
> AJ did say back then (though I can't find the mail, sorry), that he was
> working on a solution, but I don't know what happened to that.
> 
> The main issue here is that every package with a "concrete|virtual"
> alternative dep is specifying its own idea of what the ideal
> concrete package should be.  That is, each package is dictating what
> should be a system-wide policy, and this leads to a lack of uniformity
> because there's not necessarily any coordination, and different
> maintainers have different preferences.
[...]

got it.

> > [...]
> > > Java and DocBook are the two worst offenders WRT build-deps.
> > 
> > This statement makes no sense at all.
> > 
> > Both are a clear case of software where there's a spec of what output a
> > given input should provide. As such, whatever particular implementation
> > you install should not matter; the output will be the same.
> 
> It /should/ be the same.  In practice, does this always happen for
> sure?

I've never seen serious issues with java packages in that regard, don't
know docbook well enough to be able to tell.

At any rate, I'd be surprised if the difference between 'output from jvm
a' and 'output from jvm b' would be significantly larger than the
difference between the output of two versions of GCC, for example.

Modulo any bugs, of course, but those happen in any toolchain; this is
no different.

> The main issue with these two is the inconsistent use of alternatives,
> not just in the build-depends, but in the indirect dependencies, which
> leads to an unpredictable outcome which can vary depending upon
> dependency ordering.  This is directly related to the lack of
> system-wide virtual defaults as discussed above, leading to each
> individial package implementing their own, conflicting, defaults.

What's important is not which set of packages gets intalled upon trying
to build a random java-based package, but which set of packages gets
installed upon trying to build one particular java-based package. If
that is predictable based on the control file of that java-based
package, then the result is reproducible, and any bugs can be traced.

> > [...]
> > > One thing we could do is get sbuild to strip all non-arch-specific
> > > alternatives from the Build-Dependsi, so it would never use
> > > alternatives, ever.  This is for the "apt" and "aptitude" resolvers,
> > > not "internal".
> > 
> > This could make sense.
> > 
> > While the above arguments all make sense wrt building locally, there is
> > no reason to modify the current behaviour whereby sbuild will only look
> > at the first dependency (unless something else has already been
> > installed). After all, I do agree that predictability on buildd hosts is
> > a good thing...
> 
> As I touched on above, this is the way we have gone.  It's configurable,
> so it can be turned off for e.g. experimental, backports, where the
> additional dependencies have some value.  Because this is actually
> properly enforced, unlike for internal (we actually strip out the
> unwanted deps), it's actually slightly stricter than internal (though
> in practice in the current archive the first dependency is always
> satisfiable, so this won't have any impact at present).

Actually, I believe that having it slightly stricter than internal is a
good thing. Unclean chroots are a problem on buildd right now; with
this, they won't be as much anymore.

-- 
The biometric identification system at the gates of the CIA headquarters
works because there's a guard with a large gun making sure no one is
trying to fool the system.
  http://www.schneier.com/blog/archives/2009/01/biometrics.html



More information about the Buildd-tools-devel mailing list