[buildd-tools-devel] Bug#610689: Bug#610689: sbuild: cross build support

[Roger Leigh]

On Fri, Mar 25, 2011 at 05:06:02PM +0000, Hector Oron wrote:
> 2011/3/10 Roger Leigh <rleigh at codelibre.net>:
> 
> > This part can be done by the xapt resolver directly, without any
> > special user setup required.
> 
> True.
> 
> > For the cross-packages, you can add these to the core build
> > dependencies list, so they will be installed along with build-essential
> > etc.  Or (better) you can add a separate XAPT_DEPENDS variable which can
> > be appended to CORE_DEPENDS in Build.pm prior to installing the core
> > dependencies.  Does xapt not bring in these packages directly, or do
> > they always need installing by hand?
> 
> xapt downloads the foreign architecture packages using APT with -o
> APT::Architecture=$HOST_ARCH into a directory controlled by xapt
> (/var/lib/xapt/, irrc), then throws dpkg-cross into that directory to
> convert those packages and installs them.

This is the part that I think can be significantly improved.

sbuild creates a local APT archive to install dependency packages from.
We can use the same mechanism to allow the dpkg-cross packages to be
installed.  It could even be used to cache generated packages between
builds.  We can quite easily refactor the code to allow multiple
archives (if required).

> > Well, when we install the build dependencies the package isn't yet
> > unpacked.  The unpacking is done in the build() function.  But there's
> > no reason we can't reorder it if we need to.  The unpacking could
> > be split out of build() into a separate unpack_source() function,
> > which could be run earlier if xapt/embuilddeps is in use.
> >
> > What information does embuilddeps require from the unpacked source
> > package?  Is this information also available in the .dsc? (We do
> > have the .dsc information directly to hand).
> 
> It is possible to use .dsc information, but embuilddeps does not know
> how (yet). :-)

OK.  For the purpose of actually implementing this, I would like to
have a description of exactly which fields we need for cross building
from which files in the source or DSC, and what they do and why.  Is
this all documented somewhere?

> > xapt is doing two jobs it seems:
> > 1) from the package list, make a list containing all the cross packages
> >   needed
> 
> That is given with simulation option (-n).

This is with embuilddeps, not xapt?

This option appears to give a package list, prefixed with
"/usr/bin/apt-get -y".  This just isn't suitable.
- it's omitted packages which were already installed; this is not
  necessary and involves poking around under /var/lib/dpkg.
- we need to strip off the "/usr/bin/apt-get -y" which might be subject
  to change and then cause breakage, at some point in the future.
All we want is the /complete/ package list, without any removal of
packages or addition of other bits.

> > 2) install them (but it doesn't list them in the list of packages to
> >   install, which makes removal harder potentially)
> >
> > From the resolver POV, what would be really good if it could just do
> > (1).  This way, we can just give the expanded list back to the
> > resolver for it to install.  Because we need to clean up after
> > ourselves, the apt and aptitude resolvers build a dummy
> > "dependency package" from the dependency list, and then get apt/
> > aptitude to install it.
> 
> This is how old pbuilder used to do it, but it does not work with
> cross packages as adding dependencies on
> $foreign_package-$HOST_ARCH-cross won't work, as those packages are
> output from dpkg-cross and not available from repositories.

Right.  But with sbuild, as I mentioned above, we can install the
dpkg-cross-generated packages into a local archive, which *is*
accessible to apt-get/aptitude, and hence you will then be able to
use cross packages in the dependencies /directly/.  This will permit
proper cleanup after a build using the normal mechanisms sbuild uses
for all builds.

This is to say the resolver can do the following:
- get the cross dependencies from the dsc or unpacked source
- download the debs
- run dpkg-cross
- install the dpkg-cross-generated debs into the local archive
- add the cross-dependencies to the package build dependencies
...
- run dpkg-buildpackage with the necessary options to cross build
...

We already allow various things to add to the package build deps,
and so adding cross-deps will be trivial.  All we really need is the
necessary information for what to parse from the dsc or source package,
and then how to use dpkg-cross.

Just to be clear: none of the above uses xapt/embuilddeps, because
sbuild can, I think, do a better job with its local archive and which
will integrate much better with our existing cleanup code when it comes
to removing build deps.

> > We then reverse the process to clean up
> > after the build.  So ideally, we don't really want xapt to do any
> > installation: we just want it to give us the package list.  If
> > this is possible, it might not even be require to have a separate
> > XaptResolver--we just expand the cross-dependencies and give them
> > to the regular resolver.
> >
> > If xapt can't do this directly, how complex is the code that
> > implements (1)?  Would it be possible to split it out as a separate
> > tool?
> 
> embuilddeps -a armel -d /path/to/source -n will give you that
> information, and /path/to/dsc might be in the way.
> We really need to think more about how we are going to do it to be
> able to fit multiarch in the cooking.

As I mentioned above, I really don't like this.  It's not giving us the
unfiltered information we need.  And it's telling us nothing about what
xapt is actually going to install (in terms of letting us reverse the
procedure).  And, it's also in a format that requires parsing to strip
out all the unwanted bits in order to give us what we want.

embuilddeps is giving us command lines to do the job.  What we need is
the information to do that job ourselves.

> > I've rebased your patches against current sbuild.git master, and
> > they may be found here: git://git.debian.org/users/rleigh/sbuild.git
> 
> I tried to work on top of that, but it did not build for me.

Details?

> I'll be looking into how to fit pieces together for cross and
> multiarch support. Let you know once I get something.

What I'll do next is refactor the needed bits so that we can unpack the
source tree and get at the cross build deps.  Right now this isn't
possible because we get that from the dsc.

For getting things merged onto the master branch, having discrete
changes that don't break non-cross building will make it easier to merge;
rebasing it against git master will also make it easier to review.


Thanks,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20110326/4e340f3f/attachment.pgp>