[Reproducible-builds] Cross-building across architectures (reproducibly?)

[Helmut Grohne]

Hi Steven,

On Sat, Jan 02, 2016 at 09:03:21PM +0000, Steven Chamberlain wrote:
> I've begun to analyze this now, using diffoscope to compare a stage1
> debootstrapped chroot of linux-i386, from official Debian packages vs.
> my cross-built ones from kfreebsd-amd64.

Note that there are public jenkins jobs for a very similar task already.
On https://jenkins.debian.net/view/rebootstrap/ the "*_diffoscope" jobs
do compare the cross build packages to the archive versions. The
resulting text diffs are embedded into the build logs, so apart from the
missing cross kernel part (because kfreebsd-any doesn't cover release
architectures atm and jenkins.d.n doesn't have any kfreebsd-any slaves),
we do get these diffs for almost a year already.

> Unfortunately all the ELF binaries have differences, but most other
> files' contents are reproducible.

There are lots of reasons for these differences. The top reasons most
likely are:
 * Different gcc version
 * Different build path (and thus different build ids)
 * Different configure check results due to cross compilation

> The biggest problem was that the cross toolchain's linker used a
> different target ABI:
> 
> ???   ???   ??? -    OS: Linux, ABI: 3.2.0
> ???   ???   ??? +    OS: Linux, ABI: 2.6.32
> 
> and that leads to huge differences in the output binaries, such as not
> using .init_array/.fini_array sections.

This is surprising as glibc explicitly requests 3.2 via --enable-kernel.
I also cannot confirm this in the (linux-only) diffoscope logs mentioned
above. There all the cross built packages use the 3.2 ABI. Maybe some
configure check went wrong here?

In general, wrong configure results are a big problem to cross
compilation and are the main reason why I added support for running
diffoscope early. Some wrong checks simply make the build fail, but many
go unnoticed until you diff the binary. The root cause is that autotools
tends to be pessimistic rather than optimistic. When you cross build and
a particular check cannot be run, autotools tends to assume the worst
possible outcome. To solve this, we need an organized way to preseed
these check results to configure scripts. Currently, some check results
are bundled into dpkg-cross (not in jessie or stretch), but ideally
these would be distributed to the packages that are being checked (e.g.
glibc when you test whether malloc(0) returns NULL).

While my focus has been on making cross building work at all, I welcome
and support efforts to make it reproducible as well. You can find
interesting pieces in the rebootstrap_*_diffoscope jobs today for
packages like libtool, nettle-dev, bash, libfreetype6-dev,
libexpat1-dev, libicu-dev, libxml2-dev and libdebian-installer4-dev.

Please Cc me in your replies.

Helmut