Bug#871806: uscan: (dpkg, git-buildpackage) accept/mangle/store signed git tags in cases where upstream does not publish detached sigs on tarballs

[Osamu Aoki]

Hi,

I was just talking about similar things on
debian-policy at lists.debian.org and Bug#811565.

Also git tag and git HEAD packaging support is currently in progress.

On Fri, Aug 11, 2017 at 02:15:28PM -0400, Daniel Kahn Gillmor wrote:
> Package: devscripts
> Priority: wishlist
> Control: affects -1 + dpkg git-buildpackage pristine-tar
> X-Debbugs-Cc: dpkg at packages.debian.org, git-buildpackage at packages.debian.org, pristine-tar at packages.debian.org
> 
> Hi devscripts, dpkg, git-buildpackage, pristine-tar folks--
> 
> It's awesome to see the progress made on tracking upstream cryptographic
> signatures via uscan and dpkg in debian.  This provides a dataset for
> cryptographic provenance that can be useful for auditing.
> 
> We're handling detached OpenPGP signatures for tarballs at the moment,
> but not all upstreams provide this particular form of cryptographic
> assurance.
> 
> Some upstreams do provide cryptographic signatures, but only in git
> tags.
> 
> I'm not sure exactly how to do this, but what i'd like to see is a way
> for us to record and make use of signed git tags in the same way.
> 
> I'm opening this bug report in the hopes of starting discussion about
> how to best do it.
> 
> The text below assumes the following:
> 
>  * upstream maintains their sources in a git repository.
> 
>  * upstream either doesn't produce tarballs at all, or they produce
>    tarballs from their git repository using something like "git
>    archive".  For the sake of the argument here, let's assume that they
>    don't produce tarballs at all.
> 
>  * upstream does indicate releases in the form of OpenPGP-signed git
>    tags.
> 
>  * The releases offered by upstream correspond to the "upstream
>    versions" that are uploaded into debian.
> 
> 
> 
> Here's an extremely rough and inefficient approach (which i haven't
> implemented, as this is in brainstorming phase).  I've probably even got
> some of the terminology wrong, or the dataflows backward:
> 
>  * we document how we generate a debian "upstream tarball" from a git
>    tag.  for example, we put this in debian/upstream/vcs-gen-tarball:
> 
>     git archive --format=tar --prefix=${projname}-${version} ${tagname} | gzip -9n

This is already a part of uscan.  It needs a bit more refinement.

>  * make a shallow clone of the git archive at the tag, including the
>    tag. (i've confirmed that a signed git tag in a shallow repo does
>    validate correctly).
> 
>      git clone --bare --depth 1 -b ${tagname} \
>             file://path/to/upstream/${projname}.git ${projname}-${version}.git

The quesion is, we need to make local full clone if git is served dumb
http server.  (Github is smart :-)  Shallow clone or git archive does
not work if git server is dumb.

>  * create an archive of the shallow clone, combined with the command to
>    generate the tarball (we can call this a "gtsig")
> 
>      rm -rf ${projname}-${version}.git/hooks
>      cp debian/upstream/vcs-gen-tarball ./${projname}-${version}.git
>      tar cz ./${projname}-${version}.git > ./${projname}-${version}.gtsig
> 
>  * write a simple tool to verify an orig.tar.gz against a signing key
>    and a gtsig, by extracting the "shallow clone" of the git repository,
>    verifying git tag -v, using git-archive, and then comparing the
>    results.

Is this simple?  Please show me working example as shell log.

 
> Some of the outstanding concerns:
> 
>  * what if there is non-DFSG-free data in the upstream git repo?  We
>    want to make sure we avoid shipping it to our mirrors.  that's why i
>    was leaning toward the "shallow clone", but if there are other
>    techniques, i'd be curious to hear them.

gtsig is from upstream on git repo.  So check it there, I think. (Not on
tar.)

If we do non-DFSG-free tarball generation, maintainer need to sign
the tarball.
 
>  * the .gtsig will be quite large -- roughly the same size as the

I am not quite sure about what you mean here.

>    orig.tar.gz.  Is it possible to make it smaller by just storing the
>    "delta" needed to recreate the shallow clone from the orig.tar.gz?
>    Or is it possible (though dirty) to ship the .gtsig itself as the
>    orig.tar.gz?  that smells like trouble, because you couldn't
>    reconstruct the sources without having git available.
> 
>  * the .gtsig itself will show "verified" but it could contain some data
>    that isn't actually covered by the tag.  Upon verification, how do we
>    make sure it's clean?  (i note that OpenPGP signature files also have
>    covert channels where they can carry unsigned material, so this might
>    not be introducing a new bug in general).
> 
>  * is "git archive" guaranteed to produce deterministic output?
> 
> What do folks think?  I'm sure i'm not the first person to think about
> this, but i don't know whether there is any existing work done on it
> either.  Pointers, thoughts, discussion welcome.
> 
>         --dkg

Anyway, let me get the direct support of git archive first.  If it
works, we worry about signature.  If you know particular archive in
mind, let us know.

Osamu