Bug#685787: [rafael at laboissiere.net: Re: Patch for devscripts]
Andreas Tille
tille at debian.org
Tue Oct 29 09:37:29 UTC 2013
Hi again,
just to make sure that Rafael's work is propagated in case it might
help. I understood James' mail and that he could work with the existing
Git repository but since Rafael did some style changes this patch might
be more clean.
Kind regards
Andreas.
----- Forwarded message from Rafael Laboissiere <rafael at laboissiere.net> -----
Date: Tue, 29 Oct 2013 09:36:39 +0100
From: Rafael Laboissiere <rafael at laboissiere.net>
To: Andreas Tille <tille at debian.org>
Subject: Re: Patch for devscripts
* Andreas Tille <tille at debian.org> [2013-10-28 22:56]:
>
> On Mon, Oct 28, 2013 at 10:37:13PM +0100, Rafael Laboissiere wrote:
>>
>> If you agree, I can give it a try. Remember that users of Git
>> appreciate when you play the Git rules.
>
> For sure I agree! On one hand this gives another pair of eyeballs
> looking at my crappy code. On the other hand I can spent my time
> for other burning things in Debian Med. So feel free to move on
> and ping me if you need any help.
Ok, I succeeded to isolate the changes related to Files-Excluded and
put them all into a single patch, which you will find attached below.
This patch applies cleanly to the current qdevscript repository's
master:
git clone git://anonscm.debian.org/collab-maint/devscripts.git
cd devscripts
patch -p1 < /path/to/files-excluded.patch
The resulting uscan.pl script works fine for me (at least, for the
praat package). Of course, it does not have the repack-compression
feature.
You will note that I did a couple of stylistic changes. I also
removed those comments related to Parse::DebControl. The goal is to
provide a slim patch, such that its chances to be accepted are higher.
I did not revise your Perl code in detail, though (since it ain't
broken, I won't fix it!).
In order to produce a proper Git patch you must give me a decent
commit message. It does not need to be short. On the contrary, it
must be quite clear on what has been changed. I think that an
explanation as you gave in the report of Bug#685787 would be fine.
> Perhaps it makes sense to update the Wiki page about what's going on.
Probably, yes.
Best,
Rafael
diff --git a/debian/control b/debian/control
index d48f7f2..b9dc690 100644
--- a/debian/control
+++ b/debian/control
@@ -16,6 +16,7 @@ Build-Depends: debhelper (>= 9),
libparse-debcontrol-perl,
libterm-size-perl,
libtimedate-perl,
+ libtry-tiny-perl,
liburi-perl,
libwww-perl,
lsb-release,
@@ -50,6 +51,7 @@ Recommends: at,
libencode-locale-perl,
libjson-perl,
libparse-debcontrol-perl,
+ libtry-tiny-perl,
liburi-perl,
libwww-perl,
lintian,
diff --git a/scripts/uscan.1 b/scripts/uscan.1
index af4e57f..fb53f3e 100644
--- a/scripts/uscan.1
+++ b/scripts/uscan.1
@@ -444,6 +444,10 @@ Give verbose output.
.B \-\-no\-verbose
Don't give verbose output. (This is the default behaviour.)
.TP
+.B \-\-no\-exclusion
+Do not automatically exclude files mentioned in
+\fIdebian/copyright\fR field \fBFiles-Excluded\fR
+.TP
.B \-\-debug
Dump the downloaded web pages to stdout for debugging your watch file.
.TP
@@ -517,6 +521,10 @@ equivalent to the \fB\-\-destdir\fR option.
If this is set to \fIyes\fR, then after having downloaded a bzip tar,
lzma tar, xz tar, or zip archive, \fBuscan\fR will repack it to a gzip tar.
This is equivalent to the \fB\-\-repack\fR option.
+.B USCAN_NO_EXCLUSION
+If this is set to \fIyes\fR, files mentioned in the field \fBFiles-Excluded\fR
+of \fIdebian/copyright\fR will be ignored and no exclusion of files will be
+tried. This is equivalent to the \fB\-\-no-exclusion\fR option.
.SH "EXIT STATUS"
The exit status gives some indication of whether a newer version was
found or not; one is advised to read the output to determine exactly
diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index 976b368..5dc8a6e 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -27,6 +27,8 @@ use strict;
use Cwd;
use Cwd 'abs_path';
use Dpkg::IPC;
+use Dpkg::Control::Hash;
+use Try::Tiny;
use File::Basename;
use File::Copy;
use File::Temp qw/tempfile tempdir/;
@@ -46,6 +48,7 @@ BEGIN {
}
}
}
+
my $CURRENT_WATCHFILE_VERSION = 3;
my $progname = basename($0);
@@ -72,6 +75,7 @@ sub uscan_die (@);
sub dehs_output ();
sub quoted_regex_replace ($);
sub safe_replace ($$);
+sub get_main_source_dir ($$$$$);
sub usage {
print <<"EOF";
@@ -138,6 +142,8 @@ Options:
--no-conf, --noconf
Don\'t read devscripts config files;
must be the first option given
+ --no-exclusion no automatic exclusion of files mentioned in
+ debian/copyright field Files-Excluded
--help Show this message
--version Show version information
@@ -180,6 +186,7 @@ my $dehs_start_output = 0;
my $pkg_report_header = '';
my $timeout = 20;
my $user_agent_string = 'Debian uscan ###VERSION###';
+my $no_exclusion = 0;
if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
$modified_conf_msg = " (no configuration files read)";
@@ -196,6 +203,7 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
'USCAN_DEHS_OUTPUT' => 'no',
'USCAN_USER_AGENT' => '',
'USCAN_REPACK' => 'no',
+ 'USCAN_NO_EXCLUSION' => 'no',
'DEVSCRIPTS_CHECK_DIRNAME_LEVEL' => 1,
'DEVSCRIPTS_CHECK_DIRNAME_REGEX' => 'PACKAGE(-.+)?',
);
@@ -233,6 +241,8 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
or $config_vars{'USCAN_DEHS_OUTPUT'}='no';
$config_vars{'USCAN_REPACK'} =~ /^(yes|no)$/
or $config_vars{'USCAN_REPACK'}='no';
+ $config_vars{'USCAN_NO_EXCLUSION'} =~ /^(yes|no)$/
+ or $config_vars{'USCAN_NO_EXCLUSION'}='no';
$config_vars{'DEVSCRIPTS_CHECK_DIRNAME_LEVEL'} =~ /^[012]$/
or $config_vars{'DEVSCRIPTS_CHECK_DIRNAME_LEVEL'}=1;
@@ -263,7 +273,7 @@ if (@ARGV and $ARGV[0] =~ /^--no-?conf$/) {
# Now read the command line arguments
my $debug = 0;
my ($opt_h, $opt_v, $opt_destdir, $opt_download, $opt_force_download,
- $opt_report, $opt_passive, $opt_symlink, $opt_repack);
+ $opt_report, $opt_passive, $opt_symlink, $opt_repack, $opt_no_exclusion);
my ($opt_verbose, $opt_level, $opt_regex, $opt_noconf);
my ($opt_package, $opt_uversion, $opt_watchfile, $opt_dehs, $opt_timeout);
my $opt_download_version;
@@ -295,6 +305,7 @@ GetOptions("help" => \$opt_h,
"useragent=s" => \$opt_user_agent,
"noconf" => \$opt_noconf,
"no-conf" => \$opt_noconf,
+ "no-exclusion" => \$opt_no_exclusion,
"download-current-version" => \$opt_download_current_version,
)
or die "Usage: $progname [options] [directories]\nRun $progname --help for more details\n";
@@ -318,6 +329,7 @@ $timeout = 20 unless defined $timeout and $timeout > 0;
$symlink = $opt_symlink if defined $opt_symlink;
$verbose = $opt_verbose if defined $opt_verbose;
$dehs = $opt_dehs if defined $opt_dehs;
+$no_exclusion = $opt_no_exclusion if defined $opt_no_exclusion;
$user_agent_string = $opt_user_agent if defined $opt_user_agent;
$download_version = $opt_download_version if defined $opt_download_version;
@@ -1480,6 +1492,63 @@ EOF
}
}
+ my $excludesuffix = '+dfsg';
+ if ( !$no_exclusion ) {
+ my $data ;
+ $data = Dpkg::Control::Hash->new();
+ try {
+ $data->load('debian/copyright');
+ } catch {
+ print "-- No machine readable debian/copyright file.\n" if ( $verbose ) ;
+ $data->{'format'} = '' ;
+ } ;
+ my $okformat = qr'http://www.debian.org/doc/packaging-manuals/copyright-format/[.\d]+';
+ print "-- Wrong format of debian/copyright file to profit from Files-Excluded.\n" if ( $data->{'files-excluded'} and $data->{'format'} !~ m{^$okformat/?$} and $verbose ) ;
+ if ($data->{'format'} =~ m{^$okformat/?$} and $data->{'files-excluded'} ) {
+ my $tempdir = tempdir ( "uscanXXXX", TMPDIR => 1, CLEANUP => 1 );
+ my $globpattern = "*";
+ my $hidden = ".[!.]*";
+ if (defined glob("$tempdir/$hidden")) {
+ $globpattern .= " $hidden";
+ }
+ my $absdestdir = abs_path($destdir);
+ unless ( system("cd $tempdir; tar -xaf \"$absdestdir/$newfile_base\" 2>/dev/null") == 0 ) {
+ print "-- $newfile_base is no tarball. Try unzip.\n" if $verbose;
+ # try unzip if tar fails - we do want to do something sensible even if no --repack was specified
+ system('command -v unzip >/dev/null 2>&1') >> 8 == 0
+ or die("unzip binary not found. This would serve as fallback because tar just failed.\n");
+ system('unzip', '-q', '-a', '-d', $tempdir, "$destdir/$newfile_base") == 0
+ or die("Repacking from zip to tar.gz failed (could not unzip)\n");
+ }
+ # Some source archives contain a useless __MACOSX dir which would prevent a reasonable
+ # normalising of the +dfsg.orig archive - so removing it in advance in case it should
+ # be removed anyway helps creating normalised source archives.
+ my $exclude__MACOSX = grep( /\s*\/?__MACOSX\/?\s*/, $data->{"files-excluded"} );
+ my $main_source_dir = get_main_source_dir($tempdir, $pkg, $newversion, $excludesuffix, $exclude__MACOSX);
+ unless ( -d $main_source_dir ) {
+ print STDERR "Error: $main_source_dir is no directory";
+ }
+ my $nfiles_before = `find "$main_source_dir" | wc -l`;
+ foreach (split /\s+/, $data->{"files-excluded"}) {
+ # delete trailing '/' because otherwise find -path will fail
+ s?/+$?? ;
+ # use -depth to enable deleting directories
+ system('find',$main_source_dir,'-depth','-path',"$main_source_dir/$_",qw(-exec rm -rf {} ;))==0 or
+ die "failure to run find properly";
+ };
+ my $nfiles_after = `find "$main_source_dir" | wc -l`;
+ if ( $nfiles_before == $nfiles_after && ! $exclude__MACOSX ) {
+ print "-- Source tree remains identical - no need for repacking.\n" if $verbose;
+ } else {
+ my $suffix = 'gz' ;
+ my $newfile_base_dfsg = "${pkg}_${newversion}${excludesuffix}.orig.tar.$suffix" ;
+ system("cd $tempdir; GZIP='-n -9' tar --owner=root --group=root --mode=a+rX -czf \"$absdestdir/$newfile_base_dfsg\" $globpattern") == 0
+ or die("Excluding files failed (could not create tarball)\n");
+ $symlink = 'files-excluded' # prevent symlinking or renaming
+ }
+ }
+ }
+
my @renames = (
[qr/\.(tar\.gz|tgz)$/, 'gz'],
[qr/\.(tar\.bz2|tbz2?)$/, 'bz2'],
@@ -1506,6 +1575,8 @@ EOF
print " and symlinked $renamed_base to it\n";
} elsif ($symlink eq 'rename') {
print " and renamed it as $renamed_base\n";
+ } elsif ($symlink eq 'files-excluded') {
+ print " and removed files from it in ${pkg}_${newversion}${excludesuffix}.orig.tar.$suffix\n";
}
} elsif ($dehs) {
my $msg = "Successfully downloaded updated package $newfile_base";
@@ -1514,6 +1585,8 @@ EOF
$msg .= " and symlinked $renamed_base to it";
} elsif ($symlink eq 'rename') {
$msg .= " and renamed it as $renamed_base";
+ } elsif ($symlink eq 'files-excluded') {
+ $msg .= " and removed files from it in ${pkg}_${newversion}${excludesuffix}.orig.tar.$suffix\n";
} else {
$dehs_tags{'target'} = $newfile_base;
}
@@ -1524,6 +1597,8 @@ EOF
print " and symlinked $renamed_base to it\n";
} elsif ($symlink eq 'rename') {
print " and renamed it as $renamed_base\n";
+ } elsif ($symlink eq 'files-excluded') {
+ print " and removed files from it in ${pkg}_${newversion}${excludesuffix}.orig.tar.$suffix\n";
}
}
last;
@@ -2066,3 +2141,56 @@ sub safe_replace($$) {
return 1;
}
}
+
+sub get_main_source_dir($$$$$) {
+ my ($tempdir, $pkg, $newversion, $excludesuffix, $exclude__MACOSX) = @_;
+ my $fcount = 0;
+ my $main_source_dir = '' ;
+ my $any_dir = '' ;
+ opendir DIR, $tempdir or die "opendir $tempdir: $!";
+ my @files = readdir DIR ;
+ closedir DIR ;
+ foreach my $file (@files) {
+ unless ($file =~ /^\.\.?/) {
+ if ( $exclude__MACOSX && $file =~ /^__MACOSX$/ ) {
+ `rm -rf ${tempdir}/__MACOSX` ;
+ next ;
+ }
+ $fcount++;
+ if ( -d $tempdir.'/'.$file ) {
+ $any_dir = $tempdir . '/' . $file ;
+ # check whether there is some dir in upstream source which looks reasonable
+ # If such dir exists, we do not try to undirty the directory structure
+ $main_source_dir = $any_dir if ( $file =~ /^$pkg\w*$newversion$/i ) ;
+ }
+ }
+ }
+ if ( $fcount == 1 and $main_source_dir ) {
+ return $main_source_dir ;
+ }
+ if ( $fcount == 1 and $any_dir ) {
+ # Unusual base dir in tarball - should be rather something like ${pkg}-${newversion}
+ $main_source_dir = $tempdir . '/' . $pkg . '-' . $newversion . $excludesuffix . '.orig';
+ move($any_dir, $main_source_dir) or die("Unable to move $any_dir directory $main_source_dir\n");
+ return $main_source_dir ;
+ }
+ print "-- Dirty tarball found.\n" if $verbose;
+ if ( $main_source_dir ) { # if tarball is dirty but does contain a $pkg-$newversion dir we will not undirty but leave it as is
+ print "-- No idea how to create proper tarball structure - leaving as is.\n" if $verbose;
+ return $tempdir;
+ }
+ print "-- Move files to subdirectory $pkg-$newversion.\n" if $verbose;
+ $main_source_dir = $tempdir . '/' . $pkg . '-' . $newversion . $excludesuffix . '.orig';
+ mkdir($main_source_dir) or die("Unable to create temporary source directory $main_source_dir\n");
+ foreach my $file (@files) {
+ unless ($file =~ /^\.\.?/) {
+ if ( -d "${tempdir}/$file" ) {
+ # HELP: why can't perl move not move directories????
+ system( "mv ${tempdir}/$file $main_source_dir" ) ;
+ } else {
+ move("${tempdir}/$file", $main_source_dir) or die("Unable to move ${tempdir}/$file directory $main_source_dir\n");
+ }
+ }
+ }
+ return $main_source_dir;
+}
----- End forwarded message -----
--
http://fam-tille.de
More information about the devscripts-devel
mailing list