Bug#826039: liblwp-protocol-https-perl: Two versions of https.pm (6.06) have different contents and one always checks certificates

Fri Jun 3 19:35:19 UTC 2016

Thanks for the info, that makes sense.

I tried a clean install, loaded the package, then tried to add the "working"
version via CPAN, however it reports "up to date" but the module isn't
actually in the local tree, only the share tree. I then tried a cpan force
install but that still reported it was up to date. If it had installed it
would solve my particular problem, but as it refuses to install via CPAN it
is difficult to get the CPAN version and for that to become available in the
path as the first referenced module.

I guess on one of my earlier installs I loaded via CPAN before apt-get and
that's why that happened to work.

- - -  - -  - - 

# perldoc -lm LWP::Protocol::https
/usr/share/perl5/LWP/Protocol/https.pm

# cpan install LWP::Protocol::https
Loading internal null logger. Install Log::Log4perl for logging messages
Reading '/root/.cpan/Metadata'
  Database was generated on Wed, 01 Jun 2016 14:54:02 GMT
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
Reading '/root/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................
DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/root/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Fri, 03 Jun 2016 18:53:42 GMT
............................................................................
DONE
Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/root/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /root/.cpan/Metadata
LWP::Protocol::https is up to date (6.06).

# perldoc -lm LWP::Protocol::https
/usr/share/perl5/LWP/Protocol/https.pm

# ls -l /usr/local/share/perl/5.20.2/LWP/Protocol/https.pm
ls: cannot access /usr/local/share/perl/5.20.2/LWP/Protocol/https.pm: No
such file or directory

- - -  - -  - - 

# cpan force install LWP::Protocol::https
Loading internal null logger. Install Log::Log4perl for logging messages
Reading '/root/.cpan/Metadata'
  Database was generated on Fri, 03 Jun 2016 18:53:42 GMT
Warning: Cannot install force, don't know what it is.
Try the command

    i /force/

to find objects with matching identifiers.
Running install for module 'install'
Fetching with LWP:
http://www.cpan.org/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz
Checksum for
/root/.cpan/sources/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz ok
'YAML' not installed, will not store persistent state
Configuring D/DA/DAGOLDEN/install-0.01.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for install
Writing MYMETA.yml and MYMETA.json
  DAGOLDEN/install-0.01.tar.gz
  /usr/bin/perl Makefile.PL INSTALLDIRS=site -- OK
Running make for D/DA/DAGOLDEN/install-0.01.tar.gz
cp lib/install.pm blib/lib/install.pm
Manifying blib/man3/install.3pm
  DAGOLDEN/install-0.01.tar.gz
  /usr/bin/make -- OK
Running make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-MTest::Harness"
"-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib',
'blib/arch')" t/*.t
t/install.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.04 usr  0.02 sys +  0.04 cusr  0.00
csys =  0.10 CPU)
Result: PASS
  DAGOLDEN/install-0.01.tar.gz
  /usr/bin/make test -- OK
Running make install
Installing /usr/local/share/perl/5.20.2/install.pm
Installing /usr/local/man/man3/install.3pm
Appending installation info to
/usr/local/lib/x86_64-linux-gnu/perl/5.20.2/perllocal.pod
  DAGOLDEN/install-0.01.tar.gz
  /usr/bin/make install  -- OK
LWP::Protocol::https is up to date (6.06).

-----Original Message-----
From: Niko Tyni [mailto:ntyni at debian.org] 
Sent: 03 June 2016 14:16
To: Adrian Edwards; 826039 at bugs.debian.org
Subject: Re: Bug#826039: liblwp-protocol-https-perl: Two versions of
https.pm (6.06) have different contents and one always checks certificates

Control: severity -1 normal
Control: tags -1 =

On Wed, Jun 01, 2016 at 08:27:10PM +0100, Adrian Edwards wrote:
> Package: liblwp-protocol-https-perl
> Version: 6.06-2
> Severity: grave
> Tags: newcomer patch
> Justification: renders package unusable

> Trying to use $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; within perl 
> script to prevent verification of a self signed certificate, the 
> script worked sometimes and not others on different Debian 8 installs. 
> Did a bunch of installs of minimal system, added packages and modules
(apt-get and CPAN) and the module ALWAYS wanted to verify the cert and
didn't honour the environment variable to ignore the check.

> After a huge amount of wasted time I discovered there are two version 
> of the protocol module, both have the same version number 6.06 but the
contents of the files are different.
> 
> 	/usr/local/share/perl/5.20.2/LWP/Protocol/https.pm    Correct/Works
> 	/usr/share/perl5/LWP/Protocol/https.pm    In Error / Always fails
the check

Hi, first of all: packages installed with apt get installed in /usr/share
and /usr/lib (vendorlib and vendorarch). Modules installed with the cpan
shell go into /usr/local (sitelib and sitearch) and are first on the search
path, so it's possible to override packaged versions with newer ones from
CPAN if desired.

The difference between the two files on your system is because of patches in
the Debian package. You can find those in the source package under
debian/patches, or on the web at for instance
https://sources.debian.net/src/liblwp-protocol-https-perl/6.06-2/debian/patc
hes/

It looks like your issue comes from
746579-fix-peer-certificate-verification.patch, which is an intentional
change: PERL_LWP_SSL_VERIFY_HOSTNAME now only affects hostname verification
in the certificate, but doesn't disable certificate checking completely like
in the CPAN version. There's a long related discussion in
https://github.com/libwww-perl/lwp-protocol-https/pull/14 if you're
interested.

Disabling certificate checking completely should still be possible, at least
by explicitly passing both SSL_verify_mode => 0 and verify_hostname => 0 to
LWP::UserAgent->new() in ssl_opts. PERL_LWP_SSL_VERIFY_HOSTNAME in the
environment can be used for the latter option.

The same version number in the files is unfortunate: the Perl/CPAN
versioning scheme doesn't offer a version suffix or anything like that for
downstream packagers, and we can't really steal upstream version numbers
that might get used there later.

While there's clearly room for documenting things better, this doesn't make
the package unusable in the general case. I'm therefore lowering the
severity of this bug.
-- 
Niko Tyni   ntyni at debian.org