[apt-offline-devel] apt-offline install error: couldn't understand file type ..._Release.gpg

Mon Apr 4 11:43:57 UTC 2016

Hello again!

Warning: This is a long one.

I had some time to clone the current version of apt-offline from git and
try it out. It does seem like the problem with apt-offline not being
able to understand .gpg-files is fixed.

Unfortunately I ran into a couple of new issues. Here's what I did.
(In the following, the virtualisation host's hostname is 'iupiter' and
the container's hostname is 'metis'. The directory that both systems
share is always named /srv/lxc/${other_hostname}/, so it's
/srv/lxc/metis/ on the host 'iupiter' and it's /srv/lxc/iupiter/ on the
container 'metis'. I chose to omit some output with [...] where there
were no warnings or errors and everything worked as it should.)

* First, I cloned the current apt-offline version from github and
installed it on the container:

root at iupiter:~# mkdir -p git/github
root at iupiter:~# cd git/github
root at iupiter:~/git/github# git clone
https://github.com/rickysarraf/apt-offline.git
Cloning into 'apt-offline'...
[...]

root at iupiter:~/git/github# cd apt-offline/
root at iupiter:~/git/github/apt-offline# python setup.py build
running build
[...]

root at iupiter:~# cp -r git /srv/lxc/metis/

--- switch to container ---

root at metis:~# apt-get remove apt-offline
[...]

root at metis:~# cd /srv/lxc/iupiter/git/github/apt-offline/
root at metis:/srv/lxc/iupiter/git/github/apt-offline# python setup.py install
[...]

root at metis:~# which apt-offline
/usr/local/bin/apt-offline

* Then I ran a little test

root at metis:~# /usr/local/bin/apt-offline set
/srv/lxc/iupiter/apt-offline.sig

Generating database of files that are needed for an update.

WARNING: /usr/bin/apt does not have a stable CLI interface yet. Use with
caution in scripts.

Generating database of file that are needed for operation upgrade

WARNING: /usr/bin/apt does not have a stable CLI interface yet. Use with
caution in scripts.

--- switch to host ---

root at iupiter:~# /usr/bin/apt-offline get /srv/lxc/metis/apt-offline.sig
--download-dir /srv/lxc/metis/apt-offline --cache-dir
/var/cache/apt/archives

Fetching APT Data

Downloading
http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.bz2.
ERROR:
http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.bz2 -
404 - Not Found.
ERROR: Will still try with other package uris
http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.gz
done.
Downloading
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.bz2.
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.bz2
- 404 - Not Found.
ERROR: Will still try with other package uris
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.gz -
404 - Not Found.
ERROR: Will still try with other package uris
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.lzma
- 404 - Not Found.
ERROR: Will still try with other package uris
Downloading
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en.bz2.
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en.bz2
done.
Downloading http://http.debian.net/debian/dists/jessie/Release.gpg.
http://http.debian.net/debian/dists/jessie/Release.gpg done.
Downloading http://http.debian.net/debian/dists/jessie/Release.
http://http.debian.net/debian/dists/jessie/Release done.
Downloading http://http.debian.net/debian/dists/jessie/InRelease.
ERROR: http://http.debian.net/debian/dists/jessie/InRelease - 404 - Not
Found.
ERROR: Will still try with other package uris
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.bz2 - 404 -
Not Found.
ERROR: Will still try with other package uris
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.gz - 404 -
Not Found.
ERROR: Will still try with other package uris
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.lzma - 404 -
Not Found.
ERROR: Will still try with other package uris
Downloading 0.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File
"/usr/lib/python2.7/dist-packages/apt_offline_core/AptOfflineLib.py",
line 624, in run
    self.responseQueue.put( self.WorkerFunction( item, thread_name ) )
  File
"/usr/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py", line
820, in DataFetcher
    if DownloadPackages(url) is False and guiTerminateSignal is False:
  File
"/usr/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py", line
795, in DownloadPackages
    if FetcherInstance.download_from_web(url, pkgFile, Str_DownloadDir)
== True:
  File
"/usr/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py", line
248, in download_from_web
    temp = urllib2.urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 423, in open
    protocol = req.get_type()
  File "/usr/lib/python2.7/urllib2.py", line 285, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: 0

ERROR: The following files failed to be downloaded.
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.gz failed.
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.lzma
failed.
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.bz2 failed.
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.gz failed.
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.lzma failed.

Downloaded data to /srv/lxc/metis/apt-offline

* As I understand it, the error messages about the failed downloads of
Translation-en_US.... and InRelease... are nothing to fret about.
The exception is more worrying, though.
Now, at this point, the container, metis, had the current github version
of apt-offline installed in /usr/local/..., while the host, iupiter,
still ran version 1.5.1 from Debian's jessie repos.
I thought that maybe the signature file's format had subtly changed
between version, so I tried updating the host's apt-offline to the
github version, too:

root at iupiter:~# apt-get remove apt-offline
[...]

root at iupiter:~# cd git/github/apt-offline/
root at iupiter:~/git/github/apt-offline# python setup.py install
[...]

root at iupiter:~/git/github/apt-offline# which apt-offline
/usr/local/bin/apt-offline

* And then reran 'apt-offline get ...'

root at iupiter:~# apt-offline get /srv/lxc/metis/apt-offline.sig
--download-dir /srv/lxc/metis/apt-offline --cache-dir
/var/cache/apt/archives

Fetching APT Data

Downloading
http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.bz2
http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.gz
done
Downloading
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.bz2
Downloading
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en.bz2
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en.bz2 done
Downloading http://http.debian.net/debian/dists/jessie/Release.gpg
http://http.debian.net/debian/dists/jessie/Release.gpg done
Downloading http://http.debian.net/debian/dists/jessie/Release
http://http.debian.net/debian/dists/jessie/Release done
Downloading http://http.debian.net/debian/dists/jessie/InRelease
Downloading 0
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File
"/usr/local/lib/python2.7/dist-packages/apt_offline_core/AptOfflineLib.py",
line 636, in run
    self.responseQueue.put( self.WorkerFunction( item, thread_name ) )
  File
"/usr/local/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py",
line 1273, in DataFetcher
    if DownloadPackages(url) is False and guiTerminateSignal is False:
  File
"/usr/local/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py",
line 1247, in DownloadPackages
    if FetcherInstance.download_from_web(url, pkgFile, Str_DownloadDir)
== True:
  File
"/usr/local/lib/python2.7/dist-packages/apt_offline_core/AptOfflineCoreLib.py",
line 682, in download_from_web
    temp = urllib2.urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 423, in open
    protocol = req.get_type()
  File "/usr/lib/python2.7/urllib2.py", line 285, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: 0

ERROR: The following files failed to be downloaded.
Not all errors are fatal. For eg. Translation files are not present on
all mirrors.
ERROR:
http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.lzma
failed.
ERROR: http://http.debian.net/debian/dists/jessie/InRelease.lzma failed.

Downloaded data to /srv/lxc/metis/apt-offline

* Still the same exception. I decided to have a look at the signature
file to see where the "URL type 0" comes from:

root at iupiter:~# cat /srv/lxc/metis/apt-offline.sig
'http://http.debian.net/debian/dists/jessie/main/binary-amd64/Packages.bz2'
http.debian.net_debian_dists_jessie_main_binary-amd64_Packages 0 :
'http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en_US.bz2'
http.debian.net_debian_dists_jessie_main_i18n_Translation-en%5fUS 0 :
'http://http.debian.net/debian/dists/jessie/main/i18n/Translation-en.bz2' http.debian.net_debian_dists_jessie_main_i18n_Translation-en
0 :
'http://http.debian.net/debian/dists/jessie/InRelease'
http.debian.net_debian_dists_jessie_InRelease 0
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

* Clearly, the last line is the culprit. I suppose it's output from apt
that should't ordinarily end up in the .sig-file. I suspect apt's output
has changed. (To be fair, apt warns us about this. The message 'WARNING:
/usr/bin/apt does not have a stable CLI interface yet. Use with caution
in scripts.' turns up in the output of 'apt-offline set ...'.)
Since the rest of the URLs get handled correctly it seems that
everything that's needed should still get downloaded, so this is more of
an annoyance. Still, it's a bug and should get fixed. If you're not in a
great hurry about this, though, I'd file it and try my hand at fixing it
myself.
On with the programme... now I tried installing the downloaded packages
on the container:

root at metis:~# /usr/local/bin/apt-offline install
/srv/lxc/iupiter/apt-offline/
...
ERROR: Path for keyring is invalid: /etc/apt/trusted.gpg
gpgv: Signature made Sat 23 Jan 2016 02:19:39 PM CET using RSA key ID
46925553
gpgv: Good signature from "Debian Archive Automatic Signing Key
(7.0/wheezy) <ftpmaster at debian.org>"
gpgv: Signature made Sat 23 Jan 2016 02:19:39 PM CET using RSA key ID
2B90D010
gpgv: Good signature from "Debian Archive Automatic Signing Key
(8/jessie) <ftpmaster at debian.org>"
gpgv: Signature made Sat 23 Jan 2016 02:25:24 PM CET using RSA key ID
518E17E1
gpgv: Good signature from "Jessie Stable Release Key
<debian-release at lists.debian.org>"
http.debian.net_debian_dists_jessie_Release synced.
http.debian.net_debian_dists_jessie_Release.gpg synced.
http.debian.net_debian_dists_jessie_main_binary-amd64_Packages synced.
http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2 synced.
http.debian.net_debian_dists_jessie_main_i18n_Translation-en synced.
http.debian.net_debian_dists_jessie_main_i18n_Translation-en.bz2 synced.

root at metis:~# ls -l /etc/apt/trusted.gpg
ls: cannot access /etc/apt/trusted.gpg: No such file or directory

* OK, so for some reason the keyring's path is other than expected on
the container:

root at metis:~# ls -Al /etc/apt/
total 20
drwxr-xr-x 2 root root 4096 Feb 18 11:51 apt.conf.d
drwxr-xr-x 2 root root 4096 Sep 15  2015 preferences.d
-rw-r--r-- 1 root root  531 Mar 29 13:11 sources.list
drwxr-xr-x 2 root root 4096 Sep 15  2015 sources.list.d
drwxr-xr-x 2 root root 4096 Feb 18 11:50 trusted.gpg.d

root at metis:~# ls -Al /etc/apt/trusted.gpg.d/
total 36
-rw-r--r-- 1 root root 5138 Nov 30  2014 debian-archive-jessie-automatic.gpg
-rw-r--r-- 1 root root 5147 Nov 30  2014
debian-archive-jessie-security-automatic.gpg
-rw-r--r-- 1 root root 2775 Nov 30  2014 debian-archive-jessie-stable.gpg
-rw-r--r-- 1 root root 4084 Nov 30  2014
debian-archive-squeeze-automatic.gpg
-rw-r--r-- 1 root root 2853 Nov 30  2014 debian-archive-squeeze-stable.gpg
-rw-r--r-- 1 root root 3780 Nov 30  2014 debian-archive-wheezy-automatic.gpg
-rw-r--r-- 1 root root 2851 Nov 30  2014 debian-archive-wheezy-stable.gpg

* Still, the package signature could be verified OK and the packages
were installed (well, 'synced') successfully, so I should be able to
carry on:

root at metis:~# apt-get upgrade
Reading package lists... Error!
E: BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC)
E: Problem opening
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
E: The package lists or status file could not be parsed or opened.

# bunzip2 -t
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
bunzip2:
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2:
bad magic number (file not created by bzip2)

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

# bzip2recover
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
bzip2recover 1.0.6: extracts blocks from damaged .bz2 files.
bzip2recover: searching for block boundaries ...
bzip2recover: sorry, I couldn't find any block boundaries.

* It appears that the downloaded file has the same problem:

root at metis:~# bunzip2 -t
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
bunzip2:
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2:
bad magic number (file not created by bzip2)

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

root at metis:~# bzip2recover
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
bzip2recover 1.0.6: extracts blocks from damaged .bz2 files.
bzip2recover: searching for block boundaries ...
bzip2recover: sorry, I couldn't find any block boundaries.

* The two files differ, however:

root at metis:~# diff -u
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
Binary files
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
and
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
differ

--- switch to host (because I don't have 'file' installed on the
container) ---

root at iupiter:~# file --keep-going
/srv/lxc/metis/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2

/srv/lxc/metis/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2:
gzip compressed data, max compression, from Unix

root at iupiter:~# gunzip -t
/srv/lxc/metis/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2

root at iupiter:~# echo $?
0

* So it looks like, for whatever reason, this file is actually gzipped
and just has the wrong extension (.bz2, instead of .gz).

root at metis:~# mv
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
/srv/lxc/iupiter/apt-offline/http.debian.net_debian_dists
_jessie_main_binary-amd64_Packages.gz
root at metis:~# /usr/local/bin/apt-offline install
/srv/lxc/iupiter/apt-offline/

[...]
ERROR: Path for keyring is invalid: /etc/apt/trusted.gpg
gpgv: Signature made Sat 23 Jan 2016 02:19:39 PM CET using RSA key ID
46925553
gpgv: Good signature from "Debian Archive Automatic Signing Key
(7.0/wheezy) <ftpmaster at debian.org>"
gpgv: Signature made Sat 23 Jan 2016 02:19:39 PM CET using RSA key ID
2B90D010
gpgv: Good signature from "Debian Archive Automatic Signing Key
(8/jessie) <ftpmaster at debian.org>"
gpgv: Signature made Sat 23 Jan 2016 02:25:24 PM CET using RSA key ID
518E17E1
gpgv: Good signature from "Jessie Stable Release Key
<debian-release at lists.debian.org>"
[...]
http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.gz synced.
[...]

* The gzipped file with the incorrect .bz2-extension is still present in
/var/lib/apt/lists/... and causes problems when running 'apt-get
ugrade'. But it's identical to the (new) .gz-file:

root at metis:~# ls -l
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.gz
-rw-r--r-- 1 root root 33803992 Apr  4 02:36
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
-rw-r--r-- 1 root root 33803992 Apr  4 12:07
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.gz

root at metis:~# diff -q
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.gz

* So I thought it should be enough to remove it and things would work
out then, but no such luck:

root at metis:~# rm
/var/lib/apt/lists/http.debian.net_debian_dists_jessie_main_binary-amd64_Packages.bz2

root at metis:~# apt-get update
Err http://http.debian.net jessie InRelease

Err http://http.debian.net jessie Release.gpg
  Could not resolve 'http.debian.net'
Reading package lists... Error!
W: Failed to fetch http://http.debian.net/debian/dists/jessie/InRelease

W: Failed to fetch
http://http.debian.net/debian/dists/jessie/Release.gpg  Could not
resolve 'http.debian.net'

W: Some index files failed to download. They have been ignored, or old
ones used instead.
E: BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC)
E: The package lists or status file could not be parsed or opened.

root at metis:~# apt-get upgrade
Reading package lists... Error!
E: BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC)
E: The package lists or status file could not be parsed or opened.

* Now apt-get doesn't even tell me which file causes the problem, only
that it attempts to read the file with BZ2_bzread.

And that's pretty much where I got stuck and then ran out of time.
I will try looking for other files on the container with the .bz2
extension that aren't actually bzip2 files. If I can find and fix them,
that might do it.

But there are clearly a few problems here that need to be addressed.

Best
Gabriel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/apt-offline-devel/attachments/20160404/78b775fd/attachment.sig>