Introducing apt-p2p: a new P2P apt downloader

[Cameron Dale]

I recently wrote a blog post describing a new P2P downloader for apt
that I wrote called apt-p2p. I thought I'd post it here, as it's
functionality is very similar to debtorrent. You can find the original
post here:

    http://www.camrdale.org/blog/posts/May-04-2008.html



After many long months of planning and work, I have completed
*another* peer-to-peer downloader for Debian. If you've been keeping
track, that makes 2 now. This one is called apt-p2p[1], and as of
yesterday it is available in unstable.

The functionality is very similar to the first one I wrote,
DebTorrent, so if you've used that one you should feel very
comfortable. After installation you just add a `localhost:9977` to the
start of your sources.list entries (see the man page). The only
difference is the port number (DebTorrent uses 9988 by default) and
that you can use it on all your sources.list entries, whether they be
`deb`, `deb-src`, official Debian archive, or any other archive. Then
an `apt-get update` gets it started, and you can begin installing
packages. Point your web browser to `http://localhost:9977` to see
what's going on.

*IMPORTANT*: as with any P2P program, it works much better if you open
a port through your NAT or firewall. Without this crucial step, you
won't be able to share with any peers, and your lookup of peers to
download from may take longer. Make sure to forward both TCP and UDP
ports 9977, or whichever port you set in the config file. For more
details, see the port forwarding section of the FAQ[2].

The similarities with DebTorrent are all external, so let's look at
how it differs from DebTorrent internally:

 * it's **very** general, it doesn't matter what you're trying to
   download, a source package or a Packages file, from constantly-updated
   unstable or a year-old stable, for i386 or hppa architectures
   (DebTorrent only works for `.deb` files, is only supported for the
   official archive, and breaks downloaders into groups by architectures)
 * it doesn't require anything other than what's available to apt
   (DebTorrent uses piece hashes of large files, and ordering information,
   both of which are downloaded separately)
 * it can be very fast when downloading from mirrors (mirror downloads
   with DebTorrent are not so fast)
 * the code is simple, and makes use of available code, such as
   Twisted[3], Khashmir[4], and python-apt[5], which should make
   future enhancements and maintenance easier (DebTorrent is large and
   monolithic)
 * requires less memory and CPU power (50% to 75% less memory than
   DebTorrent)

Here's some technical details for those interested:

 * makes use of hashes to uniquely identify files
 * uses a Distributed Hash Table (DHT)[6] to find peers
 * also stores piece hashes in the DHT for efficient downloading
 * uses HTTP/1.1 requests to download from peers
 * no peers available causes it to fallback to a download from the
   mirror

Though I know it can be fast, I'm not yet sure if the peer lookup in the
DHT will be quick enough to keep up with the downloading. All my tests
so far show that it is, but until there are a number of peers out there
trying it, I can't be sure. Also, I have some improvements in mind to
enhance the speed, in particular the wait for a timeout to occur, so
this may improve in the future. If you see a delay in downloading where
apt seems to be stalled saying 'Waiting for headers', for now be patient
and see what happens. It may be that apt-p2p is downloading in the
background (it does this sometimes), or that it's waiting for a lookup
in the DHT to complete. If it hangs for more than a minute, or there's
errors in the log file, please file a bug so that I can look into it.

Finally, for you DebTorrent fans don't worry, I haven't given up on it.
Stay tuned for more info on it coming soon.

[1]: http://www.camrdale.org/apt-p2p.html
[2]: http://www.camrdale.org/apt-p2p/faq.html
[3]: http://twistedmatrix.com/trac/
[4]: http://khashmir.sourceforge.net/
[5]: http://packages.debian.org/unstable/python/python-apt
[6]: http://en.wikipedia.org/wiki/Distributed_hash_table

Thanks,
Cameron