[Debtorrent-devel] The Next Steps: Apt Interaction

[Cameron Dale]

On 6/1/07, Anthony Towns <aj at azure.humbug.org.au> wrote:
> So other than this, one thing I think's worth getting underway soonish
> (as we talked about on IRC), was getting the maximum piece size reduced
> to something reasonable (512kB or 256kB presumably). I've written a
> quick python script to extract the needed info (attached) and run it
> over contrib, the results of which you can see at [0].

Great work, thanks aj! I've created a branch in the svn repository for
working with this. I just tested it on the etch-contrib-i386 and it
worked fine. You can find the branch by checking out (or switching to)
/debtorrent/branches/hippy. (BTW, I love the name.)

> Rather than make it a bunch of 512kB pieces plus a variable length tail
> piece, I've made it try to keep the pieces to roughly the same size
> instead, which seemed worthwhile (though, oops, my maths is slightly
> off). I didn't include any info about packages that're 512kB or less.

Seems reasonable. I did come up with one reason why piece sizes are a
power of 2. The data is transferred between peers in blocks of 16384
bytes (or less). Worst case, if the piece size is a power of 2 (or a
multiple of 16384) is that one block (the last block of the last
piece) could be 1 byte. However, worst case with the roughly same-size
pieces is that the piece size chosen is (16384*n + 1) so then you have
number-of-pieces 1 byte blocks. (Needless to say, small blocks are
less efficient to communicate, due to increasing overhead.) It's a
minor point, but it might be better to make the roughly-the-same-size
piece size a multiple of 16384. If you need some help with the maths
on that one, let me know, I know some profs at my university who can
figure it out for us.. ;)

> Before it can realistically be run over main it'll need to cache its
> results, rather than scanning the same .deb repeatedly, but otherwise
> should be okay. I've done a test run for etch/main/i386 anyway though,
> just to see what it's like.

Caching will definitely be needed, as will constant re-runs to keep it
up to date with the current testing/unstable. I'm not sure how best to
do that, or whether to just stick to stable at this point for
simplicity.

Also, I can't quite remember all the details, but wasn't there an
issue with the Filename in the pool not being unique? Maybe the
-extrapieces files should include the SHA1 hash of the whole package
as well? Just a thought.

Cameron