[Debtorrent-devel] The Next Steps: Apt Interaction

[Anthony Towns]

On Wed, May 30, 2007 at 03:27:09PM -0700, Cameron Dale wrote:
> After the recent release, the next step in the process is a further integration
> with apt to download from archives. From the wiki:

So other than this, one thing I think's worth getting underway soonish
(as we talked about on IRC), was getting the maximum piece size reduced
to something reasonable (512kB or 256kB presumably). I've written a
quick python script to extract the needed info (attached) and run it
over contrib, the results of which you can see at [0].

Rather than make it a bunch of 512kB pieces plus a variable length tail
piece, I've made it try to keep the pieces to roughly the same size
instead, which seemed worthwhile (though, oops, my maths is slightly
off). I didn't include any info about packages that're 512kB or less.

Before it can realistically be run over main it'll need to cache its
results, rather than scanning the same .deb repeatedly, but otherwise
should be okay. I've done a test run for etch/main/i386 anyway though,
just to see what it's like.

Cheers,
aj

[0] http://merkel.debian.org/~ajt/extrapieces/
-------------- next part --------------
#!/usr/bin/env python

# hippy -- the "piece man(ager)"

import sha
import os, sys

def hash(file, piecesize):
	h = []
	while 1:
		x = file.read(piecesize)
		if x == "": break
		h.append((sha.new(x).hexdigest(), len(x)))
	return h

piecesize = 512*1024
for filename in sys.argv[1:]:
	size = os.stat(filename).st_size
	if size <= piecesize: continue

	ps = int(size / int(size/piecesize + 1))+1
	file = open(filename)
	y = hash(file, ps)
	print "Filename: %s" % (filename)
	print "SHA1-Pieces:"
	for x in y:
		print " %s %d" % x
	print ""
	file.close()

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 155 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/debtorrent-devel/attachments/20070601/deaaebf7/attachment.pgp