[Debtorrent-devel] The Next Steps: Apt Interaction

[Anthony Towns]

On Mon, Jun 04, 2007 at 11:34:57PM -0700, Cameron Dale wrote:
> [...] It's a
> minor point, but it might be better to make the roughly-the-same-size
> piece size a multiple of 16384. 

Okay, updated. (Well, updating)

> If you need some help with the maths
> on that one, let me know, I know some profs at my university who can
> figure it out for us.. ;)

Heh, it was only that I couldn't remember what python for ceil() is
(turns out it's math.ceil()). Current version uses a loop rather than
calculating a number though; damn lazy CS people...

In some cases it'd be better to have the last piece slightly larger than
all the other pieces; in others it'd be better to have the first n pieces
larger than the next m. I've ignored those possibilities as Too Hard. :)

> Also, I can't quite remember all the details, but wasn't there an
> issue with the Filename in the pool not being unique? Maybe the
> -extrapieces files should include the SHA1 hash of the whole package
> as well? Just a thought.

Well, it's consistent across the archive at any point in time; but it may
not be consistent for all points in time (ie, pool/main/xyzzy_1.0_i386.deb
might be one thing today, be deleted next week, and reappear as something
different two months from now). It's not meant to happen, but it may.

More an issue when seeding different versions of the same suite from
different points in time (ie, unstable from two months through to today),
than in generating dtorrents I think. ie, worry about it later.

Cheers,
aj

-------------- next part --------------
#!/usr/bin/env python

import sha
import os, sys

def hash(file, piecesize):
        h = []
        while 1:
                x = file.read(piecesize)
                if x == "": break
                h.append((sha.new(x).hexdigest(), len(x)))
        return h

piecesize = 512*1024
chunksize = 16*1024

def optimalpiecesize(size):
    def eval(s,c,m):
        b = m/c
        return [ i*c for i in range(int(b/2), b+1) if s - i*c*int(s/m) <= i*c ]

    def score(s,c,m):
        l = int(s/m)
        return [ (abs(i - (s - l*i)), i) for i in eval(s,c,m) ]

    def bestest(s,c,m): 
        return min( score(s,c,m) )

    return bestest(size,chunksize,piecesize)[1]

for filename in sys.argv[1:]:
        size = os.stat(filename).st_size
        if size <= piecesize: continue

        ps = optimalpiecesize(size)
        file = open(filename)
        y = hash(file, ps)
        print "Filename: %s" % (filename)
        print "SHA1-Pieces:"
        for x in y:
                print " %s %d" % x
        print ""
        file.close()


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 155 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/debtorrent-devel/attachments/20070606/2d6a45b9/attachment.pgp