<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 24, 2014 at 3:28 AM, Bill Allombert <span dir="ltr"><<a href="mailto:ballombe@debian.org" target="_blank">ballombe@debian.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="">> Hello popcon-devs,<br>
><br>
> I have been working on a re-write of the popularity-contest application.<br>
> This originally started off as being a learning exercise to both test my<br>
> reading of perl code and to have a new project for some python code. I am<br>
> mostly happy with the current state of what I am calling pypopcon and<br>
> wanted to share my work:<br>
> <a href="https://github.com/drwahl/pypopcon" target="_blank">https://github.com/drwahl/pypopcon</a><br>
><br>
> There are some interesting (to me, at least) changes to this product as<br>
> compared to the currently used popularity-contest code.<br>
<br>
</div>Hello David,<br>
<br>
For reference: I (re)wrote popularity-contest in basic perl because perl-base is<br>
Essential: yes, so that installing popcon does not affect the popularity<br>
of other packages.<br>
<br>
For that reason, it seems unlikely that your script report the same results as the<br>
standard popularity-contest perl script.<br>
<div class=""><br>
> First, pypopcon explores all the files that a package provides and checks<br>
> for atime (instead of just a key binary a package provides). I believe<br>
> this increases the accuracy as some package ship with multiple binaries and<br>
> the one that popularity-contest uses isn't always the most used binary from<br>
> the package.<br>
<br>
</div>There are various files which atimes are changed without users action, for example<br>
by cron jobs, dpkg hooks. Including the files in the list means that the<br>
package atime became meaningless. For example, all shared libraries,<br>
all python modules, etc. Thus we use a regex to limit the list to 'safe' files.<br>
<br>
Instead the popcon backend use the dependency graph to mark as voted all packages<br>
that are depended on by voted packages (transitively).<br>
<div class=""><br>
> Secondly, pypopcon is showing a pretty decent performance increase (and<br>
> there is still room for more). On my system, popularity-contest takes<br>
> about 15 seconds to run whereas pypopcon is taking about 8 seconds to run.<br>
> One thing that is interesting about this metric is that pypopcon is<br>
> actually getting the atime/ctime of more files than the perl<br>
> popularity-contest script, so it's actually doing more work than<br>
> popularity-contest is, and it is doing it in less time.<br>
<br>
</div>You need to split system time and user time in your benchmark.<br>
The system time is very much dependent on file system performance.<br>
<br>
Cheers,<br>
<span class=""><font color="#888888">--<br>
Bill. <<a href="mailto:ballombe@debian.org">ballombe@debian.org</a>><br>
<br>
Imagine a large red swirl here.<br>
</font></span></blockquote></div><br>Thanks for the response Bill. I appreciate the feedback. One thing I forgot to mention about the pypocon that I wrote is that I also intended for it to be used on non-debian (dpkg) systems or debian systems not using apt. It currently only supports yum/RPM in addition to deb/apt, but I'm thinking of adding other package managers as well like gem and pip.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">As for the performance, here is the "time" output on my system (24 cores, 1.2GHz, 16GB of RAM, SSD):</div><div class="gmail_extra"><br></div><div class="gmail_extra">
popularity-contest:</div><div class="gmail_extra"><div class="gmail_extra">real 0m16.867s</div><div class="gmail_extra">user 0m6.024s</div><div class="gmail_extra">sys 0m10.495s</div><div><br></div><div>pypopcon:</div>
<div><div>real 0m8.297s</div><div>user 0m6.391s</div><div>sys 0m1.513s</div></div><div><br></div><div><br></div><div>As for the atime of files changing without user interaction, this is something that I struggled with a bit. What it actually means to "use" a package is really kinda a grey area to me, so I kind of went my own direction with it (maybe incorrectly so). It seems to me like files that a user is using even indirectly might be considered to be "popular" as the user could be using them to maintain their system in an automated fashion. I probably need to think about that some more...</div>
<div><br></div><div>Thanks again for the feedback!</div><div>David W.</div>-- <br>Unix, because every barista in Seattle has an MCSE.
</div></div>