[Soc-coordination] PyPI to Debian repository converter

[Natalia Frydrych]

Hello,

This is my fifth report on the work progress on a project PyPI to Debian 
Repository Converter.

At the beginning I would like to say something personal - the day last 
report was about to be sent, my flat was robbed. I lost my laptop (the 
code is available on gitorious, but apparently thieves do not understand 
the idea of open source very well ;-)). I've lost a lot of time dealing 
with this situation, finding temporary hardware (which unfortunately 
turned out to be defective), installing system, setting up environment, 
etc. Therefore I had less time than I’ve expected, but I will be working 
full time on this project for at least two additional weeks to make it up.

Work
----------
After sending last report, I was able to :

* implement tests plugins (lintian[1], lintian4py[2] (which includes 
pyflakes tests))
I realized that package testing is a very broad topic, and in few days I 
could only vaguely familiarise myself with few tools. At this moment, 
each error returned by test plugin subtracts (from originally granted 10 
points) one point and each warning - half of a point. If package gathers 
less than 5 points in given test, it doesn’t pass it and is not added to 
the repository.

* implement an option to provide HTML logs
After looking at capabilities of several template systems, I decided to 
use Jinja[3] Python template engine. HTML files are not pretty, they 
need more work, including some CSS polishing. I'm going to try to 
replicate good
practice presentation of logs from similar sites, e.g. lintian logs[1] 
or buildd.debian.org[4]. At the moment logs are generated in separate 
directory, there is one file per process, for each package you can view 
logs sorted by session and by the type of process. New --list-sessions 
command line option was added to list available sessions - it can be 
used to choose sessions for which logs will be generated.

* implement build plugins: sbuild[5] and pbuilder[6]
I wanted to improve efficiency of building binary packages by adding to 
the list of available plugins more complex tools than pure 
dpkg-buildpackage. I’ve implemented above-mentioned plugins and I really 
appreciate e.g. their ability to repair unmet dependencies. 
Unfortunately, building packages on using these tools is quite 
time-consuming (especially that, as I wrote at the beginning, I’ve 
serious hardware problems) so I’m not able to show specific statistics 
here right now.

Implementing these plugins brings one more advantage - it forced me to 
look closer to the mechanism of chroot[7], which brought the idea to 
also add to my tool possibility to convert packages for various 
distribution releases and architectures.

* test and improve tool’s configuration
I’ve added to the repository sample example configuration file, which 
should help future users to run configure pypi2deb according to their needs.

* fix many small bugs

Compared to the TODO list set out in previous report, there was not 
enough time to:

* implement more advanced export plugins - this will be my priority 
tomorrow (allowing users to export packages to an existing Debian 
repository)
* improve conversion tools to support Python 3 packages
* improve PyPI XML-RPC methods to be more efficient for tool's needs

I plan to improve my tool even after the GSoC, it gave me a lot of joy 
and I learned a lot while working on it. I want it to be used in 
production so that all PyPI packages will be available for Debian users 
with apt-get install command.

Summary
------------
The most important project goal was achieved: PyPI2Deb is fetching 
Python projects from PyPI repository, converts them into Debian packages 
and creates Debian repository . This makes installing libraries not 
available in official Debian repositories much easier. Thanks to using 
plugin system, the tool is easily extensible and customizable. During 
the project I reported my first bugs ever... and I even provided patches :-)

Thanks
-----------
With last official report I would like to thank you for allowing me to 
participate in GSoC. My knowledge of packaging has extremely grown. I 
realize that the biggest challenge is to make it good enough to be used 
by you.

-----------
[1] http://lintian.debian.org/
[2] http://jwilk.net/software/lintian4python
[3] http://jinja.pocoo.org/
[4] https://buildd.debian.org/
[5] http://packages.debian.org/search?keywords=sbuild
[6] http://pbuilder.alioth.debian.org/
[7] http://wiki.debian.org/chroot