[Soc-coordination] Pluggable Acquire System for APT - Report 1

[Bogdan Purcareata]

Codename: apt-fetcher
Mentors: Michael Vogt, David Kalnischkies

Synopsis: The Debian Archive currently stores different types of
package metadata, which is handled by individual package management
applications. While this approach is currently working, it's not
scalable, as different types of metadata will be added to the Archive
e.g. the AppStream project.

Purpose: To unify the metadata acquire process and make it part of
APT's functionality. This way, separate package management
applications - e.g. apt-file, debtags - will only have to register
their specific acquire plugin in APT, and then make use of the fetched
package metadata, acquired through "apt-get update". One of the side
purposes of the project is to provide a public parser for the
sources.list file, that is exported from libapt. More details on the
project proposal page: [0].

In the community bonding period I've mostly been doing research and
browsing APT's source code. APT is one of Debian's fundamental
components and it was a great thrill seeing how all its components
work together. It's probably one of Debian's most tested packages, and
its developers' expertise shows in the quality of the code. Working
with this code is a great challenge to me. I haven't talked that much
with my mentors, as it was my finals period at uni - I've done most of
the work on my own.

At the beginning of the coding period I started a wiki page, to
present the proposed design as it was slowly evolving and have a
written base for the discussions with the mentors: [1]. The first
issue I had in mind was a way to enhance the sources.list file(s) with
additional metadata information, for the future developed plugins,
while keeping backwards compatibility. I've studied the sources.list
man page, and I saw that the format supports, for each entry, an
optional number of options, which can be silently ignored if not
understood. David suggested using this insertion point for the
enhancements, rather than defining new type of entry - my initial
idea. Once I got this figured, it was easy to see how most of the
project will come together in the future: the public parser will
provide the sources.list info in a structured way to the pluggable
acquire framework, which will decide which plugin to use for metadata
fetching, based on the entry type and defined options. Metadata types
can be activated for each entry through boolean values passed as
options.

The first task I had to work at, as suggested by the mentors, was the
public parser. David suggested I should focus on different types of
iteration through the metadata sources, and Michael advised me to take
into account future compatibility with different formats for the
sources.list file(s) - changing the format of these files should not
result into refactoring the whole parser. Having these thoughts in
mind, and a parser interface, I started working on predicate based
sources iteration. After this, I made it possible to provide a read
method for the public parser - this way, new types of sources can be
supported, such as apt-rpm sources or rfc822 sources. I've made the
parser sort of "pluggable" as well. Implementing the standard source
format support was the next step - this was mostly porting the code
from the existing APT package.

I've had a little trouble getting used to C++ again - I haven't used
it that much for the last 2 years. This slowed the development process
for a bit, especially when it came to compiling and testing what I've
done. But I managed to do that eventually, I've written a few tests as
well, and it works as expected! I was very excited when my first test
passed, and I already saw myself on the way of building code that will
be used by millions. After some time doing research on my own, trying
to figure out what's there and how it can be improved, this was a
great reward. The parser is far from being complete: I must refactor
it taking into account forward ABI compatibility (as suggested by
Michael), test it thoroughly, and who know what else might come up on
the way. At this point, I've done some of the project design and I've
brought a component in a functional state.

To sum up, here are the first two weeks' summary (and, more or less,
the community bonding period):

What I've done:
- configured the build environment, the project repo [2], and the
desing page wiki [1]
- refreshed my C++ skills
- designed the public sources.list parser and (partly) the pluggable
acquire framework (all though I haven't written it down yet)
- implemented the parser and a few tests for it
What problems I've run into:
- understanding the present APT code - not really a problem, but I
can't say it was easy doing it
- "getting back in touch" with C++ - bugs, bugs, bugs
- designing is hard when you don't fully understand how a complex project works
What I plan to do further:
- think of some more tests for the parser and refactor it to be
forward ABI compatible
- design and implement the next component: the pluggable acquire
framework - interface, generic plugin model, etc.

As a conclusion, starting a project is the hardest part, and this
applied to apt-fetcher. Fortunately, Michael and David had the right
piece of advice every time I asked them for it, and their
straightforward guildelines made development so much easier - thanks a
lot! I'm really anxious to see what this project will evolve into.

My contributions to the APT package can be found in the repo [2]: the
header file [3], the implementation file [4] and the tester [5].

Bogdan Purcareata

[0] http://wiki.debian.org/SummerOfCode2012/Projects#Pluggable_acquire-system_for_APT
[1] http://wiki.debian.org/BogdanPurcareata/PluggableAptBackend
[2] https://launchpad.net/apt-fetcher
[3] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.h
[4] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/apt-pkg/parser.cc
[5] http://bazaar.launchpad.net/~bogdan-purcareata/apt-fetcher/trunk/view/head:/test/parser/parser_tester.cc