[Aptitude-devel] Acquire considered non-threadsafe

[Julian Andres Klode]

On Thu, Nov 19, 2009 at 04:31:13AM -0800, Daniel Burrows wrote:
> On Wed, Nov 18, 2009 at 04:03:05PM +0100, Julian Andres Klode <jak at debian.org> was heard to say:
> > On Sun, Nov 15, 2009 at 12:42:38PM -0800, Daniel Burrows wrote:
> > >   There are some other very nice options involving rewriting or
> > > reimplementing the "download queue" logic (i.e., just pkgAcquire), but
> > > getting threadsafety does require changing the behavior of
> > > pkgAcquire::Item in any event.  I'm assuming that we don't want to have
> > > two parallel implementations of this functionality, and that
> > > non-backwards-compatible changes are considered unacceptable, in which
> > > case the opt-in approach is the only solution I can think of.
> > 
> > We should really start talking about APT 0.8 sometime soon, and break
> > some APIs. The problem I have with Acquire is its low speed when queuing
> > items which is probably caused by the use of a linked list (20000 items
> > take multiple minutes to be queued, whereas APT2 queues them in less than
> > a second[1]).
> 
>   Acquire is also really painful to develop against, although that's
> more just the general pain of dealing with apt.
> 
>   Also, apt2?  When did that happen?
APT2 will be a package management library + applications written in Vala,
but at the moment it's just a few parsers and a simple acquire system. And
it's intented to be cross-distro, just like PackageKit. I just had no better
name when I started it. It's currently MIT-licensed, but may switch to the
LGPL-2.1+ soon (as this is what all its dependencies like glib use).

Code is at http://git.debian.org/?p=users/jak/apt2.git;a=summary, but it
is slightly outdated compared to my local branch (which is LGPL-2.1+
licensed).


> 
> > 	(b) a new cache format which can be resized in reality
> 
>   I wonder if apt really needs an mmap'd cache here.  In fact, I wonder
> about using sqlite -- I did a little experimentation with it and got
> decent performance.  The big problem (from my point of view) would be
> teaching client code to treat the database transactionally rather than
> as a linked set of in-memory objects, but it way more cleaner and more
> flexible than what we have now.
Working with databases is a bit complicated. It makes development not
really convenient.

> 
> > 	(c) new dependency resolver [porting aptitude's one?]
> > 	(d) external dependency resolvers
> > 	(e) new buildsystem (e.g. raw autotools) and reworked
> >             packaging
> 
>   Those sound very good.
I already have a CMake buildsystem in the works which can already build
apt-pkg and apt-inst. I expect this to be able to replace the current
one and build exactly the same packages afterwards; so backwards
compatibility is preserved. I will publish a new branch soon.

I chose CMake because it does not require hundreds of files in the tree
or shipping generated files. And it also appears to be faster.

> 
> > 	(f) storing errno inside the error handling objects,
> >             for bindings (nice to have errno).
> 
>   Please, please, kill the global error queue.  This is a disaster from
> a threadsafety point of view, but also just from a general design
> perspective.  There's way too much code that does
> 
>     if(!_error->PendingError())
>       fail();
> 
>   when the error could have been caused by something entirely unrelated
> to what this code is interested in.
> 
>   Put error-handling oun either a return-code basis or an exception
> basis, but not this wird intermediate thing where random errors cause
> failure.  If you want to know about inner errors, use a logging
> framework (I prefer log4cxx).
As far as I can tell from the documentation, the APT error handling should
only be used for boolean functions. If the function has an error, it pushes
a message to the error stack and returns false. If we used it this way only,
we would have no multi-threading problems.

Exceptions are a bit tricky in C++ as they can happen unexpectedly. Google's
C++ style guide says no; so we should probably also say no.

Log4cxx would effectively cause APT binaries to be GPL-3+ licensed, as
GPL-2 is incompatible with the Apache License. This could be a problem
for GPL-2 only programs using them. But it may still be worth it.

> 
> 
>   I would add one more item:
> 
>         (g) use modern C++ coding conventions (at least for new code)
> 
>   apt was designed in 1996 or so, and it shows.  Modern C++ is a lot
> cleaner, easier to maintain and easier to write than the weird mishmash
> of C and "object-oriented programming" that apt is written in.  Also,
> a lot of modern C++ programs link against Boost, which provides a ton
> of well-designed utility libraries for stuff that you would otherwise
> have to write yourself, and that you'd end up writing badly.
> (apt-pkg/contrib/ is full of that sort of thing)

We should not forget that APT has priority important, and boost libraries
are optional. But we could switch APT to the subset of C++0x which is
supported by GCC 4.3, the version in stable. 



> 
> > But I don't think anyone wants to do this.
> 
>   Personally, I would totally love to clean up the apt code and fix
> some of my chronic annoyances with it.  I just don't have the spare
> time, and apparently neither does anyone else (except some guy who
> thinks it should be written in Perl (!?!?)). :-(
Cupt is great as a prototype and can give us a direction in terms of
features. As long as you are not running an embedded system and as long
as Perl works things are okay.


Some kind of a roadmap:

    * identify stuff we don't use anymore and remove it, eg.
      apt-pkg/vendor*, apt-inst/database*, apt-inst/deb/dpkgdb*.
    * drop compatibility for GCC < 4.3, and other stuff older
      then what is included in Debian Lenny.
    * rework the buildsystem using CMake


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.