Update to Speex

[Ron]

On Mon, Mar 17, 2008 at 09:25:08PM +1100, Jean-Marc Valin wrote:
> Ron a écrit :
> > So if I've got this straight, the problem here is that we never should
> > have distributed 1.1 at all in the first place, as it is not perfectly
> > compatible with either 1.0 or 1.2 ?  (and you never claimed it would be)
> 
> Distributing 1.1.x was the right thing to do. Depending on the API/ABI
> for the *new* stuff it included (noise suppression and AEC) was the
> wrong thing to do. None of the apps that worked with 1.0.x broke with
> 1.1.x or with 1.2.

hrm.  I think that changes the balance of things a little in my head ...

Looking at what I have here, I see a couple of little changes that
weren't forward compatible, the most immediately obvious being:

  -#define SPEEX_STEREO_STATE_INIT {1,.5,1,1}
  +#define SPEEX_STEREO_STATE_INIT {1,.5,1,1,0,0}

The extra fields being zero might save a few cases, but mostly I'd
expect this worked because no one is really using it yet ...
Thats from 1.1.12 to 1.2beta2, so it may not have been a problem
if we didn't count 1.1.x in the compatible release sequence.
(or maybe I have it wrong and its not a problem, I didn't follow
that deep)


The bigger issue though is that while adding all the extra code
was forward compatible, removing it again is not ...

Separating the stable from the active code is definitely the right
thing to do, but I'm not quite so sure if, having not done that
right from the start (which would have been ideal) we should 'hide'
the removal of these symbols from libspeex from the machine
information in the library.

To people we can say "you shouldn't have used those symbols, it was
even documented that you shouldn't rely on them", but for people who
didn't read that either, the machine has no way of knowing they've
made a mistake except when something fails catastrophically, shortly
before they do.

Personally I think if it was my call, this would tip the scales toward
using a technical solution.  The easy but blunt option is declare a
new beginning, bump the libspeex soname to so.2, (and knock speexdsp
back to so.0 if you like, from its current so.1, but probably better
to just leave that at 1 as people are already using it too now).

The best, but lots of hard work, option is to manage the symbol
changes with elf linker tricks, but I suspect you don't want to
melt your brain on that right now.

The messiest, but we can do it that way if we have to option,
is to use the package dependency relationships to manually specify
a conflict with every app that may have used the removed symbols.


We've been trying to figure out who has been doing what with the
other distros, to get some idea of how interoperable any solution
we come up with is likely to be, and it appears fedora at least
have been shipping all of the 1.2 betas.  Are they just relying
on the app maintainers to rebuild any apps that get broken with
the lib split?  (or are all such apps really bundling speex)

If the upgrade path was 1.0 -> 1.2beta3+, I think I'd agree not
bumping the soname was best, but 1.1 -> 1.2beta2 have been in
circulation for quite some time...  I'd wager a lot of people
you've never heard of have code using the extra functions ...

But that's more than enough from me on that, it's your call so
we'll do whatever we have to from there.


> > If all that is true, then it sounds like we should ship libspeex.so.1
> > from 1.2beta4, and split libspeexdsp into at least its own binary .deb
> > if not its own source package.
> 
> It definitely makes sense to ship libspeexdsp in a separate binary
> package. However, there are many header files in common, so splitting
> the source package would be complicated. Even for the development
> package, it might be a good idea to make the libspeexdsp-dev depend on
> libspeex-dev as there are two header files (basic type definitions) in
> common. The dev package dependency would be avoidable though.

Ok, that makes sense.  I haven't had a really good look at beta3 yet
except to see what symbols wound up in speexdsp.

> > I've been using the resampler and the preprocessor from 1.2beta2.
> > Splitting that out is an incompatible change to libspeex from that
> > release, but we can probably pretend that never happened too if it
> > will be handled right from here onward.
> 
> Well, the resampler was broken in 1.2beta2 ...

hrm, its been working ok here, though my testing hasn't really
exercised it very hard ...  I have a couple of other little things
on a list I've been keeping for you, but I should check the newer
code and get this thread out of the way first ;)

> > If Ekiga/Mumble/etc want to be part of a release, they'll just have
> > to be compatible with libspeexdspN for whatever N is the release
> > you recommend we ship in distro snapshots.  So long as you increase
> > N when you make an incompatible release, all will work as it should.
> 
> Where N is the soname or the package name suffix? I'd suggest just
> calling the package libspeexdsp0 (no matter what the soname is). BTW,
> I'm *expecting* the next API/ABI to be the final one. It's not like
> there'll be 10 more changes.

It seems like a reasonably sane habit to have the same N in the package
name as you use for the soname, but the only hard requirement is that
the runtime package name change when the soname does (so they can
coexist on the system during any library version transition).

I guess with 20/20 hindsight libspeexdsp.so.0 might have been a better
starting point, but the numbers themselves aren't particularly important
so the law of least surprise would favour libspeexdsp1 for beta3.  The
vast majority of users will expect libspeexdsp.so.1 to be provided by
that, and a good number of them will get confused if its not.

It also doesn't really matter if there are 10 more changes if they are
all justified.  It's definitely good news that you think its stabilising
nicely, but we have the tools to manage interim releases when some part
does need to be rethought again.  If we use them right we can keep
people on the bleeding edge testing them, and give them some nice loud
warnings (and a transition buffer) when they are going to have to do
some work to keep up with some new changes again (and have nice orderly
upgrades for end users too).

> > Or we need to package an svn snapshot.  But if you are happy enough with
> > the current code to at least make a beta4 snapshot, that seems like a
> > fairly easy better solution all round.
> 
> I'll think about it. When do you need an answer? (and a release?)

Sometime in the next couple of weeks would be nice.  The sooner the
better, but not if it involves rushing something you'd rather do first.
If its not really very different from beta3 I can get most of our side
done with that and get the transition started, then we can bump that
to beta4 when you have one ready.

> >> Can you please take what Debian has in unstable and release as Lenny.
> >> Then you can just resume work on the next release?
> > 
> > We can't quite do that, there are a few things in that which need to
> > be tidied, as well as updating it to beta3+ with libspeexdsp split out.
> 
> No that was just a joke. As if Debian would just release lenny tomorrow
> based on what they had.

Ahh ;)  That's sort of what some of us are aiming to do actually ...
to always have a daily distro that is release-worthy and ready to go,
rather than the big up/down RC bug spikes between releases as we have
now.  It's a curious game though, because what we're really trying to
do is synchronise thousands of other developers to all get their shit
together on the same as yet unknown day when all of their work happens
to be in a releaseable state, for just a few short minutes.

Then somebody has another Great Idea and it all falls down again ;)

There is rhythm to the noise, we just don't have enough taps to
predict it with much accuracy yet, and of course every year the
orchestra changes...

Release when you are ready is a pretty good metric.  You'll know
when we're getting ready because hordes of us will come from
no-where to bug you about things ;-)


> My hope was actually go have the API/ABI frozen by june, so it's
> realistic.

Cool, that's exactly how I like releases to work out ...

> BTW, any help on this would be appreciated :-)

Do you have a list of things in particular that need help right now?

I'm probably a few weeks away from really having my head in speexish
things again (aside from beta3 packaging, and updating my code to
suit), but I'll be happy to cherry pick anything I think I can along
the way.  We can also call on the hordes to any degree that might be
useful for particular problems ...

Cheers,
Ron