Atlas proposal

[Samuel Thibault]

Don Armstrong, le Tue 17 Aug 2010 17:24:05 -0700, a écrit :
> On Wed, 18 Aug 2010, Samuel Thibault wrote:
> > Don Armstrong, le Tue 17 Aug 2010 15:13:15 -0700, a écrit :
> > > All of these are things that can be detected at run time and
> > > appropriate libraries dlopened or codepaths diverged, etc.
> > 
> > Errr, then you'll need a myriad of libraries/codepaths for all the
> > combinations of L1/L2/L3 cache sizes, number of processors, speed,
> > etc. etc.
> 
> You've got them already;

Yes. Each machine builds its own combination.

> you either select them during build-time, or
> you select them during run-time.

But to select them during run-time, you need to build and upload _all_
combinations, which as I said is a myriad when you see the parameters
that Atlas takes into account.

> Optimization isn't just about using all of
> the hardware available on a machine maximally by one subset of the
> problem;

Well, talk with upstream, because it's what they consider.

> it's about maximizing the throughput of a particular problem,
> which may mean that atlas shouldn't use all of the cache, or shouldn't
> use as many cores as exist on a particular machine, etc.

That's way harder to do in an optimized way, and that's probably why
Atlas doesn't do it.

> > > But all of that is fine; we can't possibly hope to optimize to get the
> > > last iota of performance out of a system. We should attempt to provide
> > > a reasonable set of optimized binaries (whether that means one or ten
> > > is up to the package maintainer),
> > 
> > The problem is that currently the Atlas build system doesn't have
> > any way to do generic optimization, and not agressive L1/L2/L3 cache
> > size -related optimizations which will actually make performance
> > quite worse whenever running on a machine with a smaller L2 for
> > instance.
> 
> Oh, I've no doubt that there are serious design issues that need to be
> addressed by Atlas upstream, and that we may have to live with a
> suboptimal solution. We just should make sure that the packages that
> we distribute work as well as we can, and provide good documentation
> (or an -auto package?) so that people who need/want an optimized build
> can make one.

I agree on this.  What we still don't agree on is whether you can build
an optimized package at all, since Atlas will optimize it for the
machine where it got built, and the optimizations it does will
potentially make performance worse on another machine...

Samuel