[pymvpa] sphere_gnbsearchlight & Monte Carlo Testing

[Yaroslav Halchenko]

On Tue, 23 Oct 2012, wolf zinke wrote:
> >Meanwhile, on your data, could you confirm that there is some variance
> >in the obtained performances in a "good/lengthy" code, i.e. that values

> >np.std(distr_est.ca.dist_samples.samples[0], axis=1)
> Thanks for the reply. Indeed, using a sphere_searchlight with the
> Monte Carlo Testing, this gives me zeros only.

This is a really nice one, i.e. the bug which is there "by design" so
not that obvious from a first sight ;)  Also good that it should not be
relevant for 99% of other users since the constructs in this case
are elaborate enough to not be too common ;)

The reason for the observed abnormal behavior is the nproc > 1,
when searchlights are parallelized.  Then every child process acquires
"a copy of numpy", including the state of its random number generator
(RNG).  That results in identical initial states of RNG in every one of
those processes, leading to the same permutations, thus degenerate
results in your case.

quick resolution for you:  set nproc=1

longer resolution for us:

  we would need to sacrifice "exact reproducibility of results with a
  single global seeding of RNG", i.e. mvpa2.seed() (or via MVPA_SEED
  environment variable).  Every Permutator in this case should get
  its RNG randomly seeded (i.e. we should just use a new
  numpy.random.RandomState() in every one of them in _call, unless
  instructed explicitly otherwise).

  I have pushed an initial fix (only for Permutator, which is what
  important for your case here).  More (I see at least a few of other
  similar spots) to follow + feeding rng in some of the tests

-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik