[pymvpa] How you run permutation test for search-light analysis?

[J.A. Etzel]

If I read that section correctly, they ran the complete searchlight 
analysis 10 times, then pooled the calculated correlation values across 
voxels to determine the significance.

This strikes me potentially troublesome if you want to use the test to 
make strong conclusions about *which* voxels have significant 
searchlights. In other words, if (in your case) you want to make a claim 
that some of the 3000 searchlights classify significantly and others 
don't, I wouldn't suggest pooling the permutation distributions over 
searchlights. It is very possible (even likely) that different 
searchlights have drastically different null distributions.

In my opinion proper statistical control (and interpretation) of 
searchlight analyses is still an open question. Parametric statistics 
(e.g. t-test for accuracy above chance) are not perfect, but at least 
are fast. If some of your 3000 searchlights are truly of more interest 
than others you could perhaps run the permutation test for those, then 
compare those p-values with the parametric ones to give you some 
confidence in the parametric ones. Something like a Mann-Whitney U test 
might also be useful if you want to avoid too much outlier instability.

Out of curiosity, why are you running non-overlapping searchlights? Are 
you testing brain parcellation schemes?

Jo


On 3/25/2012 3:56 AM, Vadim Axel wrote:
> Hi,
>
> When I run a whole-brain search-light analysis with non-overlaping
> lights (each voxel participates only in one light) I get about 3000
> lights (half a hour classification). Running 1000 times such permutation
> analysis would obviously take weeks.  And this is a limited
> non-overlapping lights case. Is there any more realistic approach? I
> found here that people ran permutation analysis only 10 times while they
> stack the predictions of all voxels together to have their chance
> distribution:
> http://cercor.oxfordjournals.org/content/early/2011/12/20/cercor.bhr357.full
> (see "Searchlight analysis" section second paragraph). Does it make
> sense for you?
>
> Thanks,
> Vadim
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa