[pymvpa] How you run permutation test for search-light analysis?

[Jonas Kaplan]

I too, wish we had a better solution to this problem. 

What we have done recently is to do permutations for a bunch of different voxels around the brain, and upon finding that the null distributions were not different from each other for those voxels, we used the threshold from the most conservative of those distributions (corrected for multiple comparisons) across the brain.  

-Jonas


----
Jonas Kaplan, Ph.D.
Research Assistant Professor
Brain & Creativity Institute
University of Southern California

On Mar 26, 2012, at 2:35 PM, J.A. Etzel wrote:

> If I read that section correctly, they ran the complete searchlight analysis 10 times, then pooled the calculated correlation values across voxels to determine the significance.
> 
> This strikes me potentially troublesome if you want to use the test to make strong conclusions about *which* voxels have significant searchlights. In other words, if (in your case) you want to make a claim that some of the 3000 searchlights classify significantly and others don't, I wouldn't suggest pooling the permutation distributions over searchlights. It is very possible (even likely) that different searchlights have drastically different null distributions.
> 
> In my opinion proper statistical control (and interpretation) of searchlight analyses is still an open question. Parametric statistics (e.g. t-test for accuracy above chance) are not perfect, but at least are fast. If some of your 3000 searchlights are truly of more interest than others you could perhaps run the permutation test for those, then compare those p-values with the parametric ones to give you some confidence in the parametric ones. Something like a Mann-Whitney U test might also be useful if you want to avoid too much outlier instability.
> 
> Out of curiosity, why are you running non-overlapping searchlights? Are you testing brain parcellation schemes?
> 
> Jo
> 
> 
> On 3/25/2012 3:56 AM, Vadim Axel wrote:
>> Hi,
>> 
>> When I run a whole-brain search-light analysis with non-overlaping
>> lights (each voxel participates only in one light) I get about 3000
>> lights (half a hour classification). Running 1000 times such permutation
>> analysis would obviously take weeks.  And this is a limited
>> non-overlapping lights case. Is there any more realistic approach? I
>> found here that people ran permutation analysis only 10 times while they
>> stack the predictions of all voxels together to have their chance
>> distribution:
>> http://cercor.oxfordjournals.org/content/early/2011/12/20/cercor.bhr357.full
>> (see "Searchlight analysis" section second paragraph). Does it make
>> sense for you?
>> 
>> Thanks,
>> Vadim
>> 
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa