[pymvpa] How you run permutation test for search-light analysis?

J.A. Etzel jetzel at artsci.wustl.edu
Tue Mar 27 14:08:52 UTC 2012


On 3/27/2012 5:07 AM, Vadim Axel wrote:
> Thanks for your answer. I agree with your concerns (this is why I
> did not take their approach for granted). I just did not understand,
> how you run t-test for within subject search-light analysis?

The t-test I was thinking of (not the only option, of course) is done by
testing if the subjects' accuracies are greater than chance. In other
words, when running a within-subjects analysis an accuracy is calculated 
for each person and each searchlight, then those accuracies are tested 
against chance. The cross-validation folds don't directly go into the 
statistical test, just the final accuracy for each person.


> My non-overlapping analysis was just for making a fast test and to
> save time (the minimal number of lights which cover all brain).
> Clearly, the real analysis cover with overlap and then averaged
> across results for different lights.

Oh, so the non-overlapping analysis is for testing. I thought maybe you
were trying a new analysis scheme. :)

Jo


> On Mon, Mar 26, 2012 at 11:35 PM, J.A. Etzel
> <jetzel at artsci.wustl.edu <mailto:jetzel at artsci.wustl.edu>> wrote:
>
> If I read that section correctly, they ran the complete searchlight
> analysis 10 times, then pooled the calculated correlation values
> across voxels to determine the significance.
>
> This strikes me potentially troublesome if you want to use the test
> to make strong conclusions about *which* voxels have significant
> searchlights. In other words, if (in your case) you want to make a
> claim that some of the 3000 searchlights classify significantly and
> others don't, I wouldn't suggest pooling the permutation
> distributions over searchlights. It is very possible (even likely)
> that different searchlights have drastically different null
> distributions.
>
> In my opinion proper statistical control (and interpretation) of
> searchlight analyses is still an open question. Parametric
> statistics (e.g. t-test for accuracy above chance) are not perfect,
> but at least are fast. If some of your 3000 searchlights are truly of
> more interest than others you could perhaps run the permutation test
> for those, then compare those p-values with the parametric ones to
> give you some confidence in the parametric ones. Something like a
> Mann-Whitney U test might also be useful if you want to avoid too
> much outlier instability.
>
> Out of curiosity, why are you running non-overlapping searchlights?
> Are you testing brain parcellation schemes?
>
> Jo
>
>
>
> On 3/25/2012 3:56 AM, Vadim Axel wrote:
>
> Hi,
>
> When I run a whole-brain search-light analysis with non-overlaping
> lights (each voxel participates only in one light) I get about 3000
> lights (half a hour classification). Running 1000 times such
> permutation analysis would obviously take weeks.  And this is a
> limited non-overlapping lights case. Is there any more realistic
> approach? I found here that people ran permutation analysis only 10
> times while they stack the predictions of all voxels together to
> have their chance distribution:
> http://cercor.oxfordjournals.__org/content/early/2011/12/20/__cercor.bhr357.full
>
>
>
<http://cercor.oxfordjournals.org/content/early/2011/12/20/cercor.bhr357.full>
> (see "Searchlight analysis" section second paragraph). Does it make
> sense for you?
>
> Thanks, Vadim
>
> _________________________________________________ Pkg-ExpPsy-PyMVPA
> mailing list Pkg-ExpPsy-PyMVPA at lists.__alioth.debian.org
> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
> http://lists.alioth.debian.__org/cgi-bin/mailman/listinfo/__pkg-exppsy-pymvpa
>
>
>
<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
>
> _________________________________________________ Pkg-ExpPsy-PyMVPA
> mailing list Pkg-ExpPsy-PyMVPA at lists.__alioth.debian.org
> <mailto:Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
> http://lists.alioth.debian.__org/cgi-bin/mailman/listinfo/__pkg-exppsy-pymvpa
>
>
>
<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
>



More information about the Pkg-ExpPsy-PyMVPA mailing list