[pymvpa] search-light vs. ROI analysis: significance puzzle

[Vadim Axel]

Thanks  a lot for the prompt reply!

I absolutely agree  - clearly the variability between subjects makes a lot
of sense. I select my ROI based on functional criteria, which means that
(presumably) I take  the voxels which process relevant information. I never
get the overlap of these ROIs across subjects in search-light. I did not
bring this point originally in order to simplify my question - I meant the
ideal case where  I have 10 identical twins in my group-level, with a
complete ROI overlap :) But, even in this case I have a p-value issue which
does not permit me to achieve significance in search-light.

BTW, why one should use a search-light which is substantially smaller than
ROI? I deliberately make it of size of my ROI  to make results more
comparable between two analyses. I move my search-light each time one voxel
in one of the directions, so each voxel participates in dozens of lights.
At the end, I average all the predictions in each voxel.

I would appreciate if you notify me once your paper is in press.

On Mon, Feb 25, 2013 at 11:41 AM, J.A. Etzel <jetzel at artsci.wustl.edu>wrote:

> On 2/25/2013 1:10 PM, Vadim Axel wrote:
>
>> Absolutely naive question: suppose I have single a-priori defined ROI
>> where I get a modest group-level beyond chance prediction of
>> p-value=0.01 (one-tail t-test vs. 0.5, across subjects). Now I run a
>> group level whole-brain search-light and I am expected to find at least
>> one cluster of beyond chance prediction in the environment of my ROI.
>> Correct?
>>
> No.
>
> I have a paper in (hopefully the last cycle of) review that goes into
> detail about these issues. But here's a brief version of some of the
> relevant ideas. I'm assuming you're using a linear SVM and proper
> cross-validation, and also that the searchlight is substantially smaller
> than the ROI.
>
> Two possible explanations come to mind:
> 1) The single searchlights are too small to hold enough voxels to classify
> accurately, but the ROI can, because there is weak information present in
> much of the ROI. Linear SVMs can combine weak information from many voxels,
> so can sometimes classify better with more voxels.
>
> 2) There is a lot of spatial variability between subjects. Suppose only a
> small part of the ROI is informative. If that part falls withing the ROI
> for everyone, then the ROI might classify well at the group level. But if
> each person only has a small informative area on their searchlight map, the
> group map could come out non-significant (people's maps don't overlap
> enough).
>
>
> A few suggestions:
> 1) If your hypothesis is about the ROI, stick with the ROI-based analysis,
> adding control ROIs (or whatever) as necessary, but not doing the
> searchlight analysis.
>
> 2) If you need the searchlight analysis for a particular purpose, do some
> sensitivity testing, and look closely at the single-subject maps. For
> example, how much do the maps change with different searchlight radii? Did
> you normalize to atlas space before or after the searchlight? Did you
> smooth the data? Smooth the individual subject maps? etc.
>
> 3) Check the sensitivity of the ROI-based finding. For example, How much
> does it change if the ROI boundaries are altered slightly? How much
> variation is there between subjects - does the ROI classify well in most
> everyone, or just a few people?
>
>
> Hope this gets you started, and good luck.
> Jo
>
>
>
> --
> Joset A. Etzel, Ph.D.
> Research Analyst
> Cognitive Control & Psychopathology Lab
> Washington University in St. Louis
> http://mvpa.blogspot.com/
>
> ______________________________**_________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.**alioth.debian.org<Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**
> pkg-exppsy-pymvpa<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130225/8b0c829d/attachment.html>