[pymvpa] Balancing with searchlight and statistical issues.

[Jo Etzel]

You don't want to try something like Stelzer's method (or any 
statistical test, really) until you're sure the single subject analyses 
are sensible. Do the actual classification accuracy maps for each person 
look reasonable? If every searchlight is classifying at 100% accuracy, 
something is obviously wrong with the analysis code and you should fix 
that first.

It's usually good to start by classifying something that should be a 
strong signal in easily-predicted areas (e.g., which response button was 
pushed, which should have signal in motor areas). Then you can make sure 
your cross-validation scheme, event coding, etc. is set up properly 
before trying your actual analysis.

Also, you say the dataset is unbalanced, but has 12 runs, each with 10 
trials, half A and half B. That sounds balanced to me ...

Jo



On 2/25/2016 9:43 AM, Roberto Guidotti wrote:
> Hi all mvpaers,
>
> I need some theoretical help!
>
> I did some analysis on a unbalanced dataset, 12 runs with 10 trials (5
> condition A, 5 condition B), so I got 120 samples. Since I had an
> unbalanced dataset, I could have a run with 7A vs 3B or also a 9A vs 1B
> samples and/or viceversa.
> I analyzed the dataset balancing samples in each run, using a leave TWO
> run cross-validation (L2ROCV) searchlight, in order to have more
> combination of samples to train the dataset and the same for the testing
> set, I didn't analyze the dataset using different balancing since the
> searchlight in a L2ROCV is high time consuming and I had 25 subjects
> with 3 unbalanced dataset per subject!! :\
>
> Now, my questions are these:
>
> 1) I used a good approach to analyze the dataset or you suggest a
> different approach?
>
> 2) I did an average map of the 66 cross-validation map I obtained for
> each subject; to do a first exploratory analysis I did a simple t-test
> versus chance level (I didn't do the Stelzer's method because of the
> computational time) and I had almost all voxels significative (not
> corrected), because of the L2ROCV, I think. So do you think I can do
> other more robust statistical tests using these maps? Or I have to do
> the Stelzer's method? Or throw away the searchlight maps?
>
> Thank you!!
> Roberto
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>

-- 
Joset A. Etzel, Ph.D.
Research Analyst
Cognitive Control & Psychopathology Lab
Washington University in St. Louis
http://mvpa.blogspot.com/