[pymvpa] effect size (in lieu of zscore)

[Mike E. Klein]

Hi Jo,

I'd certainly like to take a look at your paper!

This whole business is beginning to remind of me a math teacher I had in
junior high school. He would give 4-choice multiple choice exams, but claim
that if anyone actually got a 0% on a test, he'd give them a perfect mark!

Yes, this is my first MVPA on this data. We're running a searchlight as we
expect local data patterns in a couple of different brain regions. Also, we
see searchlight as a quasi "hybrid" between non-searchlight MVPA and a GLM,
which I think makes certain members of my lab a bit more comfortable with
the technique (at least for a first-time analysis).

I have toyed with a bit of ROI MVPA: found some accuracies that were
above-chance, though I'm not sure if they were convincingly so. You're
suggesting that it should run an analysis with permuted labels on, for
example A1 and another area, and then look at the distribution of the
accuracies?

Also, to update on my weird results:

- This doesn't seem to be due to averaging... I just reran with some
non-averaged data and still get the slightly-negative shift.
- I'm as confident as I can be that this isn't a balancing issue... I'm
running these analyses after stripping away all but 2 conditions. Each of
these conditions have an identical N in each of the experimental runs.
- I'm currently running without averaging *and* without any z-scoring,
although I think I've seen this shift sans zscore, so expect it to still be
there.
- I'm wondering now if poly_detrend could be the culprit somehow. I used:
poly_detrend(dataset, polyord=1, chunks_attr='chunks') .
- My pre-pymvpa preprocessing is pretty normal, I think. I ran mnc2nii on
the data, used fslsplit to get the files into 3d, motion-corrected in SPM,
and used fslmerge to get the data back in 4d for python.

Thanks again!

Best,
Mike


On Tue, Jan 3, 2012 at 3:43 PM, J.A. Etzel <jetzel at artsci.wustl.edu> wrote:

> Ah, rather different problem than I'd thought. Below-chance accuracies are
> a big problem with fMRI data ... sometimes they happen when data is poorly
> fitted (e.g. improper scaling), sometimes with mistakes (e.g. mislabeled
> cases, unbalanced training data), sometimes for no apparent reason.
> Interpreting these is an open issue in MVPA, I'd say.
>
> Is this your first MVPA with this data? Searchlight analysis can be good
> for identifying very localized information but cause big troubles in other
> cases (I have a paper coming out in the MLINI proceedings about some of
> these issues and can send you a draft version if you'd like), is very
> sensitive to processing choices (normalization, etc), and can be very
> unwieldy to troubleshoot.
>
> I think you said this is sound stimuli? Perhaps you could set up a
> ROI-based analyses with positive and negative controls (e.g. auditory
> cortex if that should classify for sure and something like primary motor if
> that should not classify). Running a ROI-based analysis is a lot quicker
> than a searchlight, and should let you test the impact of things like
> averaging trials together. Once that's making sensible results you can go
> back to a searchlight analysis if that's what you really need.
>
> As a side note, I've found averaging trials together to be quite useful
> sometimes, particularly for between-subjects analyses. But you need to be
> very careful when averaging only several trials from a run - trials from
> the same run will pretty much always be more similar to each other than
> trials averaged from other runs, so can bias the results (e.g. if one of
> the averaged-trials from run #2 is in the training and another is in the
> testing it might classify better than when all averaged-trials from run #2
> are in the testing). I would either average to one per run or create
> partitions so that these sorts of splits can't happen (e.g.
> leave-one-run-out partitioning).
>
> Jo
>
>
>
> On 1/3/2012 2:07 PM, Mike E. Klein wrote:
>
>> Hi Jonas and Jo,
>>
>> Thanks for helping out with this!
>>
>> So:
>>
>> (1) I haven't done a permutation test. By "chance distribution" I just
>> meant the bulk of the data points using my real-label-coded data. While
>> I'm obviously hoping for a histogram that contains a positive skew, /at
>> worst/ I'd expect a normal distribution centered around chance. Once I
>>
>> get this error figured out, I will do some permutation testing as well,
>> but at the moment it doesn't seem necessary. (In other words, with real
>> data or fake data, I can't see why I'd ever see a /negative/ skew unless
>>
>> I'm doing something else wrong.)
>>
>> (2) I've generally been doing 3-to-1 averaging of my trials because of
>> time/processing limitations; because python seems to choke on a
>> searchlight analysis using LinearCSVMC if I don't first perform
>> run-averaging; and because I'm concerned about the cleanliness of my
>> data (sparse-sampling images). I'm re-running one of these analyses
>> using LinearNuSVMC and without any run-averaging, but my hunch is that
>> this isn't the problem. PyMVPA, after run-averaging, is showing 2 labels
>> with 27 examples each, which I what I was expecting.
>>
>> (3) This seems to be an issue with multiple subjects...it might in fact
>> be a universal problem. I saw it sporadically before, but then it seems
>> my preprocessing was incorrect (I was zscoring with only 11 samples in
>> mini-runs, instead of the 30+ I had intended).
>>
>> (4) My distribution is centered on 0 and goes from -50 to +50 because,
>> after the searchlight analysis I'm using:
>>
>> #set up searchlight
>> sl = sphere_searchlight(cv, radius=3, space='voxel_indices',
>> center_ids=center_ids, postproc=mean_sample())
>> #run searchlight on ds
>> s1_map = sl(dataset)
>> #convert into percent-wise accuracies centered around zero
>> s1_map.samples *= -1
>> s1_map.samples += 1
>> s1_map.samples *= 100
>> s1_map.samples -= 50
>>
>> -Mike
>>
>
>
> ______________________________**_________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.**alioth.debian.org<Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org>
> http://lists.alioth.debian.**org/cgi-bin/mailman/listinfo/**
> pkg-exppsy-pymvpa<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20120104/32ab8de6/attachment.html>