[pymvpa] mysterious whole-brain decoding

[Etzel, Jo]

In addition to Yarik's comments - are the classes (movement types) 
balanced within each run (assuming you're cross-validating on the runs)? 
Small imbalances (e.g., more of this type of movement than that in run 
1) can make a massive difference in classification.

I also suggest not to do significance testing until you're pretty 
confident that the bugs are out of the single-subject results - e.g., 
that the single-subject maps have plausible patterns in most people. 
That might be something like accuracy > 0.35 in motor/somatosensory 
areas in this case. If each subject has this sort of pattern, with 
approx. chance accuracy in other areas of the brain, then something 
clearly went wrong in the group-level analysis. A voxel-wise t-test can 
also help for troubleshooting.

For motion - have you plotted the 6 realignment parameters (x, y, z, 
roll, pitch, yaw) plus event timings for each person and looked for 
correspondences? (these aren't task motion, but are the sorts of plots I 
mean: 
http://mvpa.blogspot.com/2017/09/yet-more-with-respiration-and-motion.html) 
It can also be useful to make similar plots but snipped to each event 
onset, all trials of the same type overplotted: if you have clear 
task-linked motion (whether actual or apparent/respiration-linked) you 
will see consistent curve shapes.

Jo


On 1/21/2020 3:16 PM, yoh at onerussian.com wrote:
> 
> On Tue, 21 Jan 2020, Floris van Vugt wrote:
>> ...
>> **Shortest version**
>> What may I be doing wrong if I my group-level significance maps for
>> searchlight decoding cover the entire brain?
> 
>> **Slightly longer version**
>> I ran a movement study in fMRI and I am decoding on a subject-level basis
>> which of a set of 4 movements was performed on each trial.
> 
> and that is where IMHO the likely source of your trouble is: a physical
> movement was performed by subjects.  Depending on the type of movement
> (one thing is finger tapping; moving different arms or legs would
> be a different story), such movement is likely to be reflected in the
> "head movement".   Movement introduces non-linear effects into your
> data.  "Motion correction" is doing its best to eliminate some effects
> of motion (ideally not correlated with the experiment, which is the case
> here), but simply cannot remove it entirely for a variety of reasons.
> So, motion corrected data will reflect motion.  That is why
> resting-state data analysis typically includes "scrubbing", where
> periods of notable motion are simply "removed" or some other way
> "imputed".
> 
> So what the physical motion in your experiment?
> 
> Was motion correlated with the design?:
> Before anything, I would have just trained a  classifier on the motion
> estimate parameters (translation, rotation, may be with mean displaement
> added as yet another featyre) instead of fmri data itself.  If,
> across subjects, you have "significant" decoding performance -- it would
> mean that your fmri data would have those confounds in it as well, may be
> represented more weekly after motion correction, but still there.  That
> could explain your "whole brain" decoding.  If you fail to achieve
> decoding on motion correction parameters,  it is a good sign but not a
> guarantee that data is not reflecting actual physical motion.
> 
> Now some comments inline:
> 
>> decode which movement they did. I use off-the-shelf cross-validated
>> spherical searchlight decoding with SVM on the subject level, yielding maps
>> that to me look pretty decent, with decoding close to theoretical chance
>> level.
> 
> if there is consistent (across subjects) skew in errors to "good
> side", then those little effects would be your 'significant whole brain
> cluster'.
> 
>> Then I generate group-level significance levels using the Stelzer
>> method (GroupClusterThreshold), i.e. I re-run decoding on subject-level
>> datasets with permuted labels, and then sample from those to create a
> 
> here also -- permutation could be of a tricky business depending on your
> design.  What was it -- block vs single trial?  completely randomized across
> subjects or used some pregenerated sequences and swapped them across
> runs?
> 
>> the entire brain. That is, each and every voxel is deemed to significantly
>> decode the movement that subjects make.
>> See the linked glass brain image, in which the decoding accuracy is shown
>> for each voxel: https://drive.google.com/file/d/1Lb6e1qZlKDcmu5WDAV84McRKDi1ITL4P/view?usp=sharing
> 
> would be interesting to see a colorbar and/or distribution of values.  I
> guess chance level should be 0.25 (since 4 types of movements)
> 
>> I can also submit scripts but it's a bit complex because it's part of a
>> large pipeline.
>> I felt maybe some of you off the top of your head can already indicate what
>> might be going wrong here or maybe somebody has experienced something
>> similar.
> 
> see above
> 
>> If you want an actual reproducible pipeline then I can generate that, I just
>> wanted to see if I could spare myself that effort :)
> 
> well... if you can achieve that -- it would be of benefit to yourself,
> and your future publication!
> 
> NB shameless plug -- you might like to familiarize yourself with another
> project which we have been developing for the past decade pretty much --
> http://datalad.org . datalad run   would be your friend here to "commit"
> a record of your reproducible step.  datalad-container extension could
> also come handy to guarantee that you have the computing environment
> anywhere you go now, or come back to it all later on ;)
> 
>> So at this point, feel free to respond: "need more details, send code" and I
>> will do that.
> 
> "need more details", see above questions.  others might like to see code
> as well, but I am afraid the devil is in the data.
>