<div dir="ltr">Thanks Yarik, for the ideas! In fact, it is not that I am interested too much in this particular case. I was curious in principle what is the reason for that. So, as far I understand from I deal with a suboptimal classification. I thought that maybe cross-validation is a more conservative way to evaluate the difference.<div><br></div><div>Per your suggestions:</div><div>1. LDA gives similar to SVM result (low prediction)</div><div>2. I make z-scoring of the data (since it is one dimension, there only one possibility). But when I drop any preprocessing the prediction is even worse.</div><div>3. I make classification of the raw volumes (no GLM).</div><div>4. t-test is significant when I just compare the data that go into classification. In other words, for each subject I save one data point per condition and then do t-test between conditions at group level.</div><div><br></div><div>I am using matlab, so sharing the code will be messy.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 1, 2017 at 4:49 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
On Wed, 01 Feb 2017, Vadim Axelrod wrote:<br>
<br>
> I have encountered a rather peculiar scenario. My experiment consists of<br>
> two conditions and I have ~40 subjects. A standard group-level<br>
> random-effects analysis t-contrast resulted in a very significant cluster<br>
> (p<0.001, cluster size correction p<0.05). Now, for this cluster and for<br>
> the same data for each subject I run SVM classification between two<br>
> conditions (leave one session out cross-validation). Classification is done<br>
> with one dimension (average response of ROI). So, for each subject I get a<br>
> hit-rate which I submit to group t-test vs. 0.5. The surprising thing is<br>
> that even despite a clear double-dipping, I fail to get beyond chance<br>
> significant decoding in this cluster (hit-rate is only ~0.51-0.52). How<br>
> this can be? Because classification does not care about direction of a<br>
> difference, I thought that it should be always more sensitive than a<br>
> directional comparison of activations. Thinking and simulating lead me to<br>
> think that it is probably not a bug. Consider an extreme case, that in<br>
> every one of my subjects condition_1 is slightly above condtion_2. Group<br>
> t-test will show highly significant difference. But in individual<br>
> classification, my prediction will fluctuate around 0.5 with a slight above<br>
> 0.5 bias, that would not be enough to reach significance above 0.5 at group<br>
> level. Indeed, in case of my ROI, for 90% of the subjects the difference<br>
> was in one direction. Does all this make sense to you? If so, what does it<br>
> tell us about reliability of a standard random-effects group level analysis.<br>
<br>
</span>There could be multiple reasons I guess:<br>
<br>
- choice of the classifier.<br>
<br>
SVM is a non-parametric (doesn't have a model of the distributions) thus<br>
somewhat more flexible and might be the least sensitive to univariate<br>
effects... To match closer assumptions of univariate GLM/t-test<br>
you could try GNB (on that one feature probably should be similar to<br>
LDA?) or mean-one-nearest-neighbor (probably the best detector if your<br>
signal is normally distributed with equal variance), and see if you get<br>
more consistent "significant" excursion.<br>
<br>
- Ideally then you should also get an effect in "random effects" analysis<br>
while classifying across subjects (since you are using just mean of ROI<br>
within subject, you could easily carry it out across subjects).<br>
<br>
- another related question is -- do you see significant t-test result when<br>
doing t-test against 0 on that difference between conditions on average<br>
ROI? may be somehow averaging also annihilates the effect<br>
<br>
- also preprocessing:<br>
- what do you take for classification (per trial betas or ...?)<br>
- how you z-score (if you do) within each subject<br>
might introduce noise across your samples for classification (e.g.<br>
if you had just few samples for each condition within each run and<br>
zscored within each run based on those only few samples)<br>
<br>
So the best might be -- if you share a dataset with the extract of<br>
that ROI before doing any pre-processing and then complete analysis<br>
script -- then interested parties could play with it to determine the<br>
underlying culprit.<br>
<span class="m_8160338005093617685HOEnZb"><font color="#888888"><br>
--<br>
Yaroslav O. Halchenko<br>
Center for Open Neuroscience <a href="http://centerforopenneuroscience.org" rel="noreferrer" target="_blank">http://centerforopenneuroscie<wbr>nce.org</a><br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834" target="_blank">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419" target="_blank">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" rel="noreferrer" target="_blank">http://www.linkedin.com/in/ya<wbr>rik</a><br>
<br>
______________________________<wbr>_________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth<wbr>.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org<wbr>/cgi-bin/mailman/listinfo/pkg-<wbr>exppsy-pymvpa</a><br>
</font></span></blockquote></div><br></div></div>