<div dir="ltr">it seems my email never got through moderation.<br>=]<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 26, 2016 at 10:30 AM, basile pinsard <span dir="ltr"><<a href="mailto:basile.pinsard@gmail.com" target="_blank">basile.pinsard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Richard, Yaroslav,<br><br></div>Thanks for your replies.<br><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sat, Jul 23, 2016 at 9:49 AM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><br>
On Fri, 22 Jul 2016, basile pinsard wrote:<br>
<br>
> Hi PyMVPA community,<br>
<br>
> I wanted to have an advice on a problem I have using PyMVPA.<br>
> My pipeline includes a Searchlight on BOLD data, for which I used the<br>
> optimized GNBSearchlight because I plan to run ~100 permutations to perform<br>
> statistical testing and it is the only one offering reasonable processing<br>
> time (or maybe the optimized KNN).<br>
<br>
> I have 2 classes x 8 samples for each (1 sample per chunk), the partitioner<br>
> (thanks @Yaroslav) I use is:<br>
> prtnr_2fold_factpart = FactorialPartitioner(<br>
> NFoldPartitioner(cvtype=2,attr='chunks'),<br>
> attr='targets',<br>
> selection_strategy='equidistant',<br>
> count=32)<br>
> this way I repeatedly take out 2 samples of each of the 2 classes for<br>
> testing and train on the remaining 2x6 samples, 'equidistant' allows all<br>
> the samples to be tested approximately the same number of time, thus being<br>
> equally represented in the final accuracy score.<br>
<br>
> The problem is that the distribution of accuracy in searchlight map is very<br>
> wide with significantly below-chance classification, and the results are<br>
> very variable across scans/subjects.<br>
<br>
</span>first let's make sure that we we looking at the correct thing -- default<br>
errorfx=mean_mismatch_error i.e. it is an error (lower is better ;) ),<br>
not accuracy<br></blockquote><div><br></div></div></div><div>yes the measure is mean_match_accuracy, so higher is better<br> <br></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span><br>
> So what I did to check if there was any problem in the analysis was to<br>
> replace my BOLD signal with random data from normal distribution, thus<br>
> removing any potential temporal dependency (even if the design was using<br>
> DeBruijn cycles for balancing carry-over effects) that could also result<br>
> from GLM (GLM-LSS, Mumford 2012), detrending or else.<br>
<br>
> Results, I get some accuracy from ~10% to ~90%, far below above chance<br>
> expected by normal approximation to binomial distribution (25-75%).<br>
<br>
</span>as already mentioned, parametric assumption might not apply here since<br>
trials might be all dependent and depending on what distribution you are<br>
looking at. e.g. if looking at distribution of results across<br>
searchlights, not across random data samples for the same dataset/roi,<br>
you should expect lots of dependencies across different spatial<br>
locations. Even if all data is randomly generated, neighboring<br>
searchlights would use the same voxels... although should matter less in<br>
case of random data.<br></blockquote><div> </div></span><div>Yes, random data no have longer temporal dependency, nor spatial dependency except for the radius of the searchlight causing the voxel being used multiple times.<br></div><div>The way I interpret this is that with 2 classes x 8 samples by drawing random voxel data a large number of time, it can happen by chance that their values are correlated to targets.<br></div><div>By using this voxel multiple times in neighboring searchlight, this accuracy is repeated, creating blobs of high accuracy, and inflate the accuracy distribution tails. <br></div><div>Certainly, having more classes and/or more samples, random structured noise is less likely to appear, thus random accuracy distribution should be lower.<br></div><div><br></div><div>It seems to depend on the classifier, some of them allow one feature/voxel to outweight the other.<br></div><div>But the high classification on random data can be obtained also with LinearCSVMC and LDA, but not at the same location.<br>Using LDA at a searchlight where GNB found spuriously high accuracy, it often results around chance accuracy.<br>However
I modified LDA code to add regularization that is often performed in
MVPA (adding a percentage of the trace to the diagonal of cov matrix,
allowing more independence between voxels), then it again produces high
accuracy value.<br><br></div><div>I also ran a small simulation on random data, running cross-validation on subset of the features repeatedly with different classifiers (GNB, LinearCSVMC, kNN) and with variable number of samples (2*8,2*16,2*32), and variable size of "searchlight" taking 16,32 or 64 random features. All classifier have a comparable distribution, Linear SVM is slightly wider than the 2 others. <br></div><div>For the same set of features, the accuracy is correlated between classifiers, but it can still vary a lot.<br></div><div>Having more samples reduces accuracy distribution, so it seems that it is a low-sample size issue.<br></div><div>Varying the size of searchlight does not changes the distribution much, so curse of dimensionality does not seems to be the problem here.<br></div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span><br>
> It seems that either from the design, pipeline or algorithm the information<br>
> is found by chance in the random data.<br>
<br>
> I took the neighborhood of where I got these results and ran a<br>
> cross-validation using the same partitioner but with GNB, LinearCSVMC, LDA.<br>
> GNB gives the same accuracy, so this is not the optimized GNBSearchlight<br>
> that causes this<br>
> LinearCSVMC and LDA gives about chance (50%) accuracy for the same<br>
> neighborhood.<br>
<br>
</span>that is on real data, right? how did you pre-process it?<br></blockquote><div> </div></span><div>No this is on random data from normal distribution.<br></div><div>About my preprocessing for my real data, it is similar to HCP data, with 'grayordinates' for cortex surfaces and subcortical ROI voxels.<br></div><div>I tried multiple detrending techniques.<br></div><div>I also tried multiple GLM estimation.<br></div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
could you share the plots of distributions (just curious)<br></blockquote></span><div>find enclosed distributions of searchlight accuracy maps across subjects.<br></div><div>As you can see they are not always perfectly centered around chance, so there is a bias in the data/mvpa that I cannot understand.<br> <br>So the question that remain is what is significant in a searchlight map, when random data can create a blob the size of the searchlight radius as high as 90% accuracy?<br></div><div>Maybe the 2-class design is to be avoided, or the number of samples should be proportional to the number of features we use.<br><br></div><div>Thanks.<br></div><div><br></div><div><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5">
<div><div><br>
> This can be reproduced by creating a random dataset from scratch with 2<br>
> classes and randomly selecting some features:<br>
> ds_rand2=dataset_wizard(<br>
> np.random.normal(size=(16,10000)),<br>
> targets=[0,1]*8,<br>
> chunks=np.arange(16))<br>
> cvte=CrossValidation(<br>
> GNB(common_variance=True),<br>
> prtnr_2fold_factpart,<br>
> errorfx=mean_match_accuracy)<br>
> np.max([cvte(ds_rand2[:,np.random.randint(0,ds_rand2.nfeatures,size=64)]).samples.mean()<br>
> for i in range(1000)])<br>
> 0.8828125<br>
> np.min([cvte(ds_rand2[:,np.random.randint(0,ds_rand2.nfeatures,size=64)]).samples.mean()<br>
> for i in range(1000)])<br>
> 0.1484375<br>
<br>
> So is there something specific to GNB that gives this kind of lucky<br>
> overfitting of random data when use many times as in Searchlight?<br>
> Also as this lucky features are included in multiple overlapping<br>
> neighborhood it results in nice blobs in the searchlight which sizes<br>
> depends on radius.<br>
> I tried the GNB with and without common_variance (thus piecewise quadratic<br>
> or linear) and it is quite similar.<br>
> Does anybody have been using it to produce sensible results?<br>
> Maybe it work better with more that 2 classes.<br>
<br>
> LDA when applied to more features than samples is incredibly slow, thus is<br>
> unrealistic for searchlight and even more with permutation testing, but I<br>
> have seen it used in many papers (maybe not with permutation though), so i<br>
> wonder if it is PyMVPA algorithm, or my python setup.<br>
> Do you think an optimized LDA searchlight would be possible or there is<br>
> lengthy computation (eg: matrix inversion) that cannot be factorized?<br>
<br>
</div></div>try first that m1nn as Richard recommended. It is simple but powerful<br>
if your data distribution matches its assumptions which are really close<br></div></div>
to the one behind LDA ;)<b>p</b><span class=""><br>
<span><br>
> Otherwise what kind of classifier would you recommend, that would not be<br>
> too computationally intensive? Or maybe I have to deal with that?<br>
<br>
> Many thanks for any idea about that.<br>
--<br>
</span>Yaroslav O. Halchenko<br>
Center for Open Neuroscience <a href="http://centerforopenneuroscience.org" rel="noreferrer" target="_blank">http://centerforopenneuroscience.org</a><br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834" target="_blank">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419" target="_blank">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" rel="noreferrer" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<div><div><br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></span></blockquote></div><br><br clear="all"><span class=""><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><font size="1">Basile Pinsard<br></font></div><i><font size="1">PhD candidate, <br></font></i></div><font size="1">Laboratoire d'Imagerie Biomédicale, UMR S 1146 / UMR 7371, Sorbonne Universités, UPMC, INSERM, CNRS</font><br><font size="1"><span><span style="color:rgb(51,51,51)"><span style="font-family:Arial,serif"><span lang="en-GB"><i>Brain-Cognition-Behaviour Doctoral School </i></span></span></span><span style="color:rgb(51,51,51)"><span style="font-family:Arial,serif"><span lang="en-GB"><b>, </b>ED3C<b>, </b>UPMC, Sorbonne Universités<br>Biomedical Sciences Doctoral School, Faculty of Medicine, Université de Montréal <br></span></span></span></span></font><font size="1">CRIUGM, Université de Montréal</font><br></div></div></div></div>
</span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><font size="1">Basile Pinsard<br></font></div><i><font size="1">PhD candidate, <br></font></i></div><font size="1">Laboratoire d'Imagerie Biomédicale, UMR S 1146 / UMR 7371, Sorbonne Universités, UPMC, INSERM, CNRS</font><br><font size="1"><span><span style="color:#333333"><span style="font-family:Arial,serif"><span lang="en-GB"><i>Brain-Cognition-Behaviour Doctoral School </i></span></span></span><span style="color:#333333"><span style="font-family:Arial,serif"><span lang="en-GB"><b>, </b>ED3C<b>, </b>UPMC, Sorbonne Universités<br>Biomedical Sciences Doctoral School, Faculty of Medicine, Université de Montréal <br></span></span></span></span></font><font size="1">CRIUGM, Université de Montréal</font><br></div></div></div></div>
</div></div></div>