<div dir="ltr"><div><div><div><div><div><div><div><div><div>Thanks again Yaroslav.<br><br></div>I agree that the classifier might end up giving 0 or very small balanced accuracy (or micro accuracy) values but I think thats still a better measure than using overall accuracy (or macro accuracy). There are couple of other measures that can be useful for imbalanced datasets:<br><br></div>1. A-mean: arithmetic mean, same as average class-wise accuracy or micro-accuracy<br></div>2. G-mean: geometric mean instead of arithmetic mean above<br></div>3. F-measure<br></div>4. Area under the ROC curve<br><br></div>Of course a better solution would be using a classifier that can handle imbalanced datasets, as you suggested. I have previously used SVMperf that can optimize AU-ROC: <a href="https://www.cs.cornell.edu/people/tj/svm_light/svm_perf.html">https://www.cs.cornell.edu/people/tj/svm_light/svm_perf.html</a><br><br></div>Not sure how easy it is to incorporate new classifiers in PyMVPA but I could give it a try with some guideline.<br><br></div>Best regards,<br></div>Kaustubh<br><br><div><div><div><div><br><div><div><div><div><div><div><br></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 15, 2016 at 11:08 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
On Fri, 15 Jan 2016, Kaustubh Patil wrote:<br>
<br>
> Thanks Yaroslav.<br>
<br>
> I tried your solution and it seems to work for this particular dataset but<br>
> unfortunately not for other datasets as the labels cannot be balanced<br>
> easily.<br>
<br>
> Maybe it's possible to directly calculate balanced measures in the CV? I<br>
> guess I will have to change the code to do that, any suggestions where to<br>
> start?<br>
<br>
</span>some toolboxes compute 'mean of within class accuracies' (not mean<br>
overall accuracy) which allows to account for disbalance. I guess we<br>
could code it quite easily if you like<br>
<br>
BUT the problem really would remain: with small number of samples<br>
classifier might just take the "majority" label since it would minimize<br>
error more than low performace decision. So you would hurt yourself<br>
more than help.<br>
<br>
another solution is to try a classifier which provides weighting<br>
to the classes, e.g. as GNB with default prior setting does. you could<br>
try it and see how it goes. It is not the greatest classifier but a<br>
start. then you could add similar class weighting to some other<br>
classifiers supporting that.<br>
<span class=""><br>
> Best regards<br>
> On Sat, Dec 19, 2015 at 3:51 PM, Yaroslav Halchenko<br>
> <<a href="mailto:debian@onerussian.com">debian@onerussian.com</a>> wrote:<br>
<br>
> On Sat, 19 Dec 2015, Kaustubh Patil wrote:<br>
<br>
> > Hi,<br>
<br>
> > I want to use PyMVPA for whole-brain searchlight analysis on some<br>
> existing<br>
> > data. The data has been already preprocessed (skull stripping, motion<br>
> > correction etc.). Each subject data contains 10 runs and each run was<br>
> processed<br>
> > separately, so there is a separate full brain boolean mask for each<br>
> run.<br>
<br>
> > My question is what is the recommended/correct a way to use this data<br>
> to<br>
> > perform run-wise cross-validation searchlight?<br>
<br>
> you have a problem here, since you have done per run preprocessing, in<br>
> particular motion-correction, your volumes are misaligned across runs.<br>
> (used FSL, didn't you? )<br>
<br>
> ideally, you redo preprocessing while motion correcting to the same<br>
</span>> volume across all the runs.A Alternatively, you reslice all the runs<br>
<span class="">> into the same space (could well be the common space your toolkit used<br>
> for analysis across runs -- common anatomical or MNI) and then do<br>
> analysis there, while again unifying your mask, which must be the same<br>
> across all the runs.<br>
> > As I understand, each run has to be in the same space (same number of<br>
> voxels)<br>
> > so that training and test can be performed, so the whole brain masks<br>
> have to be<br>
> > somehow aligned. How would you recommend doing this?<br>
<br>
> it is not a mere 'number of voxels' problem but rather that you have<br>
</span>> misaligned across runs volumes.A if just voxel number -- choose<br>
<div class="HOEnZb"><div class="h5">> intersection of all masks.<br>
--<br>
Yaroslav O. Halchenko<br>
Center for Open Neuroscience <a href="http://centerforopenneuroscience.org" rel="noreferrer" target="_blank">http://centerforopenneuroscience.org</a><br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" rel="noreferrer" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>