<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
Hi Jo,<br><br>Thanks very much for your reply which is very helpful! I will look further to the link you sent and do leave-two/three-run-out cross-validation and see what I'll get. I'll keep you updated.<br><br>Best,<br>Meng<br><br><div><div id="SkyDrivePlaceholder"></div>> Date: Mon, 26 Nov 2012 11:09:53 -0600<br>> From: jetzel@artsci.wustl.edu<br>> To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> Subject: Re: [pymvpa] FW: What does a classification accuracy that is significantly lower than chancel level mean?<br>> <br>> Sorting out below-chance accuracy is really vexing. If you haven't seen <br>> it before, this topic has been discussed on this (and other mailing <br>> lists) before, see the thread at <br>> http://comments.gmane.org/gmane.comp.ai.machine-learning.pymvpa/611 . <br>> Googling "below-chance accuracy" also brings up some useful links.<br>> <br>> I have seen this phenomenon (permutation distribution looks reasonably <br>> normal and centered near chance but true-labeled accuracy in the left <br>> tail) occasionally in my own data.<br>> <br>> I don't have a good explanation for this, but tend to think it has to do <br>> with data that doesn't make a linear-svm-friendly shape in hyperspace. <br>> As typical in MVPA, you don't have a huge number of examples <br>> (particularly if you have more than a hundred or so voxels in the ROI), <br>> which also can make the classification results unstable.<br>> <br>> If you are reasonably sure that the dataset is good (the examples are <br>> properly labeled, the ROI masks fit well, etc) then I would try altering <br>> the cross-validation scheme to see if you can get the individual <br>> accuracies at (or above!) chance. For example, I'd try leaving two or <br>> three runs out instead of just one for the cross-validation. Having a <br>> small testing set (like you do with leave-one-run-out) can make a lot of <br>> variance in the cross-validation folds (i.e. the accuracy for each of <br>> the 6 classifiers going into each person's accuracy). Things seem to <br>> often go better when all the cross-validation folds have fairly similar <br>> accuracies (0.55, 0.6, 0.59, ...) rather than widely variable ones (0.5, <br>> 0.75, 0.6, ...).<br>> <br>> Good luck, and I'd love to hear if you find a solution.<br>> Jo<br>> <br>> <br>> On 11/26/2012 7:21 AM, Meng Liang wrote:<br>> > Dear Yaroslav,<br>> ><br>> > I'm still puzzled by the results of classification accuracy lower than<br>> > chance level. I've provided some details to your questions my previous<br>> > email, and I hope you could help me understand this puzzle. Many thanks<br>> > in advance!<br>> ><br>> > Best,<br>> > Meng<br>> ><br>> > ------------------------------------------------------------------------<br>> > From: meng.liang@hotmail.co.uk<br>> > To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> > Date: Sat, 10 Nov 2012 19:19:19 +0000<br>> > Subject: Re: [pymvpa] FW: What does a classification accuracy that is<br>> > significantly lower than chancel level mean?<br>> ><br>> > Dear Yaroslav,<br>> ><br>> > Thanks very much for your reply! Please see below for details.<br>> ><br>> > > > I'm running MVPA on some fMRI data (four different stimuli, say A, B, C<br>> > > > and D; six runs in each subject) to see whether the BOLD signals from a<br>> > > > given ROI can successfully predict the type of the stimulus. The MVPA<br>> > > > (leave-one-run-out cross-validation) was performed on each subject for<br>> > > > each two-way classification task. In a particular classification<br>> > task (say<br>> > > > classification A vs. B), in some subjects, the classification<br>> > accuracy was<br>> > > > (almost) significantly LOWER than the chance level (somewhere<br>> > between 0.2<br>> > > > and 0.4).<br>> > ><br>> > ><br>> > > depending on number of trials/cross-validation scheme even values of 0<br>> > > could come up by chance ;-) but indeed should not be 'significant'<br>> > ><br>> > > > What could be the reason for a significantly-lower-than-chance-level<br>> > > > accuracy?<br>> > ><br>> > > and how significant is this 'significantly LOWER'?<br>> ><br>> > The significant level was assessed by P value obtained from 10,000<br>> > permutations. Permutation was done within each subject, by randomly<br>> > assigning stimulus labels to each trial (the number of trials under each<br>> > label was still balanced; there were 8 trials per condition in each run,<br>> > and there were six runs in total). The P value was calculated as the<br>> > percentage of random permutations in which the resultant classification<br>> > accuracy was higher than the actual classification accuracy obtained<br>> > from the correct labels (for example, if none of 10,000 random<br>> > permutations led to a classification accuracy that was higher than the<br>> > actual classification accuracy, the P value would be 0). In this way, in<br>> > 5 out of 14 subjects, the P values were greater than 0.95. In other<br>> > words, the actual classification accuracy was located around the end of<br>> > the left tail of the null distribution in these 5 subjects (the shape of<br>> > the null distribution is like a bell, centered around 50%). In other 9<br>> > subjects, the actual classification accuracies were near or higher than<br>> > chance level.<br>> ><br>> > > details of # trials/cross-validation?<br>> ><br>> > There were 8 trials per condition in each run, and there were six runs<br>> > in total. Leave-one-run-out cross-validation was performed, that is, the<br>> > classifier (linear SVM) was trained on the data obtained from five runs<br>> > and tested on the remaining run (repeat the same procedure six times and<br>> > each time using a different run as a testing dataset).<br>> ><br>> > > > The P value was obtained from 10,000 permutations.<br>> > ><br>> > > is that permutations within the subject which at the end showed<br>> > > significant below 0? how permuations were done?<br>> ><br>> > I hope the reply above provide enough details of how the permutation was<br>> > done. Please let me know if there is anything unclear.<br>> ><br>> > ><br>> > > > But the<br>> > > > accuracies of all other classifications look fine in all subjects.<br>> > ><br>> > > fine means all above chance or still distributed around chance?<br>> ><br>> > By 'fine' I mean the classification accuracy was around (i.e. not far<br>> > from the chance level, can be lower or higher than chance level) or<br>> > above chance level. To me, around or above chance level makes more sense<br>> > than significantly lower than chance level.<br>> ><br>> > Thanks,<br>> > Meng<br>> <br>> -- <br>> Joset A. Etzel, Ph.D.<br>> Research Analyst<br>> Cognitive Control & Psychopathology Lab<br>> Washington University in St. Louis<br>> http://mvpa.blogspot.com/<br>> <br>> _______________________________________________<br>> Pkg-ExpPsy-PyMVPA mailing list<br>> Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br></div> </div></body>
</html>