<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
Dear Yaroslav,<br><br>Thanks very much for your reply! Please see below for details.<br><br><div><div id="SkyDrivePlaceholder"></div>> > I'm running MVPA on some fMRI data (four different stimuli, say A, B, C<br>> > and D; six runs in each subject) to see whether the BOLD signals from a<br>> > given ROI can successfully predict the type of the stimulus. The MVPA<br>> > (leave-one-run-out cross-validation) was performed on each subject for<br>> > each two-way classification task. In a particular classification task (say<br>> > classification A vs. B), in some subjects, the classification accuracy was<br>> > (almost) significantly LOWER than the chance level (somewhere between 0.2<br>> > and 0.4).<br>> <br>> <br>> depending on number of trials/cross-validation scheme even values of 0<br>> could come up by chance ;-) but indeed should not be 'significant'<br>> <br>> > What could be the reason for a significantly-lower-than-chance-level<br>> > accuracy?<br>> <br>> and how significant is this 'significantly LOWER'?<br><br>The significant level was assessed by P value obtained from 10,000 permutations. Permutation was done within each subject, by randomly assigning stimulus labels to each trial (the number of trials under each label was still balanced; there were 8 trials per condition in each run, and there were six runs in total). The P value was calculated as the percentage of random permutations in which the resultant classification accuracy was higher than the actual classification accuracy obtained from the correct labels (for example, if none of 10,000 random permutations led to a classification accuracy that was higher than the actual classification accuracy, the P value would be 0). In this way, in 5 out of 14 subjects, the P values were greater than 0.95. In other words, the actual classification accuracy was located around the end of the left tail of the null distribution in these 5 subjects (the shape of the null distribution is like a bell, centered around 50%). In other 9 subjects, the actual classification accuracies were near or higher than chance level. <br><br>> details of # trials/cross-validation?<br><br>There were 8 trials per condition in each run, and there were six runs in total. Leave-one-run-out cross-validation was performed, that is, the classifier (linear SVM) was trained on the data obtained from five runs and tested on the remaining run (repeat the same procedure six times and each time using a different run as a testing dataset). <br><br>> > The P value was obtained from 10,000 permutations. <br>> <br>> is that permutations within the subject which at the end showed<br>> significant below 0? how permuations were done?<br><br>I hope the reply above provide enough details of how the permutation was done. Please let me know if there is anything unclear.<br><br>> <br>> > But the<br>> > accuracies of all other classifications look fine in all subjects.<br>> <br>> fine means all above chance or still distributed around chance?<br><br>By 'fine' I mean the classification accuracy was around (i.e. not far from the chance level, can be lower or higher than chance level) or above chance level. To me, around or above chance level makes more sense than significantly lower than chance level. <br><br>Thanks,<br>Meng<br></div> </div></body>
</html>