<div dir="ltr">Hi,<br><div style="padding: 0px; overflow: hidden; visibility: hidden; left: -5000px; position: absolute; z-index: 9999; margin-left: 0px; margin-top: 0px; word-wrap: break-word; color: black; font-size: 10px; text-align: left; line-height: 130%;" id="avg_ls_inline_popup">
</div><br>Thank you both for the answers!<br><br><br>1. The mean chance is a perfect 0.5. The 0.6 is a tail.<br><br>2. I have 6 trials per block and 25 blocks for each condition in total. So, in one scenario I average the trials within block and make classification based on 25 data points per condition. There I get 0.6 permutated prediction in the tail. In other case I do not average and run classification based on 25x6=150 data points per condition. There I get ~0.55 permutated prediction in the tail. I attach the histograms for mean and for non-mean permutation. For raw data definitely looks more normal.<br>
<br>3. I did a manual permutation by reshuffling the labels. In particular, I have a matrix of data values [trials X voxels] and a vector of correct labels [correct labels x 1]. For each permutation test I randomize the order of correct labels vector. Makes sense? As far as I understand the Monte-Carlo simulation works for artificially generated data values. But I am using my original data labels.<br>
<br>BTW, I did not use for this analysis PyMVPA, so you have no reason to worry about potential bug in your code :) <br>
<br><br>Thanks again,<br>Vadim<br><br><br><br><div class="gmail_quote">On Mon, May 16, 2011 at 7:53 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">d'oh -- just now recalled that I have this email in draft:<br>
<br>
eh, picture (histogram) would have been useful:<br>
<div><br>
> I wanted to ask your opinion about some weird result that I get.<br>
> To establish the significance I randomly permute my labels and I get a<br>
> prediction rate of 0.6 and even above it (p-value=0.05). In other words 5%<br>
> of of permuted samples result in 0.6+ prediction rate. The training/test<br>
> samples are independent and ROI size is small (no overfitting).<br>
<br>
</div>just to make sure: 0.6 is not a mean-chance performance across the<br>
permutations. You just worry that the distribution of chance<br>
performances is so wide that the right 5% tail is above 0.6 accuracy.<br>
<br>
if that is the case, it is indeed a good example case ;)<br>
<div><br>
> Interestingly, the described result I get when I average trials within block<br>
> (use one data-point per block; ~25 blocks in total). When I run the<br>
<br>
</div>so it is 25 blocks for 2 conditions? which one has more? ;)<br>
<div><br>
> classification on raw trials, my permutation threshold becomes ~0.55. In<br>
> both cases for non-permuted labels the prediction is around significance<br>
> level.<br>
> How should I treat such a result? What might have gone wrong?<br>
<br>
</div>I guess nothing went wrong and everything is logical. With of<br>
random chance performances distribution is conditioned on many factors,<br>
such as independence of samples, presence of order effects, how<br>
permutation is done (disregarding dependence of samples or not) etc.<br>
<br>
So, to troubleshoot we could start with:<br>
<br>
* histogram<br>
* what kind of permutation testing have you done? (i.e. what was<br>
permutted exactly? was testing set labels permutted?)<br>
have you seen recently improved<br>
<a href="http://www.pymvpa.org/examples/permutation_test.html" target="_blank">http://www.pymvpa.org/examples/permutation_test.html</a><br>
? ;)<br>
<font color="#888888"><br>
--<br>
.-.<br>
=------------------------------ /v\ ----------------------------=<br>
Keep in touch // \\ (yoh@|www.)<a href="http://onerussian.com" target="_blank">onerussian.com</a><br>
Yaroslav Halchenko /( )\ ICQ#: 60653192<br>
Linux User ^^-^^ [175555]<br>
</font><div><div></div><div><br>
<br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>