<div dir="ltr">Thanks a lot, Jo.<br><br><div style="padding: 0px; overflow: hidden; visibility: hidden; left: -5000px; position: absolute; z-index: 9999; margin-left: 0px; margin-top: 0px; word-wrap: break-word; color: black; font-size: 10px; text-align: left; line-height: 130%;" id="avg_ls_inline_popup">
</div>I just wanted to make sure I understand your suggestions:<br><br>1. If I understand correctly, the way you propose to report the permutation results is the one explained here: <a href="http://www.bcn-nic.nl/txt/people/publications/Etzel2009.pdf">www.bcn-nic.nl/txt/people/publications/Etzel2009.pdf</a> <br>
When you refer to "single p-value for the across-subjects mean" you mean that based on permutation test I establish significance individually for each subject and then I just average those p-vals across subjects?<br>
<br>2. The FDR search light case: I first establish the significance in whatever way for each light separately, without any correction. Then, I pass my p-vals vector to FDR routine while I provide desired FDR threshold. At the end I get back which lights are significant after FDR correction and which are not. Correct?<br>
<br><br><div style="padding: 0px; overflow: hidden; visibility: hidden; left: -5000px; position: absolute; z-index: 9999; margin-left: 0px; margin-top: 0px; word-wrap: break-word; color: black; font-size: 10px; text-align: left; line-height: 130%;" id="avg_ls_inline_popup">
</div><div class="gmail_quote">On Thu, May 19, 2011 at 8:51 PM, J.A. Etzel <span dir="ltr"><<a href="mailto:jetzel@artsci.wustl.edu">jetzel@artsci.wustl.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On 5/19/2011 1:35 AM, Vadim Axel wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Yes, I agree with you. However, I somehow feel that reporting<br>
significance based on permutation values is more cumbersome than<br>
t-tests. Consider the case that out of 10 subjects 8 have significant<br>
result (based on permutation) and two remaining are not. What should I<br>
say in my results? Does the ROI discriminate between two classes? When I<br>
use group t-test everything is simple - the result is true or false for<br>
the whole group. Now, suppose that I have more than one ROI and I want<br>
to compare their results. Though I can show average prediction rate<br>
across subjects, I am afraid that when I start to report for each ROI<br>
for how many subjects it was significant and for how many not,<br>
everybody (including myself) would be confused....<br>
</blockquote></div>
Yes, more detail is required when reporting a permutation test; I like to see a description of the label permutation scheme and number of permutations, at minimum.<br>
<br>
For describing a within-subjects analysis (accuracy calculated for each subject separately, but you want to talk about general results - not just each person separately) my usual strategy is to calculate the p-value for the across-subjects mean, using the permutations calculated for each person separately. You can then report a single p-value for the across-subjects mean, plus the individual subjects' p-values as well if you want.<br>
<br>
Specifically, I pre-calculate my label permutations, and use the same permutations for every subject (as much as possible, if missings). This gives (say) 1000 accuracies for each person: accuracy for subject 1 label rearrangement 1, subject 2 rearrangement 1, etc. I use those 1000 accuracies to get the p-value for each person's accuracy. But you can also use them to make a group distribution by averaging the accuracies for each of the permutations (mean of subject 1 rearrangement 2, subject 2 rearrangement 1, etc), then comparing against the real average accuracy.<br>
<br>
Comparing the results from multiple ROIs is tricky; I don't know that I've seen a really satisfactory general answer. Building up a test for each particular analysis is probably the way to go; answer questions like: exactly what are you trying to compare? Do the ROIs have a similar number of voxels? Are they spatially very distinct or perhaps overlapping?<div class="im">
<br>
<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
BTW, how you recommend to correct for multiple comparisons? For example<br>
I run 100 search lights.Making Bonferoni correction (0.05/100) = 0.0005<br>
results in very high threshold. Consider my case with the mean values,<br>
which is based on 1000 tests only. Based on 0.0005 threshold I need to<br>
get classification of 0.75+ (!). My data are not that good :( What<br>
people are doing for whole brain when the number of search lights is<br>
tens of thousands...<br>
</blockquote></div>
For ROI-based analyses with only a few ROIs Bonferroni is fine. But I have went back to parametric for searchlight, using the FDR/cluster size/etc. stats built into SPM. Kriegeskorte describes some permutation tests in the original searchlight paper, but most people seem to use parametric stats adapted from GLM fMRI analyses.<br>
<br>
Jo<div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>