<div><span style>Hi Nynke,</span></div><div><span style><br></span></div><div><font color="#222222" face="arial, sans-serif">Could you help me about the question that you have asked?</font></div><div><font color="#222222" face="arial, sans-serif"><br>
</font></div><div><span style>“How could I perform the permutation test for each</span><br style><span class="il" style="background-image:initial;background-color:rgb(255,255,204);color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">searchlight</span><span style> and let the </span><span class="il" style="background-image:initial;background-color:rgb(255,255,204);color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">searchlight</span><span style> output also the p values for</span><br style>
<span style>testing against the null distribution?”</span></div><div><span style><br></span></div><div><span style>I want to do the same thing and have you solved this?</span></div><div><font color="#222222" face="arial, sans-serif"><br>
</font></div><div><font color="#222222" face="arial, sans-serif">Best!</font></div><div><span style><br></span></div><div><span style>Xiangzhen</span></div><br><br><div class="gmail_quote">On Tue, Aug 2, 2011 at 8:00 PM, <span dir="ltr"><<a href="mailto:pkg-exppsy-pymvpa-request@lists.alioth.debian.org">pkg-exppsy-pymvpa-request@lists.alioth.debian.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Pkg-ExpPsy-PyMVPA mailing list submissions to<br>
<a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org">pkg-exppsy-pymvpa@lists.alioth.debian.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:pkg-exppsy-pymvpa-request@lists.alioth.debian.org">pkg-exppsy-pymvpa-request@lists.alioth.debian.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:pkg-exppsy-pymvpa-owner@lists.alioth.debian.org">pkg-exppsy-pymvpa-owner@lists.alioth.debian.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Searchlight + cross validation & significance (permutation<br>
test) (Nynke van der Laan)<br>
2. Re: Searchlight + cross validation & significance<br>
(permutation test) (Francisco Pereira)<br>
3. Re: Searchlight + cross validation & significance<br>
(permutation test) (J.A. Etzel)<br>
4. Searchlight + cross validation & significance (permutation<br>
test) (Nynke van der Laan)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Mon, 1 Aug 2011 16:34:03 +0200<br>
From: Nynke van der Laan <<a href="mailto:nynkevanderlaan@gmail.com">nynkevanderlaan@gmail.com</a>><br>
To: <a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org">pkg-exppsy-pymvpa@lists.alioth.debian.org</a><br>
Subject: [pymvpa] Searchlight + cross validation & significance<br>
(permutation test)<br>
Message-ID:<br>
<CADHpHBOUFyc4nfboeYjBOngHry=<a href="mailto:m8JV5Cmg0w_75CWQ6nCrZxQ@mail.gmail.com">m8JV5Cmg0w_75CWQ6nCrZxQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
Hello all,<br>
<br>
I have a question about the following: I am performing a searchlight<br>
analysis (v04) with a crossvalidation (leave one subject out, 20<br>
samples per subject, 2 labels (10 samples label A, 10 samples label<br>
B)).<br>
I have adapted the example scripts to generate a map with the mean<br>
classification error over de NFold crossvalidation. Now, in addition I<br>
would like to be able to say something about the significance of the<br>
accuracies that the analysis yielded.<br>
In the module reference<br>
(<a href="http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight" target="_blank">http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight</a>)<br>
I noticed the null_prob and null_T are available variables. Null_prob<br>
is enabled as default. However, the analysis now only results in the<br>
searchlight errormap. How can I acces the null_prob information? Is<br>
this averaged over the NFolds?<br>
<br>
cv = CrossValidatedTransferError(<br>
TransferError(LinearCSVMC()),<br>
NFoldSplitter())<br>
sl = Searchlight(cv, radius=6)<br>
sl_map = sl(dataset_total)<br>
<br>
An additional question I have is whether there is any easy way to<br>
combine a searchlight analysis as described above with a permutation<br>
test. In a previous discussion I read about the NullhypothesisTest<br>
(<a href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html" target="_blank">http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html</a>)<br>
but this does not seem to be present in the v.04 version anymore (?).<br>
I predict that this analysis will be very costly in time as the<br>
permutation is done in each searchlight, but I wouldn't know any<br>
good/valid alternatives. Does anyone else have a suggestion?<br>
<br>
Thanks a lot!<br>
<br>
Best,<br>
Nynke<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Mon, 1 Aug 2011 10:47:35 -0400<br>
From: Francisco Pereira <<a href="mailto:francisco.pereira@gmail.com">francisco.pereira@gmail.com</a>><br>
To: Development and support pf PyMVPA<br>
<<a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org">pkg-exppsy-pymvpa@lists.alioth.debian.org</a>><br>
Subject: Re: [pymvpa] Searchlight + cross validation & significance<br>
(permutation test)<br>
Message-ID:<br>
<<a href="mailto:CAFdubX0HLpLfKFtoQKZCg7ViZjUq0EcAi7HHNpkHXXptwt8scQ@mail.gmail.com">CAFdubX0HLpLfKFtoQKZCg7ViZjUq0EcAi7HHNpkHXXptwt8scQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan<br>
<<a href="mailto:nynkevanderlaan@gmail.com">nynkevanderlaan@gmail.com</a>> wrote:<br>
><br>
> An additional question I have is whether there is any easy way to<br>
> combine a searchlight analysis as described above with a permutation test.<br>
<br>
For a particular classifier (GNB) it's possible to do this fast<br>
enough, if I recall correctly this is the PyMVPA implementation<br>
<br>
<a href="http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html" target="_blank">http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html</a><br>
<br>
cheers,<br>
Francisco<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Mon, 01 Aug 2011 09:58:19 -0500<br>
From: "J.A. Etzel" <<a href="mailto:jetzel@artsci.wustl.edu">jetzel@artsci.wustl.edu</a>><br>
To: <a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org">pkg-exppsy-pymvpa@lists.alioth.debian.org</a><br>
Subject: Re: [pymvpa] Searchlight + cross validation & significance<br>
(permutation test)<br>
Message-ID: <<a href="mailto:4E36BF0B.9080607@artsci.wustl.edu">4E36BF0B.9080607@artsci.wustl.edu</a>><br>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br>
<br>
My usual strategy is to split up the permutations and run the jobs on a<br>
cluster; with luck your university will have access to one. Then you can<br>
have dozens of computers working on the job at the same time, for<br>
example one node doing permutations 1 to 5, another 6 to 10, etc.<br>
<br>
Jo<br>
<br>
<br>
<br>
On 8/1/2011 9:47 AM, Francisco Pereira wrote:<br>
> On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan<br>
> <<a href="mailto:nynkevanderlaan@gmail.com">nynkevanderlaan@gmail.com</a>> wrote:<br>
>><br>
>> An additional question I have is whether there is any easy way to<br>
>> combine a searchlight analysis as described above with a permutation test.<br>
><br>
> For a particular classifier (GNB) it's possible to do this fast<br>
> enough, if I recall correctly this is the PyMVPA implementation<br>
><br>
> <a href="http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html" target="_blank">http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html</a><br>
><br>
> cheers,<br>
> Francisco<br>
><br>
> _______________________________________________<br>
> Pkg-ExpPsy-PyMVPA mailing list<br>
> <a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
> <a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Tue, 2 Aug 2011 11:55:21 +0200<br>
From: Nynke van der Laan <<a href="mailto:nynkevanderlaan@gmail.com">nynkevanderlaan@gmail.com</a>><br>
To: <a href="mailto:pkg-exppsy-pymvpa@lists.alioth.debian.org">pkg-exppsy-pymvpa@lists.alioth.debian.org</a><br>
Subject: [pymvpa] Searchlight + cross validation & significance<br>
(permutation test)<br>
Message-ID:<br>
<CADHpHBPh=<a href="mailto:6jy0BSYObZi08aVy0i0%2BiXgWb4btOX7Yf84seNPvQ@mail.gmail.com">6jy0BSYObZi08aVy0i0+iXgWb4btOX7Yf84seNPvQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
Thanks for your replies!<br>
Francisco, great suggestion, I will absolutely consider that<br>
classifier for my next study. For the present study/additional<br>
analysis though, I would like to stick with the v4.0 version and the<br>
linear SVM as this showed good results (and I preferably don't switch<br>
to a different version in between analyses/revisions).<br>
Jo, using multiple computers is indeed a good option,<br>
<br>
However, in the documentation I read that if there is a strong<br>
assumptiom of the data being normally/gaussian distributed<br>
(<a href="http://v04.pymvpa.org/examples/permutation_test.html" target="_blank">http://v04.pymvpa.org/examples/permutation_test.html</a>), 30<br>
permutations would also suffice. As my data indeed approaches a normal<br>
distribution, I would like to try this first. But, unfortunately I did<br>
not manage to implement this into the searchlight. In the example, it<br>
is clearly described how this permutation test would be done in a<br>
'normal' analysis, but how exactly to combine this with the<br>
searchlight is unclear to me. I assume the null_dist = MCNullDist...<br>
should be included in the TransferError in the script below, but the<br>
searchlight now only returns the error scores for each voxel and no<br>
other results. How could I perform the permutation test for each<br>
searchlight and let the searchlight output also the p values for<br>
testing against the null distribution?<br>
<br>
cv = CrossValidatedTransferError(<br>
TransferError(LinearCSVMC()),<br>
NFoldSplitter())<br>
sl = Searchlight(cv, radius=6)<br>
sl_map = sl(dataset_total)<br>
<br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
<br>
End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1<br>
************************************************<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse;color:rgb(80,0,80)"><div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse;color:rgb(80,0,80)">-----------------------------------------------------------------------------------------</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse;color:rgb(80,0,80)">Kong Xiangzhen</span></div>State Key Laboratory of Cognitive Neuroscience and Learning,<br>Beijing Normal University,<br>
Beijing, China, 100875.</span><br>