[pymvpa] Searchlight + cross validation & significance
Nynke van der Laan
nynkevanderlaan at gmail.com
Fri Apr 13 11:07:55 UTC 2012
Dear Xiangzhen,
in the end I did not try to implement permutation-test for each crossv
in each searchlight because this was rather impractical in terms of
computation time. As a 30 time permutation (which was my first
thought) is quite low, and with 500 permutation it took too much time,
I decided to use other methods of ensuring the validity of the
results. E.g., I shuffled the labels, and I looked in a control region
(which shouldn't have predictive value) if this resulted in
significant predictions.
Good luck,
Best regards,
Nynke
----------------------------------------------------
Nynke van der Laan, MSc
PhD-candidate
Image Sciences Institute
University Medical Center Utrecht
Heidelberglaan 100
Post: Rm Q0S.459/Office: Rm E.02.557
3584 CX Utrecht
The Netherlands
Phone: +31 88 75 58562
http://www.isi.uu.nl/People/?nynke
http://www.selfregulationlab.nl/index.php/nynke-van-der-laan
On Thu, Apr 12, 2012 at 6:29 PM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
> Hi guys,
>
> sorry for not following up with speedy replies ...
>
> Searchlight is just a Measure so its chance distribution (per each
> searchlight) and corresponding p-values could be computed the same
> fashion as any other measure, e.g. like in
> http://www.pymvpa.org/examples/permutation_test.html
> done for CrossValidation
>
> and e.g. look into this unittest
> https://github.com/PyMVPA/PyMVPA/blob/HEAD/mvpa2/tests/test_searchlight.py#L236
>
> hope this helps
>
> cheers,
>
> On Fri, 13 Apr 2012, Xiangzhen Kong wrote:
>
>> Hi Nynke,
>> Could you help me about the question that you have asked?
>> �How could I perform the permutation test for each
>> searchlight�and let the�searchlight�output also the p values for
>> testing against the null distribution?�
>> I want to do the same thing and have you solved this?
>> Best!
>> Xiangzhen
>
>> On Tue, Aug 2, 2011 at 8:00 PM,
>> <[1]pkg-exppsy-pymvpa-request at lists.alioth.debian.org> wrote:
>
>> Send Pkg-ExpPsy-PyMVPA mailing list submissions to
>> � � � �[2]pkg-exppsy-pymvpa at lists.alioth.debian.org
>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> � � �
>> �[3]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>> or, via email, send a message with subject or body 'help' to
>> � � � �[4]pkg-exppsy-pymvpa-request at lists.alioth.debian.org
>
>> You can reach the person managing the list at
>> � � � �[5]pkg-exppsy-pymvpa-owner at lists.alioth.debian.org
>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Pkg-ExpPsy-PyMVPA digest..."
>
>> Today's Topics:
>
>> � 1. Searchlight + cross validation & significance (permutation
>> � � �test) (Nynke van der Laan)
>> � 2. Re: Searchlight + cross validation & significance
>> � � �(permutation test) (Francisco Pereira)
>> � 3. Re: Searchlight + cross validation & significance
>> � � �(permutation test) (J.A. Etzel)
>> � 4. Searchlight + cross validation & significance (permutation
>> � � �test) (Nynke van der Laan)
>
>> ----------------------------------------------------------------------
>
>> Message: 1
>> Date: Mon, 1 Aug 2011 16:34:03 +0200
>> From: Nynke van der Laan <[6]nynkevanderlaan at gmail.com>
>> To: [7]pkg-exppsy-pymvpa at lists.alioth.debian.org
>> Subject: [pymvpa] Searchlight + cross validation & significance
>> � � � �(permutation � �test)
>> Message-ID:
>> � � �
>> �<CADHpHBOUFyc4nfboeYjBOngHry=[8]m8JV5Cmg0w_75CWQ6nCrZxQ at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>
>> Hello all,
>
>> I have a question about the following: I am performing a searchlight
>> analysis (v04) with a crossvalidation (leave one subject out, 20
>> samples per subject, 2 labels (10 samples label A, 10 samples label
>> B)).
>> I have adapted the example scripts to generate a map with the mean
>> classification error over de NFold crossvalidation. Now, in addition I
>> would like to be able to say something about the significance of the
>> accuracies that the analysis yielded.
>> In the module reference
>> ([9]http://v04.pymvpa.org/modref/mvpa.measures.searchlight.html#module-measures-searchlight)
>> I noticed the null_prob and null_T are available variables. Null_prob
>> is enabled as default. However, the analysis now only results in the
>> searchlight errormap. How can I acces the null_prob information? Is
>> this averaged over the NFolds?
>
>> cv = CrossValidatedTransferError(
>> � � � � TransferError(LinearCSVMC()),
>> � � � � NFoldSplitter())
>> sl = Searchlight(cv, radius=6)
>> sl_map = sl(dataset_total)
>
>> An additional question I have is whether there is any easy way to
>> combine a searchlight analysis as described above with a permutation
>> test. In a previous discussion I read about the NullhypothesisTest
>> ([10]http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2008q2/000031.html)
>> but this does not seem to be present in the v.04 version anymore (?).
>> I predict that this analysis will be very costly in time as the
>> permutation is done in each searchlight, but I wouldn't know any
>> good/valid alternatives. Does anyone else have a suggestion?
>
>> Thanks a lot!
>
>> Best,
>> Nynke
>
>> ------------------------------
>
>> Message: 2
>> Date: Mon, 1 Aug 2011 10:47:35 -0400
>> From: Francisco Pereira <[11]francisco.pereira at gmail.com>
>> To: Development and support pf PyMVPA
>> � � � �<[12]pkg-exppsy-pymvpa at lists.alioth.debian.org>
>> Subject: Re: [pymvpa] Searchlight + cross validation & significance
>> � � � �(permutation test)
>> Message-ID:
>> � � �
>> �<[13]CAFdubX0HLpLfKFtoQKZCg7ViZjUq0EcAi7HHNpkHXXptwt8scQ at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>
>> On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
>> <[14]nynkevanderlaan at gmail.com> wrote:
>
>> > An additional question I have is whether there is any easy way to
>> > combine a searchlight analysis as described above with a permutation
>> test.
>
>> For a particular classifier (GNB) it's possible to do this fast
>> enough, if I recall correctly this is the PyMVPA implementation
>
>> [15]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html
>
>> cheers,
>> Francisco
>
>> ------------------------------
>
>> Message: 3
>> Date: Mon, 01 Aug 2011 09:58:19 -0500
>> From: "J.A. Etzel" <[16]jetzel at artsci.wustl.edu>
>> To: [17]pkg-exppsy-pymvpa at lists.alioth.debian.org
>> Subject: Re: [pymvpa] Searchlight + cross validation & significance
>> � � � �(permutation test)
>> Message-ID: <[18]4E36BF0B.9080607 at artsci.wustl.edu>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>> My usual strategy is to split up the permutations and run the jobs on a
>> cluster; with luck your university will have access to one. Then you can
>> have dozens of computers working on the job at the same time, for
>> example one node doing permutations 1 to 5, another 6 to 10, etc.
>
>> Jo
>
>> On 8/1/2011 9:47 AM, Francisco Pereira wrote:
>> > On Mon, Aug 1, 2011 at 10:34 AM, Nynke van der Laan
>> > <[19]nynkevanderlaan at gmail.com> �wrote:
>
>> >> An additional question I have is whether there is any easy way to
>> >> combine a searchlight analysis as described above with a permutation
>> test.
>
>> > For a particular classifier (GNB) it's possible to do this fast
>> > enough, if I recall correctly this is the PyMVPA implementation
>
>> > [20]http://www.pymvpa.org/generated/mvpa.measures.gnbsearchlight.html
>
>> > cheers,
>> > Francisco
>
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > [21]Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>
>> [22]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>> ------------------------------
>
>> Message: 4
>> Date: Tue, 2 Aug 2011 11:55:21 +0200
>> From: Nynke van der Laan <[23]nynkevanderlaan at gmail.com>
>> To: [24]pkg-exppsy-pymvpa at lists.alioth.debian.org
>> Subject: [pymvpa] Searchlight + cross validation & significance
>> � � � �(permutation � �test)
>> Message-ID:
>> � � �
>> �<CADHpHBPh=[25]6jy0BSYObZi08aVy0i0+iXgWb4btOX7Yf84seNPvQ at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>
>> Thanks for your replies!
>> Francisco, great suggestion, I will absolutely consider that
>> classifier for my next study. For the present study/additional
>> analysis though, I would like to stick with the v4.0 version and the
>> linear SVM as this showed good results (and I preferably don't switch
>> to a different version in between analyses/revisions).
>> Jo, using multiple computers is indeed a good option,
>
>> However, in the documentation I read that if there is a strong
>> assumptiom of the data being normally/gaussian distributed
>> ([26]http://v04.pymvpa.org/examples/permutation_test.html), 30
>> permutations would also suffice. As my data indeed approaches a normal
>> distribution, I would like to try this first. But, unfortunately I did
>> not manage to implement this into the searchlight. In the example, it
>> is clearly described how this permutation test would be done in a
>> 'normal' analysis, but how exactly to combine this with the
>> searchlight is unclear to me. I assume the null_dist = MCNullDist...
>> should be included in the TransferError in the script below, but the
>> searchlight now only returns the error scores for each voxel and no
>> other results. How could I perform the permutation test for each
>> searchlight and let the searchlight output also the p values for
>> testing against the null distribution?
>
>> cv = CrossValidatedTransferError(
>> � � � � TransferError(LinearCSVMC()),
>> � � � � NFoldSplitter())
>> sl = Searchlight(cv, radius=6)
>> sl_map = sl(dataset_total)
>
>> ------------------------------
>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> [27]Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> [28]http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
>
>> End of Pkg-ExpPsy-PyMVPA Digest, Vol 42, Issue 1
>> ************************************************
> --
> =------------------------------------------------------------------=
> Keep in touch www.onerussian.com
> Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic
More information about the Pkg-ExpPsy-PyMVPA
mailing list