<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div>Hi Christopher,<br><br></div>I think your method works, if I do the following<br><br></div><div>res=[]<br></div>gnb=GNB()<br></div>nf=NFoldPartitioner()<br></div>permutator=AttributePermutator('targets',number_of_permutations)<br></div>sl_gnb=sphere_gnbsearchlight(gnb,nf,reuse_neighbors=True,radius=3)<br></div>for i in permutator.generate(ds): <br> res.append(sl_gnb(ds))<br></div><br></div>And then I combine all the resulting results in res. I couldn't make it work with a ChainNode containing a permutator and the sl_gnb; it would only run once, so I had to use the permutator explicitly as a generator.<br><br></div>If I do things this way, there's no way to just permute the labels within the training data, right? Isn't it better to do that?<br><br></div><div>I also don't understand, if the gnbsearchlight is not seeing the permutator, how its implementation is saving time. It has to do that pre-calculation on each permuted dataset, right? Does it just take that much less time to run a gnbsearchlight through the brain on each dataset?<br></div><div><br><br></div>I had also gotten permutation testing working with a linear SVM, using the following:<br><br> repeater = Repeater(count=number_of_permutations)<br> permutator = AttributePermutator('targets',limit={'partitions':1},count=1)<br> nf = NFoldPartitioner()<br> clf = LinearCSVMC()<br> null_cv = CrossValidation(clf,ChainNode([nf,permutator],space=nf.get_space()))<br> distr_est = MCNullDist(repeater,tail='left',measure=null_cv,enable_ca=['dist_samples'])<br> cv = CrossValidation(clf,nf,null_dist=distr_est)<br> sl = sphere_searchlight(cv,radius=3,center_ids=range(0,10),enable_ca='roi_sizes',pass_attr=[('ca.roi_sizes','fa')])<br> res = sl(ds)<br><br></div>Is there anything wrong with doing it this way? Other than it just taking a very long time.<br></div><div><br>I would still need to make sure to pass on the null_prob from cv to sl to res, and I can't figure out how to pass the null distribution, if that's something I decide to do. I'm leaning towards not saving the null distribution -- it's good to know it's feasible but I'm not sure it's worth it. I didn't have any specific plans to use it, I just wanted to have it around in case it turned out to be useful.<br><br></div>Thanks,<br></div>Bill<br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 16, 2015 at 3:29 PM, Christopher J Markiewicz <span dir="ltr"><<a href="mailto:effigies@bu.edu" target="_blank">effigies@bu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Bill,<br>
<br>
Hopefully what I write below won't be entirely wrong. I've done<br>
permutation testing, but with PyMVPA at the bottom of the loop, not<br>
itself managing null distributions and statistics.<br>
<span class=""><br>
On 06/16/2015 02:50 PM, Bill Broderick wrote:<br>
> Hi all,<br>
><br>
> I'm trying to implement permutation testing with searchlights and, after<br>
> going through the manual and the mailing list archives, I'm still not<br>
> sure how to do it.<br>
><br>
> According to this thread<br>
> <a href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2012q1/002071.html" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2012q1/002071.html</a>,<br>
> the fastest way to get searchlight permutations is to use the GNB<br>
> searchlight; otherwise it takes so long as to be impractical. However,<br>
> when trying to set up the GNB searchlight with a null_dist, as shown in<br>
> here<br>
> <a href="https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/tests/test_usecases.py#L168" rel="noreferrer" target="_blank">https://github.com/PyMVPA/PyMVPA/blob/master/mvpa2/tests/test_usecases.py#L168</a>,<br>
> I get a NotImplementedError: "GNBSearchlight does not yet support<br>
> partitions altering the target (e.g. permutators)", as warned about on<br>
> the documentation page for GNB searchlight and mentioned in this thread<br>
> <a href="http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2012q4/002304.html" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/2012q4/002304.html</a>.<br>
><br>
> However, if I can't use an attribute permutator with the GNB<br>
> searchlight, how can I use it to run permutation tests and get a null<br>
> probability? What am I missing?<br>
<br>
</span>I looked at GNB searchlight a while back, and I believe it works by<br>
creating a distribution per-target, per-voxel, and then moving a<br>
searchlight around these pre-calculated distributions. So it makes sense<br>
that running a permutator inside the GNB won't work, since it negates<br>
the pre-computation advantage.<br>
<br>
An almost* equivalent problem is to permute the class labels and run a<br>
GNB searchlight. (*There is the difference that you'll be sampling the<br>
same subspace of permutations at each voxel, but that shouldn't make a<br>
large difference if you perform enough permutations.)<br>
<br>
So suppose you have something like this (I have not checked the docs to<br>
make sure this is particularly sensible):<br>
<br>
partitioner = ChainNode(NFoldPartitioner(), AttributePermutator())<br>
sl = GNBSearchlight(GNB(), partitioner)<br>
err = sl(dataset)<br>
<br>
You might instead do something like:<br>
<br>
gnb = GNBSearchlight(GNB(), NFoldPartitioner())<br>
sl = ChainNode(gnb, AttributePermutator())<br>
err = sl(dataset)<br>
<br>
Again, not sure if this is really how the pieces fit together, but the<br>
idea would be to permute the class labels and run an entirely new<br>
GNBSearchlight on them. Assuming I did what I meant to do, err should<br>
actually be your null distribution (a matrix of voxels-by-permutations<br>
errors).<br>
<span class=""><br>
> Additionally, and this is a side note, is there any way to pass the null<br>
> distribution from the searchlight's null_dist attribute to the results<br>
> dataset? Or should I just give up on that, because trying to save the<br>
> distribution for each searchlight would result in a huge file?<br>
<br>
</span>Assuming 50k voxels (post-masking) and a desired resolution of 0.001<br>
from your nonparametric test, with 32-bit float representations, a null<br>
distribution would only require 200MB. To save that as an uncompressed<br>
nifti with dimensions 128x128x40 would require something closer to 5GB.<br>
Large, but within the bounds of memory on normal systems and easily<br>
within normal disk storage constraints.<br>
<br>
If you want higher resolution (e.g. the Stelzer et al. test suggests<br>
10^5 permutations), you'll need to multiply those figures by 100 and now<br>
you're out of memory by a long shot and filling half a terabyte per<br>
file. Our strategy for these kinds of problems is to create<br>
memory-mapped arrays and sort a few rows at a time. This is outside the<br>
scope of PyMVPA, but it's doable with numpy.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Christopher J Markiewicz<br>
Ph.D. Candidate, Quantitative Neuroscience Laboratory<br>
Boston University<br>
<br>
</font></span><br>_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" rel="noreferrer" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br></blockquote></div><br></div></div>