[pymvpa] how to do multiprocessing in pymvpa?

[Meng Liang]

Hi Michael and Jonas,

Thanks very much for your prompt reply! Jonas's suggestion solved the problem - If I do not put 'nproc = 3' explicitly in the searchlight command, I always get the error "IOError: [Errno 2] No such file file or directory: '/proc/cpuinfo'".

Thanks again!

Best,
Meng

From: jtkaplan at usc.edu
Date: Mon, 4 Feb 2013 10:14:45 -0800
To: pkg-exppsy-pymvpa at lists.alioth.debian.org
Subject: Re: [pymvpa] how to do multiprocessing in pymvpa?

I have pprocess working on Mac OSX for searchlights. The trick is that it fails when checking the number of cores, so you have to specify nproc. 
Jonas

On Feb 4, 2013, at 9:57 AM, Meng Liang <meng.liang at hotmail.co.uk> wrote:





Hi Nick,

Thanks so much for your reply! Following your suggestion, I downloaded and installed the pprocess on Neurodebian virtual machine on a Windows 7 PC and now the script runs much faster! 

However, it does not work on Mac. I googled but didn't find much relevant info about it - I got the feeling that pprocess is not supported in Mac. Is it true? If so, doesn't that mean running pymvpa on Mac machines cannot utilize multi-core computation?

Finally, an irrelevant question: is the value of 'radius' parameter in the Searchlight command in millimeter? It says so in Pymvpa documentation ( http://www.pymvpa.org/modref/mvpa.measures.searchlight.html ). But in a given example on Pymvpa website, it seems to suggest that the value of 'radius' means the number of voxels ( http://www.pymvpa.org/examples/searchlight_minimal.html ): "# setup searchlight with 2 voxels radius" and "sl = sphere_searchlight(cv, radius=2, space='myspace',...". It's a bit confusing to me...

Also, do you know if multi-core support is also available for other commands other than searchlight? I find that cross-validation with large number of permutations is also quite slow - it would be good if multi-core support is also implemented in the permutation scenario. 

Best,
Meng



> Date: Mon, 4 Feb 2013 14:17:10 +0100
> From: nikolaas.oosterhof at unitn.it
> To: pkg-exppsy-pymvpa at lists.alioth.debian.org
> Subject: Re: [pymvpa] how to do multiprocessing in pymvpa?
> 
> On 4 February 2013 14:09, Meng Liang <meng.liang at hotmail.co.uk> wrote:
> > I'm trying to run a searchlight analysis using the whole brain mask but I
> > found it impossible to finish within a reasonable time - it has been running
> > three days for one two-way classification task in one subject, and it still
> > hasn't finished. To make the computing faster, would multiprocessing be an
> > option? I didn't see any part of the tutorial mentioning multiprocessing -
> > is it because it is already embedded/implemented in pymvpa commands?
> 
> Yes, multi-core support is already part of pyMVPA's searchlight
> implementation. It requires, however, the presence of pprocess [1].
> The Searchlight constructor has an optional parameter 'nproc' that
> controls the number of parallel processes to use. By default
> (nproc=None) it uses the number of cores of the machine - unless
> pprocess is not available, in which case nproc=1.
> 
> As per the example (doc/examples/searchlight.py) it is recommended to
> enable the searchlight debug messages - these include messages related
> to  parallelization.
> 
> In my experience it's easy to use pprocess with neurodebian. On a mac
> (OS X Lion) I did not get it to work, although I have to admit I did
> not try very hard.
> 
> [1] http://pypi.python.org/pypi/pprocess
> 
> 
> 
> >
> > Best,
> > Meng
> >
> > _______________________________________________
> > Pkg-ExpPsy-PyMVPA mailing list
> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
> 
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
 		 	   		  
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130204/e7f3bf6f/attachment.html>