<div dir="ltr"><div><div>Nick, I think I have in the region of 250000 voxels! All I did was extract the voxels with a brain mask. I didn't realise that this would be so memory consuming. So is ridge regression n^2m memory usage for n features and m subjects? <br><br></div>Michael, I just use the following:<br><br>ds = Dataset(newMetaData['maskedData'])<br><a href="http://ds.sa">ds.sa</a>['diagnoses'] = newMetaData['diagnosesT1']<br><br></div>and then my ds.summary() is<br><br><pre>Dataset: 133x217105@float32, <sa: diagnoses,runtype>, <a: mapper> stats: mean=-2.9664e-06 std=0.966266 var=0.93367 min=-5.76981 max=11.4395</pre><br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 24, 2014 at 5:14 PM, Michael Hanke <span dir="ltr"><<a href="mailto:michael.hanke@gmail.com" target="_blank">michael.hanke@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>could you please show the code that assembles the dataset? Also, the output of</div><div><br></div><div>print ds.summary()</div><div><br></div><div>could help to find the problem.</div><div><br></div><div>Michael</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Nov 24, 2014 at 4:35 PM, Thomas Nickson <span dir="ltr"><<a href="mailto:thomas.nickson@gmail.com" target="_blank">thomas.nickson@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div>Hey All, <br><br>I'm using some structural data, 133 subjects with a
size of about 350 meg, on a machine with 32gig of ram. I'm trying to
set up a basic algorithm to make sure that everything works okay. I have
chosen ridge regression and I'm attempting to use a HalfPartitioner and
I'm running all of the code from ipython notebook.<br><br> from mvpa2.clfs.ridge import RidgeReg<br> a = HalfPartitioner(attr='runtype')<br> clf = RidgeReg()<br> cv = CrossValidation(clf, a)<br> cv_results = cv(ds) <br><br></div>However, I get the following memory error, even when I use a very minimal subset such as 2 subjects.<br><br><pre><span>---------------------------------------------------------------------------</span>
<span>MemoryError</span> Traceback (most recent call last)
<span><ipython-input-205-fafb290e8469></span> in <span><module></span><span>()</span>
<span>----> 1</span><span> </span>cv_results <span>=</span> cv<span>(</span>ds<span>)</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 257</span> <span>"used and auto training is disabled."</span><span></span>
<span> 258</span> % str(self))
<span>--> 259</span><span> </span><span>return</span> super<span>(</span>Learner<span>,</span> self<span>)</span><span>.</span>__call__<span>(</span>ds<span>)</span><span></span>
<span> 260</span> <span></span>
<span> 261</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 119</span> <span></span>
<span> 120</span> self<span>.</span>_precall<span>(</span>ds<span>)</span><span></span>
<span>--> 121</span><span> </span>result <span>=</span> self<span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 122</span> result <span>=</span> self<span>.</span>_postcall<span>(</span>ds<span>,</span> result<span>)</span><span></span>
<span> 123</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 495</span> <span># always untrain to wipe out previous stats</span><span></span><span></span>
<span> 496</span> self<span>.</span>untrain<span>(</span><span>)</span><span></span>
<span>--> 497</span><span> </span><span>return</span> super<span>(</span>CrossValidation<span>,</span> self<span>)</span><span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 498</span> <span></span>
<span> 499</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 324</span> ca<span>.</span>datasets<span>.</span>append<span>(</span>sds<span>)</span><span></span>
<span> 325</span> <span># run the beast</span><span></span><span></span>
<span>--> 326</span><span> </span>result <span>=</span> node<span>(</span>sds<span>)</span><span></span>
<span> 327</span> <span># callback</span><span></span><span></span>
<span> 328</span> <span>if</span> <span>not</span> self<span>.</span>_callback <span>is</span> None<span>:</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 257</span> <span>"used and auto training is disabled."</span><span></span>
<span> 258</span> % str(self))
<span>--> 259</span><span> </span><span>return</span> super<span>(</span>Learner<span>,</span> self<span>)</span><span>.</span>__call__<span>(</span>ds<span>)</span><span></span>
<span> 260</span> <span></span>
<span> 261</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/node.pyc</span> in <span>__call__</span><span>(self, ds)</span>
<span> 119</span> <span></span>
<span> 120</span> self<span>.</span>_precall<span>(</span>ds<span>)</span><span></span>
<span>--> 121</span><span> </span>result <span>=</span> self<span>.</span>_call<span>(</span>ds<span>)</span><span></span>
<span> 122</span> result <span>=</span> self<span>.</span>_postcall<span>(</span>ds<span>,</span> result<span>)</span><span></span>
<span> 123</span> <span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/measures/base.pyc</span> in <span>_call</span><span>(self, ds)</span>
<span> 598</span> for i in dstrain.get_attr(splitter.get_space())[0].unique])
<span> 599</span> <span># ask splitter for first part</span><span></span><span></span>
<span>--> 600</span><span> </span>measure<span>.</span>train<span>(</span>dstrain<span>)</span><span></span>
<span> 601</span> <span># cleanup to free memory</span><span></span><span></span>
<span> 602</span> <span>del</span> dstrain<span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/base/learner.pyc</span> in <span>train</span><span>(self, ds)</span>
<span> 130</span> <span># things might have happened during pretraining</span><span></span><span></span>
<span> 131</span> <span>if</span> ds<span>.</span>nfeatures <span>></span> <span>0</span><span>:</span><span></span>
<span>--> 132</span><span> </span>result <span>=</span> self<span>.</span>_train<span>(</span>ds<span>)</span><span></span>
<span> 133</span> <span>else</span><span>:</span><span></span>
<span> 134</span> warning<span>(</span><span>"Trying to train on dataset with no features present"</span><span>)</span><span></span>
<span>/home/orkney_01/tnickson/Programming/pyVirtualEnv/lib/python2.6/site-packages/mvpa2/clfs/ridge.pyc</span> in <span>_train</span><span>(self, data)</span>
<span> 76</span> <span>if</span> self<span>.</span>__lm <span>is</span> None<span>:</span><span></span>
<span> 77</span> <span># Not specified, so calculate based on .05*nfeatures</span><span></span><span></span>
<span>---> 78</span><span> </span>Lambda <span>=</span> <span>.05</span><span>*</span>data<span>.</span>nfeatures<span>*</span>np<span>.</span>eye<span>(</span>data<span>.</span>nfeatures<span>)</span><span></span>
<span> 79</span> <span>else</span><span>:</span><span></span>
<span> 80</span> <span># use the provided penalty</span><span></span><span></span>
<span>/usr/lib64/python2.6/site-packages/numpy/lib/twodim_base.pyc</span> in <span>eye</span><span>(N, M, k, dtype)</span>
<span> 208</span> <span>if</span> M <span>is</span> None<span>:</span><span></span>
<span> 209</span> M <span>=</span> N<span></span>
<span>--> 210</span><span> </span>m <span>=</span> zeros<span>(</span><span>(</span>N<span>,</span> M<span>)</span><span>,</span> dtype<span>=</span>dtype<span>)</span><span></span>
<span> 211</span> <span>if</span> k <span>>=</span> M<span>:</span><span></span>
<span> 212</span> <span>return</span> m<span></span>
<span>MemoryError</span>: <br><br><br></pre><pre>Surely, this can't really be because I'm out of memory can it? The 32gig machine should be <br></pre><pre>more than able to load the a small portion of the 350 meg dataset.<br><br></pre><pre>Is there something I'm missing or have I configured something wrongly?<br><br></pre><pre>Also, I'm not really sure how the halfpartitioner works. Does it just generate the complement <br></pre><pre>of the sets I determine or can it split the dataset in two for me based on the <br></pre><pre>groups? If that's possible, how can I do that?<br><br></pre><pre>Thanks,<br><br>Tom </pre></div>
<br></div></div><span class="">_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div>Michael Hanke</div><div><a href="http://mih.voxindeserto.de" target="_blank">http://mih.voxindeserto.de</a></div><div><div><br></div></div></div></div>
</font></span></div>
<br>_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br></blockquote></div><br></div>