<div dir="ltr"><div style>and would it be possible to get predicted probabilities instead of just labeled predictions? Something similar to 'predict_prob' from SVS in scikit-learn?</div><div style><br></div></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 31, 2014 at 9:44 PM, Arman Eshaghi <span dir="ltr"><<a href="mailto:arman.eshaghi@gmail.com" target="_blank">arman.eshaghi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks so much!<div><br></div><div>All the best,</div><div>Arman</div></div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 31, 2014 at 9:22 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com" target="_blank">debian@onerussian.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br>
On Fri, 31 Jan 2014, Arman Eshaghi wrote:<br>
<br>
</div><div>> MyData is structural MRI data coming from fmri_dataset function. There are<br>
> two chunks, and similar to clf.predictions (in tutorial), I'm wondering<br>
> whether I can get each predicted label, because I want to compare AUC in<br>
<br>
</div>so each sample is a subject. ok<br>
cvte.stats.sets would have sets of original targets and their<br>
predictions per each cross-validation split.<br>
<br>
also if you set your errorfx=None I guess you would also get raw<br>
predictions (and possibly original targets) in your results... yeap:<br>
<br>
In [2]: cv = CrossValidation(kNN(), HalfPartitioner(attr='chunks'), errorfx=None, enable_ca=['stats'])<br>
<br>
In [3]: from mvpa2.testing.datasets import datasets as tdatasets<br>
<br>
In [4]: results = cv(tdatasets['uni2small'])<br>
<br>
In [5]: results<br>
Out[5]: <Dataset: 24x1@|S2, <sa: cvfolds,targets>><br>
<br>
In [6]: print results.targets, results.samples<br>
['L0' 'L0' 'L0' 'L0' 'L0' 'L0' 'L1' 'L1' 'L1' 'L1' 'L1' 'L1' 'L0' 'L0' 'L0'<br>
'L0' 'L0' 'L0' 'L1' 'L1' 'L1' 'L1' 'L1' 'L1'] [['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L0']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L1']<br>
['L1']]<br>
<br>
*In [8]: print cv.ca.stats.sets<br>
[(array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',<br>
'L1'],<br>
dtype='|S2'), array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',<br>
'L1'],<br>
dtype='|S2'), [{'L0': 1.0, 'L1': 1.0}, {'L0': 1.0, 'L1': 1.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}]), (array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1', 'L1',<br>
'L1'],<br>
dtype='|S2'), array(['L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L0', 'L1', 'L1', 'L1', 'L1',<br>
'L1'],<br>
dtype='|S2'), [{'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 2.0, 'L1': 0.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}, {'L0': 0.0, 'L1': 2.0}])]<br>
<br>
and here are some snippets for you for AUC (you need a classifier which<br>
would provide estimates, not just final decisions):<br>
<br>
*In [10]: print cv.ca.stats.stats['AUC']<br>
[nan, nan]<br>
<br>
*In [11]: cv = CrossValidation(SMLR(enable_ca=['estimates']), HalfPartitioner(attr='chunks'), errorfx=None, enable_ca=['stats'])<br>
<br>
In [12]: results = cv(tdatasets['uni2small'])<br>
<br>
In [13]: print cv.ca.stats.stats['AUC']<br>
[1.0, 1.0]<br>
<br>
In [14]: tdatasets['uni2small'].samples += np.random.normal(size=tdatasets['uni2small'].shape)*0.5<br>
<br>
In [15]: results = cv(tdatasets['uni2small'])<br>
<br>
In [16]: print cv.ca.stats.stats['AUC']<br>
[0.81944444444444442, 0.81944444444444442]<br>
<br>
*In [17]: results = cv(tdatasets['uni4small'])<br>
<br>
In [18]: print cv.ca.stats.stats['AUC']<br>
[1.0, 1.0, 1.0, 1.0]<br>
<br>
*In [19]: tdatasets['uni4small'].samples += np.random.normal(size=tdatasets['uni4small'].shape)*0.5<br>
<br>
In [20]: results = cv(tdatasets['uni4small'])<br>
<br>
In [21]: print cv.ca.stats.stats['AUC']<br>
[0.64814814814814814, 0.68518518518518512, 0.76388888888888884, 0.55092592592592593]<br>
<div><div><br>
<br>
--<br>
Yaroslav O. Halchenko, Ph.D.<br>
<a href="http://neuro.debian.net" target="_blank">http://neuro.debian.net</a> <a href="http://www.pymvpa.org" target="_blank">http://www.pymvpa.org</a> <a href="http://www.fail2ban.org" target="_blank">http://www.fail2ban.org</a><br>
Senior Research Associate, Psychological and Brain Sciences Dept.<br>
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755<br>
Phone: <a href="tel:%2B1%20%28603%29%20646-9834" value="+16036469834" target="_blank">+1 (603) 646-9834</a> Fax: <a href="tel:%2B1%20%28603%29%20646-1419" value="+16036461419" target="_blank">+1 (603) 646-1419</a><br>
WWW: <a href="http://www.linkedin.com/in/yarik" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org" target="_blank">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>