<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'><div dir='ltr'>
<div>Hi Yaroslav,</div><div><br></div>It seems working but I'll have to wait until the final results come out and see how they look like. <div><br></div><div>Thanks!</div><div>Meng</div><div><br><div>> Date: Tue, 6 Dec 2011 14:11:05 -0500<br>> From: debian@onerussian.com<br>> To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> Subject: Re: [pymvpa] Permutation test for each entry of contribution matrix?<br>> <br>> quick and ugly-ish answer... try it... but I also spotted some nuisance<br>> in our implementation which we might adjust (that the original error<br>> sample is "included" while we estimate CDF for its value within H0<br>> distribution... then we get a problem if classifier during permutation<br>> was spitting out the same values because there is no signal... then<br>> CDF(chance)=1.0 since all values are the same and we would have chosen<br>> low p for the right tail).<br>> <br>> Meanwhile -- try this which I will push into a unittest. does it provide<br>> "feasible" results? ;) I have used upcoming mvpa2 , but probably it should<br>> work with 0.6 as well.<br>> <br>> class ConfusionMatrixError(object):<br>> """Custom error "function"<br>> """<br>> def __init__(self, labels=None):<br>> self.labels = labels<br>> def __call__(self, predictions, targets):<br>> cm = ConfusionMatrix(labels=list(self.labels),<br>> targets=targets, predictions=predictions)<br>> ## print cm.matrix<br>> # We have to add a degenerate leading dimension<br>> # so we could separate them into separate 'samples'<br>> return cm.matrix[None, :]<br>> <br>> ds = datasets['uni2small'] # choose your data<br>> <br>> clf = GNB() <br>> num_perm = 10<br>> permutator = AttributePermutator('targets',<br>> limit='chunks',<br>> count=num_perm)<br>> cv = CrossValidation(<br>> clf, NFoldPartitioner(),<br>> errorfx=ConfusionMatrixError(labels=ds.sa['targets'].unique),<br>> postproc=mean_sample(),<br>> null_dist=MCNullDist(permutator,<br>> tail='left',<br>> enable_ca=['dist_samples']),<br>> enable_ca=['stats'])<br>> cmatrix = cv(ds)<br>> ## print cmatrix.samples<br>> cvnp = cv.ca.null_prob.samples<br>> <br>> <br>> On Tue, 06 Dec 2011, Meng Liang wrote:<br>> <br>> > Hi Yaroslav,<br>> > Thanks for the quick response! I'm using Pymvpa 0.6. <br>> > Best,<br>> > Meng<br>> <br>> > > Date: Tue, 6 Dec 2011 11:10:15 -0500<br>> > > From: debian@onerussian.com<br>> > > To: pkg-exppsy-pymvpa@lists.alioth.debian.org<br>> > > Subject: Re: [pymvpa] Permutation test for each entry of contribution<br>> > matrix?<br>> <br>> > > Good thinking Meng Liang -- that is what we also encourage people to do<br>> > > to look at the actual confusion matrix where possible... to help you<br>> > > best, please let us know which version of PyMVPA you are using?<br>> <br>> > > On Tue, 06 Dec 2011, Meng Liang wrote:<br>> <br>> > > > Dear all,<br>> > > > I am trying to run a permutation test for a four-way classification<br>> > and<br>> > > > I'm interested in the classification accuracy of each class rather<br>> > than<br>> > > > the overall accuracy averaged across classes. I tried MCNullDist but<br>> > it<br>> > > > only gives me the null distribution of the overall accuracy. Is there<br>> > a<br>> > > > way to get the null distribution of the accuracy of each class? The<br>> > best<br>> > > > thing would be to get the null distribution of each entry of the<br>> > confusion<br>> > > > matrix so that I can calculate null distribution for any measure based<br>> > on<br>> > > > the confusion matrix. <br>> > > > Any suggestions?<br>> > > > Best,<br>> > > > Meng<br>> <br>> > > > _______________________________________________<br>> > > > Pkg-ExpPsy-PyMVPA mailing list<br>> > > > Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> <br>> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br>> <br>> <br>> > > --<br>> > > =------------------------------------------------------------------=<br>> > > Keep in touch www.onerussian.com<br>> > > Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic<br>> <br>> > > _______________________________________________<br>> > > Pkg-ExpPsy-PyMVPA mailing list<br>> > > Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> <br>> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br>> <br>> > _______________________________________________<br>> > Pkg-ExpPsy-PyMVPA mailing list<br>> > Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br>> <br>> <br>> -- <br>> =------------------------------------------------------------------=<br>> Keep in touch www.onerussian.com<br>> Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic<br>> <br>> _______________________________________________<br>> Pkg-ExpPsy-PyMVPA mailing list<br>> Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org<br>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa<br></div></div> </div></body>
</html>