[pymvpa] Train and test on different classes from a dataset

[Francisco Pereira]

I never tried that, and I still need to read your experiment in
detail. My point was about the statement one can make about what the
test is doing. The only paper I can pitch in with is tangential to the
discussion (i.e. not about permutation tests but about the statements,
in particular the discussion of variability in results)

http://www.lri.fr/~mary/dietterich98approximate.ps.gz

Francisco

On Tue, Feb 5, 2013 at 8:51 AM, Yaroslav Halchenko
<debian at onerussian.com> wrote:
> Hi Francisco,
>
> Great that you followed up -- could you please clarify for me if indeed
> in one of your publications you did some power/ROC analysis of such
> permutation scheme (keeping testing set labels assignment) against
> "classical" (permute all independent assignments)?  I have vague memory
> that you did but I could be wrong.
>
> NB A will argue with Michael in reply to his post ;)
>
> On Tue, 05 Feb 2013, Francisco Pereira wrote:
>> I'm catching up with this long thread and all I can say is I fully
>> concur with Michael, in particular:
>
>> On Tue, Feb 5, 2013 at 3:11 AM, Michael Hanke <mih at debian.org> wrote:
>> > Why are we doing permutation analysis? Because we want to know how
>> > likely it is to observe a specific prediction performance on a
>> > particular dataset under the H0 hypothesis, i.e. how good can a
>> > classifier get at predicting our empirical data when the training
>> > did not contain the signal of interest -- aka chance performance.
>
>> Permuting the test set might make sense, perhaps, if you wanted to
>> make a statement about the result variability over all possible test
>> sets of that size if H0 was true.
>
> --
> Yaroslav O. Halchenko
> Postdoctoral Fellow,   Department of Psychological and Brain Sciences
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa