[pymvpa] Train and test on different classes from a dataset

[Yaroslav Halchenko]

On Thu, 31 Jan 2013, J.A. Etzel wrote:

> This reply caught my eye. My answer would be that a, b, or c could
> be fine. For example, you might want to consider the condition that
> made the A/B or C/D as the 'truth', in which case it might make
> sense to just permute the other labels. And if you want to classify
> both directions (train A/B test C/D and train C/D test A/B) it might
> be most sensible to permute both sets of labels.
> 
> Assuming that the label permutations are done sensibly within the
> structure of the data (such as within runs and/or within people), my
> preference is usually to permute both the training AND testing set
> labels
> (http://mvpa.blogspot.com/2012/12/which-labels-to-permute.html).
> 
> Why do you say in the tutorial that "Doing a whole-dataset
> permutation is a common mistake ..." ? I don't see that permuting
> the test set labels hurts the inter-sample dependencies ... won't I
> still have (say) 5 A and 5 B in my test set?

There it was not only about "test set" but about the "whole-dataset"
(i.e.  traing+test sets).  


-- 
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik