[pymvpa] Train and test on different classes from a dataset

[Michael Hanke]

On Fri, Feb 1, 2013 at 9:41 PM, Yaroslav Halchenko <debian at onerussian.com>wrote:

> Thanks Michael,
>
> It is a nice example showing that we better permute  testing set
> as well.  Otherwise, the stronger our signal is -- worse we become in
> detecting it (kinda awkward isn't it?).
>

That is not the case. For the higher SNR value the accuracy is at 100% and
the
associated p-value is at the theoretical minimum.


> e.g. with 2 chunks and SNR 30.00 (i.e. where signal is obviously
> there and classification on unpermutted labels probably is stable 100%)
> -- we would not be able to tell if that is "significant" if we maintain
> original order in the testing set while doing permutation testing.
>

Not the case -- if you run the script and increase the bin size you can see
that the p-value
is a nice one ;-)


> now (if you are already asleep, I will try doing it later) - just extend
> it into the power (or ROC detection) analysis -- what we are actually
> after here.  Add SNR=0.00 -- collect both types of samples and
> describe each plot with its power to detect the true signal using both
> kinds of permutation.   Ideally your power should grow with SNR ;)


I am not asleep, but I am having a good time doing nothing ;-)

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20130202/4674f37c/attachment.html>