[pymvpa] confusion matrix in searchlight

[Michael Hanke]

Hey,

On Wed, Mar 19, 2014 at 10:44 PM, Wouter De Baene
<Wouter.DeBaene at ugent.be>wrote:

> Hi all,
>
> I’m running my first pymvpa-analysis after attending a very nice pymvpa
> workshop in Delmenhorst.
>

Glad you liked it!


> Although there are a few related questions in the archive, I don’t seem to
> get this working.
> I’m setting up a searchlight analysis with Nfold cross-validation. I'm
> trying to look at the confusion matrix for each cross-validation for each
> searchlight. This information seems to be lost when using searchlights.
> As I understood there were some plans to aggregate the output per
> searchlight into the output of the total results. Is there any news about
> this? OR could I get to this information in another way for now?
>

The easiest way to aggregate this kind of non-scalar information within a
searchlight is to
get the confusion matrix to come out of the CrossValidation as a vector
(flattened matrix).
For the first part of a possible solution we have "Confusion", a node that
turns raw targets
predictions into a confusion table. Here is a sketch: no error computing in
the classifier, return
raw predictions, use Confusion as a post-processing node to yield matrices.

     CrossValidation(some_classifier,
                      some_partitioner,
                      errorfx=None,
                      postproc=Confusion())

Now the last bit is to flatten the confusion matrix into a vector. You
could write a simple Node
implementation that does that. Here is some pseudo code:

class MyFlatMatrix(Node):
  def _call(ds):
    return Dataset(ds.samples.ravel())

and now use this class as a post-processing node for Confusion itself.

    CrossValidation(some_classifier,
                      some_partitioner,
                      errorfx=None,
                      postproc=Confusion(postproc=MyFlatMatrix()))

stick this into a searchlight and it should harvest what you want. This is
all written as a sketch and wasn't tested...

Instead of a custom Node class you could also use a ChainMapper comprised
of a FlattenMapper and a TransposeMapper
to achieve the same thing....

Good luck!

Michael


-- 
Michael Hanke
http://mih.voxindeserto.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/pkg-exppsy-pymvpa/attachments/20140320/497e3f8b/attachment.html>