[pymvpa] multi-class major voting scheme paradox?

[josef.pktd at gmail.com]

On Fri, May 14, 2010 at 2:27 PM, Vadim Axel <axel.vadim at gmail.com> wrote:
> Fine, thanks.
> So, then, the interpretation of mutli-class confusion can be misleading.
> I have one more related question:
> Does it make sense to interpret the confusion matrix off-diagonal values as
> a tuning? For example, if my three classes are the circles of three
> different size. For the row of the large circle I get 0.6 classified as
> large (the correct one), 0.3 as a medium and 0.1 as a small. Can I say that
> I get sort of tuning to circle size?

I'm not sure I understand the term tuning in this context
(I'm not a classification expert, but I like puzzles.)

To me, these are just probabilities that depend on the distinguishing
information that is in the data, and how well the classifier is able
to distinguish the separating characteristics.

If your circles also have color, then a color sensitive classifier
might end up with a different confusion matrix than a size (or shape)
sensitive classifier. In that sense the classifier in your example is
"atuned" or sensitive to size (and size is a distinguishing
characteristic).

But as in your initial paradox, I could imagine other cases, e.g. the
middle sized circle is further away and clearly defined, while middle
and large (1 and 3) are fuzzy and close to each other, then the large
and small circles would be more often confused with each other, than
with 2.

I hope this makes some kind of sense.

Josef


>
> Thanks again.
>
>
> On Fri, May 14, 2010 at 4:29 PM, <josef.pktd at gmail.com> wrote:
>>
>> On Fri, May 14, 2010 at 9:20 AM, Vadim Axel <axel.vadim at gmail.com> wrote:
>> > Hi guys,
>> >
>> > I apply multi-class major voting scheme for three classes (all pairs
>> > classification). I try to understand how the confusion matrix should
>> > look
>> > like when two classes in a pair classification are not discriminated
>> > (chance
>> > level). Consider pathological case where classes 1,2 and 2,3 are
>> > classified
>> > with 100% and 1,3 are at chance level (50%). The confusion matrix I
>> > which
>> > get looks like:
>> > 0.584    0.083    0.333
>> > 0    1    0
>> > 0.327    0.071    0.602
>> >
>> > So, all of sudden it seems that classes 1 and 3 are discriminated. Isn't
>> > it
>> > paradoxical?
>> >
>> > When I checked out how I get this result, I have found that it indeed
>> > makes
>> > sense. Consider class 1 as a correct label:
>> > pair 1: the classification of classes 1,2 always results in '1' (we are
>> > at
>> > 100%, by definition)
>> > pair 2: the classification of classes 1,3 results in half trials in '1'
>> > and
>> > other half in '3' (we are at chance by definition).
>> > pair 3: the classification of classes 2,3 results in half trials in '2'
>> > and
>> > other in '3' (in case that classes are unrelated, the classifier should
>> > be
>> > at chance here).
>> >
>> > The bottom line: since all (1) pairs and half (2) pairs results in '1',
>> > I am
>> > already at 50% hit rate for correct class.
>> >
>> > What do you think about all this? Is there any flaw in my logic?
>> > If someone is interested, I can send my matlab simulation.
>>
>> looks right to me if the tie-breaker is unbiased
>>
>> Probs if 1 is tru:
>>
>> >>> 0.5*0.5 + 0.5*0.5/3.   # 3 wins, by majority and tie-breaker
>> 0.33333333333333331
>> >>> 0.5*0.5/3.   # 2 wins, by tie-breaker
>> 0.083333333333333329
>> >>> 1-(0.5*0.5 + 0.5*0.5/3. + 0.5*0.5/3.)  # 1 wins
>> 0.58333333333333337
>>
>> Josef
>>
>> >
>> > Thanks for help,
>> > Vadim
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Pkg-ExpPsy-PyMVPA mailing list
>> > Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> > http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>> >
>> >
>>
>> _______________________________________________
>> Pkg-ExpPsy-PyMVPA mailing list
>> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
>> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
>
> _______________________________________________
> Pkg-ExpPsy-PyMVPA mailing list
> Pkg-ExpPsy-PyMVPA at lists.alioth.debian.org
> http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa
>
>