<div dir="ltr">Hi,<br><br>I see. The error rate was the best with the full set of features, so no features were selected. However, then I don't understand how I achieve the selection of fixed number of features using RFE. More specifically:<br>
<br>1. I would like to get 30 features, based on which I get the best prediction. I don't care that with 31 features (or 3022) I will get a better prediction. Isn't your graph Fig.1 in "Full Brain Classiï¬cation: There Is No “Face†Identiï¬cation Area" paper was the result of such analysis? In addition, attached the output which I get from classification of another dataset,
which resulted in desired 30 features. However, I am confused to
understand what I see there. If I read out it correctly, starting from
step 1, one subset of 30 features was selected and classified all the
time. What about the other possible subsets? How RFE knows that it is
the best one? It just picked the best ranks from original 3022? I am
not sure that it is very optimal. If you have some working example of correct / optimal RFE usage, I would very appreciate you sending me.<br><br>2. Unfortunately, even after reading Guyon 2002, I feel that I don't fully understand RFE algorithm. Particularly, what is the size of the original features subset, that algorithm starts with? Does it really start with full features set, although for 1000 voxels it is an evident overfitting? The solution with 3022 voxels, which I got, is not going to generalize well (given that I have 480 trials only), what is a benefit from such a solution? Any reference, which will clarify me all those issues are more than welcomed.<br>
<br>Thank you for your assistance.<br>I really consider using PyMVPA, because I was impressed by robustness of this software. However, although your doc is well written and organized, I am still got stuck in some places.<br>
<br>Vadim<br><br><br><div class="gmail_quote">2009/4/26 Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com">debian@onerussian.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
actually I should have discovered the problem before asking you to<br>
upload the data...<br>
<br>
in your code you use<br>
N_FEATURES = 30<br>
...<br>
<div class="im"> Â Â Â feature_selector=FixedNElementTailSelector(N_FEATURES,<br>
     tail='upper', mode='select'),<br>
<br>
<br>
</div>so you aren't doing RFE per se ;) you just select 30 features right<br>
on first step of RFE.... then, those 30 features lead to higher<br>
generalization error than if you took all of them, therefore initial<br>
dataset with all features is taken as the result.<br>
<br>
to see that you had just to enable RFE debug target (or all RFE ones)<br>
with<br>
<br>
debug.active += ['RFE.*']<br>
<br>
to see what is happening:<br>
<br>
In [12]:## working on region in file /tmp/python-8102meB.py...<br>
[RFEC ] DBG: Â Â Â Â Â Step 0: nfeatures=3022<br>
[RFEC ] DBG: Â Â Â Â Â Step 0: nfeatures=3022 error=0.2125 best/stop=1/0<br>
[RFEC_] DBG: Â Â Â Â Â Sensitivity: [-0.00507313 Â 0.00025722 Â 0.00159871 ..., -0.00212875 Â 0.00078268<br>
 -0.00027174], nfeatures_selected=30, selected_ids: [ 120  338  341  356  462  472  483  501  517  571  573  574  594  612  619<br>
 634  635  636  659  676  677  760  778  779  796  872 1109 1338 1545 1677]<br>
[RFEC ] DBG: Â Â Â Â Â Step 1: nfeatures=30<br>
[RFEC ] DBG: Â Â Â Â Â Step 1: nfeatures=30 error=0.2500 best/stop=0/0<br>
[RFEC_] DBG: Â Â Â Â Â Sensitivity: [ 0.09779742 Â 0.16359045 Â 0.02775154 Â 0.09486282 -0.0804099 Â -0.04392221<br>
 -0.06721182  0.09752928  0.03872871  0.08811431  0.14541801  0.13167303<br>
 0.13925132  0.03046704  0.04748648  0.09525846 -0.04226041  0.06917038<br>
 0.03207438  0.06333298  0.01423283  0.02703152  0.16574083  0.05634531<br>
 0.11383484  0.03402658  0.07105218 -0.02116503  0.24369252  0.20591227], nfeatures_selected=30, selected_ids: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24<br>
 25 26 27 28 29]<br>
[RFEC ] DBG: Â Â Â Â Â Step 2: nfeatures=30<br>
[RFEC ] DBG: Â Â Â Â Â Step 2: nfeatures=30 error=0.2500 best/stop=0/0<br>
[RFEC_] DBG: Â Â Â Â Â Sensitivity: [ 0.09779742 Â 0.16359045 Â 0.02775154 Â 0.09486282 -0.0804099 Â -0.04392221<br>
 -0.06721182  0.09752928  0.03872871  0.08811431  0.14541801  0.13167303<br>
 0.13925132  0.03046704  0.04748648  0.09525846 -0.04226041  0.06917038<br>
 0.03207438  0.06333298  0.01423283  0.02703152  0.16574083  0.05634531<br>
 0.11383484  0.03402658  0.07105218 -0.02116503  0.24369252  0.20591227], nfeatures_selected=30, selected_ids: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24<br>
 25 26 27 28 29]<br>
[RFEC ] DBG: Â Â Â Â Â Step 3: nfeatures=30<br>
[RFEC ] DBG: Â Â Â Â Â Step 3: nfeatures=30 error=0.2500 best/stop=0/0<br>
[RFEC_] DBG: Â Â Â Â Â Sensitivity: [ 0.09779742 Â 0.16359045 Â 0.02775154 Â 0.09486282 -0.0804099 Â -0.04392221<br>
 -0.06721182  0.09752928  0.03872871  0.08811431  0.14541801  0.13167303<br>
 0.13925132  0.03046704  0.04748648  0.09525846 -0.04226041  0.06917038<br>
 0.03207438  0.06333298  0.01423283  0.02703152  0.16574083  0.05634531<br>
 0.11383484  0.03402658  0.07105218 -0.02116503  0.24369252  0.20591227], nfeatures_selected=30, selected_ids: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24<br>
 25 26 27 28 29]<br>
<br>
....<br>
<br>
see original RFE definition on how to actually do RFE ;) or just try SMLR<br>
which might be more efficient, who knows ;)<br>
<div><div></div><div class="h5"><br>
<br>
On Sat, 25 Apr 2009, Yaroslav Halchenko wrote:<br>
<br>
> at first I thought that I know what is the reason, but then I realized<br>
> that it shouldn't be... didn't test though. to expedite things would you<br>
> mind uploading your data + code to the address I will provide you in a<br>
> followup email? ;)<br>
<br>
> On Sat, 25 Apr 2009, Vadim Axel wrote:<br>
<br>
> > Â Â Hi,<br>
> > Â Â I implemented some simple RFE logic, similar to what was described<br>
> > Â Â here: [1]<a href="http://www.pymvpa.org/featsel.html" target="_blank">http://www.pymvpa.org/featsel.html</a><br>
> > Â Â At the end of the classification procedure, I verify the the features<br>
> > Â Â that were selected based on what was described here:<br>
> > Â Â [2]<a href="http://www.pymvpa.org/faq.html#how-do-i-know-which-features-were-fin" target="_blank">http://www.pymvpa.org/faq.html#how-do-i-know-which-features-were-fin</a><br>
> > Â Â ally-selected-by-a-classifier-doing-feature-selection<br>
> > Â Â Now the problem: sometimes the resulted number of selected features is<br>
> > Â Â the exact number, which is required (I use FixedNElementTailSelector),<br>
> > Â Â whereas in some other case, for completely unknown reason, I get full<br>
> > Â Â set of features. The issue is really weired, since for two sessions of<br>
> > Â Â a subject I get selected feature set, but for two other sessions of the<br>
> > Â Â same subject I get full feature set. I suspect, that the problem might<br>
> > Â Â be in updating the feature_ids variable and not with classification,<br>
> > Â Â because the classification error rate was pretty low.<br>
> > Â Â Attached my code. Is it any problem with it?<br>
> > Â Â I can also upload my dataset (~50 Mb zip). I didn't succeed to<br>
> > Â Â reproduce it with smaller amount of data.<br>
> > Â Â Thanks for your help,<br>
> > Â Â Vadim<br>
<br>
> > СÑылки<br>
<br>
> > Â Â 1. <a href="http://www.pymvpa.org/featsel.html" target="_blank">http://www.pymvpa.org/featsel.html</a><br>
> > Â Â 2. <a href="http://www.pymvpa.org/faq.html#how-do-i-know-which-features-were-finally-selected-by-a-classifier-doing-feature-selection" target="_blank">http://www.pymvpa.org/faq.html#how-do-i-know-which-features-were-finally-selected-by-a-classifier-doing-feature-selection</a><br>
--<br>
Yaroslav Halchenko<br>
Research Assistant, Psychology Department, Rutgers-Newark<br>
Student  Ph.D. @ CS Dept. NJIT<br>
Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171<br>
    101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102<br>
WWW: Â Â <a href="http://www.linkedin.com/in/yarik" target="_blank">http://www.linkedin.com/in/yarik</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div>