<html>
<body>
<font color="#800000">We have a study dataset with subject, label
(stroke/no stroke), and 60 features; I'd like to make an SVM classifier
and test its significance, most important features, etc. I get results,
but also a few cryptic (to me) errors and some warnings.<br><br>
Also, if I try NFoldPartitioner() rather than HalfPartitioner(...) I get
a traceback about missing chunks, so it seems I need to set them
explicitly?<br><br>
I also can't get searchlight set up correctly.<br><br>
<br>
My test code, based on the tutorial:<br>
</font><tt><font face="Courier New, Courier" size=1>from
mvpa2.tutorial_suite import *<br><br>
d = file(r"C:\temp\test3_redo.csv").readlines()<br>
lol= [x[:-1].split(",") for x in d]<br>
print lol[0]<br>
## the list of subject names<br>
subjects = [r[0] for r in lol]<br>
## the feature data<br>
dat = [[float(c) for c in row[6:]] for row in lol]<br><br>
labels = [r[1] for r in lol]<br>
tmp = [l.replace('Normal', '0') \<br>
for l in [l.replace('Stroke',
'1') for l in labels]]<br>
## the truth values<br>
labels = [int(x) for x in tmp]<br><br>
ds = Dataset(samples=dat)<br>
ds.sa['subject'] = subjects<br>
ds.sa['targets'] = labels<br>
print ds, '\n'<br><br>
clf = LinearCSVMC()<br>
cvte = CrossValidation(clf, HalfPartitioner(count=2, <br>
selection_strategy='random', attr='subject'), <br>
errorfx=lambda p, t: np.mean(p == t),
enable_ca=['stats'])<br>
cv_results = cvte(ds)<br>
print cvte.ca.stats.as_string(description=True)<br>
print cvte.ca.stats.matrix<br><br>
aov = OneWayAnova()<br>
f = aov(ds)<br>
print 'aov:', f<br><br>
fsel = SensitivityBasedFeatureSelection(<br>
OneWayAnova(),<br>
FixedNElementTailSelector(5, mode='select',
tail='upper'))<br>
fsel.train(ds)<br>
ds_p = fsel(ds)<br>
print '\nfixed:', ds_p.shape<br><br>
results = cvte(ds_p)<br>
print np.round(cvte.ca.stats.stats['ACC%'], 1)<br>
print cvte.ca.stats.matrix<br>
print <br><br>
fsel = SensitivityBasedFeatureSelection(<br>
OneWayAnova(),<br>
FractionTailSelector(0.05, mode='select',
tail='upper'))<br>
fclf = FeatureSelectionClassifier(clf, fsel)<br>
cvte = CrossValidation(fclf, HalfPartitioner(count=2, <br>
selection_strategy='random', attr='subject'),<br>
enable_ca=['stats'])<br>
results = cvte(ds)<br>
print 'fractional', np.round(cvte.ca.stats.stats['ACC%'], 1)<br><br>
<br>
</font></tt><font color="#800000">Errors:<br>
</font><font face="MS Shell Dlg 2, Geneva" size=1 color="#400000">'gcc'
is not recognized as an internal or external command,<br>
operable program or batch file.<br>
C:\Python27\lib\site-packages\scipy\integrate\quadpack.py:288:
UserWarning: Extremely bad integrand behavior occurs at some points of
the<br>
integration interval.<br>
warnings.warn(msg)<br>
C:\Python27\lib\site-packages\mvpa2\misc\errorfx.py:102: RuntimeWarning:
invalid value encountered in divide<br>
([0], np.cumsum(t)/t.sum(dtype=np.float), [1]))<br>
C:\Python27\lib\site-packages\scipy\stats\stats.py:274: RuntimeWarning:
invalid value encountered in double_scalars<br>
return np.mean(x,axis)/factor<br>
C:\Python27\lib\site-packages\mvpa2\misc\errorfx.py:106: RuntimeWarning:
invalid value encountered in divide<br>
([0], np.cumsum(~t)/(~t).sum(dtype=np.float), [1]))<br>
C:\Python27\lib\site-packages\mvpa2\clfs\transerror.py:678:
RuntimeWarning: invalid value encountered in divide<br>
stats['PPV'] = stats['TP'] / (1.0*stats["P'"])<br>
C:\Python27\lib\site-packages\mvpa2\clfs\transerror.py:679:
RuntimeWarning: invalid value encountered in divide<br>
stats['NPV'] = stats['TN'] / (1.0*stats["N'"])<br>
C:\Python27\lib\site-packages\mvpa2\clfs\transerror.py:680:
RuntimeWarning: invalid value encountered in divide<br>
stats['FDR'] = stats['FP'] / (1.0*stats["P'"])<br>
C:\Python27\lib\site-packages\mvpa2\measures\anova.py:111:
RuntimeWarning: invalid value encountered in divide<br>
msb = ssbn / float(dfbn)<br><br>
<br><br>
<br>
</font><font color="#800000">Output:<br>
</font><font face="MS Shell Dlg 2, Geneva" size=1>['S001', 'Stroke',
'Structural', 'DL', 'A3+4', 'L', '33175.5142', '14408.18074',
'10849.84165', '8059.24706', '8010.452299', '14', '45', '40', '55', '50',
'56060.79132', '24908.80989', '16687.6343', '10154.6501', '7901.745475',
'14', '45', '40', '50', '30', '64268.60726', '12620.57744',
'992.4884881', '825.5158143', '751.0024413', '19', '27', '33', '67',
'40', '2170.966193', '1879.560843', '1741.498856', '1340.718439',
'959.5283252', '32', '15', '19', '42', '23', '0', '0', '0', '0', '0',
'0', '0', '0', '0', '0', '13424.62045', '9142.678538', '8140.41212',
'6403.125282', '5807.041425', '15', '19', '66', '32', '41']<br>
<Dataset: 62x60@float64, <sa: subject,targets>> <br><br>
WARNING: Only 1 sets have estimates assigned from 2 sets. ROC estimates
might be incorrect.<br>
* Please note: warnings are printed only once, but underlying
problem might occur many times *<br>
----------.<br>
predictions\targets 0.0 1<br>
`------ ---- ---- P' N' FP FN
PPV NPV TPR SPC FDR MCC
F1 AUC<br>
0.0
0 23 23 39 23
24 0 0.38 0 0.39
1 -0.61 0 nan<br>
1
24 15 39 23 24 23
0.38 0 0.39 0 0.62 -0.61 0.39
nan<br>
Per target:
---- ----<br>
P
24 38<br>
N
38 24<br>
TP
0 15<br>
TN
15 0<br>
Summary \ Means: ---- ---- 31 31 23.5 23.5
0.19 0.19 0.2 0.2 0.81 -0.61 0.19 nan<br>
CHI^2 25.68 p=1.1e-05<br>
ACC 0.24<br>
ACC% 24.19<br>
# of
sets 2<br><br>
Statistics computed in 1-vs-rest fashion per each target.<br>
Abbreviations (for details see
http://en.wikipedia.org/wiki/ROC_curve):<br>
TP : true positive (AKA hit)<br>
TN : true negative (AKA correct rejection)<br>
FP : false positive (AKA false alarm, Type I error)<br>
FN : false negative (AKA miss, Type II error)<br>
TPR: true positive rate (AKA hit rate, recall, sensitivity)<br>
TPR = TP / P = TP / (TP + FN)<br>
FPR: false positive rate (AKA false alarm rate, fall-out)<br>
FPR = FP / N = FP / (FP + TN)<br>
ACC: accuracy<br>
ACC = (TP + TN) / (P + N)<br>
SPC: specificity<br>
SPC = TN / (FP + TN) = 1 - FPR<br>
PPV: positive predictive value (AKA precision)<br>
PPV = TP / (TP + FP)<br>
NPV: negative predictive value<br>
NPV = TN / (TN + FN)<br>
FDR: false discovery rate<br>
FDR = FP / (FP + TP)<br>
MCC: Matthews Correlation Coefficient<br>
MCC = (TP*TN - FP*FN)/sqrt(P N P' N')<br>
F1 : F1 score<br>
F1 = 2TP / (P + P') = 2TP / (2TP + FP +
FN)<br>
AUC: Area under (AUC) curve<br>
CHI^2: Chi-square of confusion matrix<br>
LOE(ACC): Linear Order Effect in ACC across sets<br>
# of sets: number of target/prediction sets which were
provided<br><br>
[[ 0 23]<br>
[24 15]]<br>
aov: <Dataset: 1x60@float64, <fa: fprob>><br><br>
fixed: (62, 5)<br>
WARNING: Obtained degenerate data with zero norm for training of
<LinearCSVMC>. Scaling of C cannot be done.<br>
61.3<br>
[[ 0 0]<br>
[24 38]]<br><br>
fractional 61.3<br><br>
</font></body>
</html>