Hi Yaroslav,<div><br></div><div>Thanks for the help! I've answered your questions for this message inline below. I'm currently running a couple analyses to address questions from your 2nd message… should be ready to go by tomorrow.</div>
<div><br></div><div>Best,</div><div>Mike<br><div><br><div class="gmail_quote">On Thu, Dec 22, 2011 at 12:31 PM, Yaroslav Halchenko <span dir="ltr"><<a href="mailto:debian@onerussian.com">debian@onerussian.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mike,<br>
<br>
First of all thanks for looking into the details and sharing it here --<br>
it is an important and under-explored topic imho. My comments are below<br>
-- sorry if they are not very well structured<br>
<div class="im"><br>
> With old z-scoring, C=-5:<br>
> Accuracies are around 60% with a heavy selection bias towards the rest<br>
> condition (chooses "rest" correctly 27/27 times, but also chooses rest for<br>
> 21/27 sound conditions).<br>
<br>
</div>am I taking you right -- you have multi-class classification (multiple<br>
'sound conditions') here? or did you collapse all non-rest conditions<br>
into 1?<br></blockquote><div><br></div><div>Sorry, I should have made this more clear. For this sanity check test, I got rid of 2 of my 3 "sound" conditions. The experiment involved 81 presentations of each of 3 sound conditions (243 total), and 81 presentations of of rest/baseline. After some averaging, this is reduced to 27 of each. For this basic test, I was comparing 27 examples of one of these sounds against the 27 rest (just a single pairwise classifier)… so all should be balanced in numbers. In the problematic subject who I'm currently looking at/testing, I had to throw out a couple of experimental runs, so the numbers are actually 21 and 21, which is reflected in the summary output below. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
by "any of the sound conditions vs. the silence condition at 98-100" did<br>
you mean separate pair-wise classifiers or multi-class?<br>
<br>
what is the output of print dataset.summary()?<br></blockquote><div><br></div><div><b>I'm pasting it here:</b></div><div><br></div><div><div>In [37]: print dataset.summary()</div><div>Dataset: 42x43049@float64, <sa: chunks,targets,time_coords,time_indices>, <fa: thr_glm,voxel_indices>, <a: imghdr,imgtype,mapper,voxel_dim,voxel_eldim></div>
<div>stats: mean=0.00947037 std=0.609248 var=0.371183 min=-2.06233 max=2.07801</div><div><br></div><div>Counts of targets in each chunk:</div><div> chunks\targets major rest</div><div> --- ---</div><div>
0.0 1 1</div><div> 1.0 1 1</div><div> 2.0 1 1</div><div> 3.0 1 1</div><div> 4.0 1 1</div><div> 5.0 1 1</div><div>
6.0 1 1</div><div> 7.0 1 1</div><div> 8.0 1 1</div><div> 9.0 1 1</div><div> 10.0 1 1</div><div> 11.0 1 1</div><div>
12.0 1 1</div><div> 13.0 1 1</div><div> 14.0 1 1</div><div> 15.0 1 1</div><div> 16.0 1 1</div><div> 17.0 1 1</div><div>
18.0 1 1</div><div> 19.0 1 1</div><div> 20.0 1 1</div><div><br></div><div>Summary for targets across chunks</div><div> targets mean std min max #chunks</div><div> major 1 0 1 1 21</div>
<div> rest 1 0 1 1 21</div><div><br></div><div>Summary for chunks across targets</div><div> chunks mean std min max #targets</div><div> 0 1 0 1 1 2</div><div> 1 1 0 1 1 2</div>
<div> 2 1 0 1 1 2</div><div> 3 1 0 1 1 2</div><div> 4 1 0 1 1 2</div><div> 5 1 0 1 1 2</div><div> 6 1 0 1 1 2</div><div> 7 1 0 1 1 2</div>
<div> 8 1 0 1 1 2</div><div> 9 1 0 1 1 2</div><div> 10 1 0 1 1 2</div><div> 11 1 0 1 1 2</div><div> 12 1 0 1 1 2</div><div> 13 1 0 1 1 2</div>
<div> 14 1 0 1 1 2</div><div> 15 1 0 1 1 2</div><div> 16 1 0 1 1 2</div><div> 17 1 0 1 1 2</div><div> 18 1 0 1 1 2</div><div> 19 1 0 1 1 2</div>
<div> 20 1 0 1 1 2</div><div>Sequence statistics for 42 entries from set ['major', 'rest']</div><div>Counter-balance table for orders up to 2:</div><div>Targets/Order O1 | O2 |</div>
<div> major: 0 21 | 20 0 |</div><div> rest: 20 0 | 0 20 |</div><div>Correlations: min=-1 max=1 mean=-0.024 sum(abs)=41</div></div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
1. with dominance of rest condition samples (if you haven't<br>
ballanced it out) classifier might be just preferring 'rest' condition<br>
overall<br>
<br>
2. with multiclass you might also be hitting here the present problem of<br>
non-arbitrary breakage of the ties - thus leading to collapsing into<br>
'rest' condition. With just released mvpa2 -- how does it look if you<br>
try SMLR or kNN?<br></blockquote><div><br></div><div>So I don't think these 2 apply, as it should be balanced and not multi-class. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
<br>
> With old z-scoring, C=-1:<br>
> Same as above, except with accuracies around 54%<br>
> With NO z-scoring, C=-5 or C=-1:<br>
> Accuracies are 98-100%<br>
> With C=-5 (or C=-1) and zscore(dataset, chunks_attr='chunks',<br>
> dtype='float32'):<br>
> Accuracies are about 98%<br>
<br>
> So it looks like:<br>
> (a) Using C=-5 (as opposed to C=-1) helps a little with the zscore against<br>
> rest method. Although it might help across the board, but there's a<br>
> ceiling effect with the other combinations.<br>
<br>
</div>just to make it clear -- by 'old z-scoring' you meant z-scoring against<br>
rest condition, right? and in new one you just z-scored across all<br>
conditions (including rest) and it lead to good generalization...<br></blockquote><div><br></div><div>Yes…that's correct.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
question: what was exact line you used to z-score against rest<br>
condition? we might like also to check if everything implemented as it<br>
should but also it might simply be that due to smaller number of trials<br>
for rest condition alone, estimates of mean/variance were not stable<br>
thus leading to noisy 'standardized' samples thus lower performance.<br></blockquote><div><br></div><div><b>The old line was:</b></div><div>zscore(dataset, chunks_attr='chunks', param_est=('targets', ['rest']), dtype='float32')</div>
<div>which I just ripped off from one of the website's tutorial pages.</div><div>The new line removed the "param_est=('targets', ['rest'])" and left everything else the same. </div><div><br></div>
<div>I, too, am worried about the noisy baseline, though I'd thought that 25% of all trials should be plenty. (Additionally, it seems to act as a normal baseline in the GLM… auditory+ regions are activated in sound>silence contrast.)</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> (b) There's a huge difference between whether I zscore against rest or<br>
> with the whole time series. I'm not sure what's up... running sounds ><br>
> silence GLMs in FSL show obvious responses in the expected brain regions.<br></div></blockquote><div><br></div><div>Just to be clear: 9 runs, each containing 39 volumes (this was slow event-related sparse sampling). 3 of these were used for an orthogonal behavioral task and thrown out. The remaining 36 (in each run) were 9xsilence, 9xSound1, 9xSound2, and 9xSound3. For my "against silence" MVPA sanity check above, I threw out 2 of the sound conditions, so the remaining sound and silence should be balanced. For my more empirically interesting analyses, I've been throwing out the rest and one of the sound conditions, so the classifier looks at 2 balanced sound conditions.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
</div>so once again knowing number of samples in each chunk and how z-scoring<br>
against was done would help us to get better clue<br></blockquote><div><br></div><div>Thanks again! -Mike</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
<br>
--<br>
=------------------------------------------------------------------=<br>
Keep in touch <a href="http://www.onerussian.com" target="_blank">www.onerussian.com</a><br>
Yaroslav Halchenko <a href="http://www.ohloh.net/accounts/yarikoptic" target="_blank">www.ohloh.net/accounts/yarikoptic</a><br>
<br>
_______________________________________________<br>
Pkg-ExpPsy-PyMVPA mailing list<br>
<a href="mailto:Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org">Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org</a><br>
<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa" target="_blank">http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa</a><br>
</div></div></blockquote></div><br></div></div>