[Shootout-list] Directions of various benchmarks

[skaller]

On Tue, 2005-05-17 at 14:02, Jon Harrop wrote:
> On Tuesday 17 May 2005 04:35, skaller wrote:
> > On Tue, 2005-05-17 at 10:53, Jon Harrop wrote:
> > > I disagree. Anything which fits well into an interpreted language (e.g.
> > > regexps in Perl) will be both fast and succinct.
> >
> > How interesting is it to compare Perl/PCRE with C/PCRE
> > with Python/PCRE with Ocaml/PCRE .. is it interesting to
> > compare PCRE with PCRE??
> 
> Why does the OCaml implementation of regex current use Markus' PCRE rather 
> than OCaml's built-in Str module?

Perhaps it doesn't, but the point is the same: we're not
testing the language involved in ANY of these cases,
in all cases we're comparing C libraries.

> I expect everyone would agree. The problem is that clarity of code and 
> developer effort are inherently qualitative. LOC is much more objective and 
> quantitative but still suffers from "do we or don't we count lines with a 
> single brace on in C/C++/Java/C#" etc.?

Yes, but as I have pointed out a few times here the actual
performance measurements are ALSO just as flaky! 

For example I just checked ackermann and takfp on my AMD64,
and guess what? Felix TRASHES gcc on both by a significant
margin!

Hey hey, isn't Felix compiled by gcc? Yeah, well Felix
uses -O3 -fomit-frame-pointer .. and gcc with those options
trashes Felix.

So we could just use LOC until we have a better measure.
Token count, for example (requires a lexer though).
One could add up the number of named identifiers -- that's
indicative too, then penalise people for NOT using
identifiers by subtracting points based on the size
of the largest expression :)

My opinion is: we should count lines, and ALL lines including
comments and blanks, and make sure the tests are all equally
nicely spaced and commented.

> I like the idea of asking authors to emphasise performance when <100 LOC but 
> emphasise brevity when >100 LOC. 

Interesting. However that would encourage things like manually
unfolding ackermann's function in languages with a compact
representation but which were hopeless at optimisation... 

> > Hmm .. I wonder about a performance vs. loc picture?
> > This would be a two dimensional grid with an X for each
> > language..
> 
> That's an absolutely excellent idea! 

Except that 'dots' on graphs are hard to see :)


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net