[Shootout-list] Directions of various benchmarks

[Bengt Kleberg]

On 2005-05-19 18:26, John Skaller wrote:
> On Thu, 2005-05-19 at 13:22 +0200, Bengt Kleberg wrote:

...deleted
>>this approach (what and how is in the specification) makes it simpler to 
>>design tests, too. this is worth remembering.
> 
> 
> Yes but this is the status quo, and resulted in a test Felix was
> doing well in being disallowed arbitrarily and unfairly,  and half
> a dozen other tests being allowed to stand even though they're
> demonstrably cheating: regexp is one, another (possibly deprecated now)
> required maxiumum 4K I/O buffer but the Ocaml code explicitly set
> the buffer to 8K .. in one case (Python was it?) performance
> was improved by simply not running destructors/garbage collection
> at the end of the test .. but that code wasn't accepted ..

i do not agree that it is the current practice (status quo?) of ''what 
and how is in the specification'', that is the main reason that 
implementrations are disallowed arbitrarily and unfairly. i would 
instead put the blame on the persons doing the disallowing.

i would argue that if the what and how says 4k buffer, and the entry 
still is allowed, this proves my point. i do not think it proves your 
point. (this is built upon if i i have correctly understood that your 
point. i think it is:
what and how is the reason for tests being allowed to stand even though 
they're demonstrably cheating).


bengt