[Shootout-list] Re: Timing variations

[Bengt Kleberg]

On 2005-05-24 19:21, John Skaller wrote:
> On Mon, 2005-05-23 at 09:56 +0200, Bengt Kleberg wrote:

...deleted
>>that could be quantified if we find out the granularity of the timer. 
>>and decide upon a accuracy level. ie, we would still be encountering 
>>measurement noise but we would know how much it is.
> 
> 
> No. I think you are 'barking up the wrong tree'.
> 
> I have just done some preliminary experiments with some test,
> and I have found on my slow x86 box variations of +/- 20%.
> This is pretty serious and suggests averaging just 3 runs is
> inadequate -- and we're assuming here tests of the order
> of 5 seconds run time where the timer resoltion is definitely
> adequate.

you are correct. if we have smallest run times that are adequatly longer 
than the granularity of the timer i agree that it is important to 
discuss how many times to run the test.

while we are still using the granularity of the timer as an accceptable 
shortest time i will continue to bark at that practice.


bengt