Lies, damned lies and benchmarking

[sawyl]

In his classic paper on benchmark skewing, Twelve Ways to Fool the Masses, Bailey cites as an example the dubious practice of presenting the timings from the inner most parts of a solver and reporting them as though they're the performance of the entire application.

It seems the lesson has been well learnt: one of the cell processor benchmarks has been caught playing with time in the same way. Thus, the application reports that performance scales linearly as more PEs are enabled, whilst the wall-clock time of the application remains relatively constant thanks to the dominance of start-up costs etc.

Very naughty.

Comments

[doctor-squale.livejournal.com]

Funny article, though incomplete.
There are many other ways to tweak the results.

All these techniques are of course very immoral, and I would only use them
under undue pressure.

[sawyl.livejournal.com]

I seem to remember someone requesting a series of very dubious SFS local cache tunings in order to get an IO bound benchmark through some acceptance tests. I wonder who that might have been. (Un)fortunately, I can't quite seem to remember...