Research about Benchmarking

[Sze]

@mnemnion

Each of the metrics in that link is a single number. When you run a benchmark, you get one best time, one median, one mean, one envelope (say, 3x difference between best and worst), and so on. There might be hundreds of metrics, but it’s one measure per metric that gets reported. Each benchmark gets run a bunch of times, but the purpose of that is to reduce random sources of variance. Many columns, one row.

I think that the presentation of the results, maybe could be done better then just some numbers, if we can produce measurements that can be described by normal distributions.

So I don’t necessarily disagree that these simple metrics seem typical, but maybe instead of seeing this as how it is, we should wonder whether we should/could apply the techniques of Stabilizer^[1]?

Then if we have normal distributions showing these with graphs in some way (maybe over many different runs of the benchmarks) could not only show the measurements but also some of the information about how precise these measurements are or maybe we even could calculate more with that?.

But looking at the wikipedia article only makes me realize how much I don’t know Normal distribution - Wikipedia.

---

1. Performance Matters by Emery Berger
   https://www.youtube.com/watch?v=r-TLSBdHe1A
   Stabilizer | Emery Berger ↩︎