Research about Benchmarking

One thing I didn’t add to the post, is that it seems to me that benchmarking and profiling really kind of belong together and possibly could be two sides of one coin, opposites of a spectrum.

I think in an ideal world you could easily switch from interacting with a program to find where something is slow, to then looking at how it behaves there and essentially leaving a probe/marker there (kind of like a permanent breakpoint for performance monitoring) and that would essentially create a benchmark.

While I think that the distinction makes sense to be able to communicate what you are focusing on, I think better tools could blur the lines a lot.

Also a lot of the benchmarking seems to reinvent profiler things, just in a program first kind of way, instead of through allowing you to create benchmarks directly you need to write them as code, this reminds me a bit of this talk Bret Victor - The Future of Programming

1 Like