Benchmarking

Here is a pull request with two numbers for the same change. My machine said the multi scenario cost 230% over a plain build. CI said 81%. Both numbers were honest readings of a stopwatch. I think only one of them was measuring the pull request. We’ll work out which, and find the one question that would have told me before CI did. This is a companion to the Why Your Go Benchmarks Are Lying series, between Part 3 (the laptop) and Part 4 (CI). Those parts use microbenchmarks. This story is about a macrobenchmark, a whole build timed end to end, and it didn’t fit my GopherCon UK talk. Disclosure: I work at Datadog and I’m one of otelc’s maintainers. ...

Every benchmark number starts life on somebody’s laptop. We write the function, run it, like what we see and open the pull request, long before CI or a pinned runner gets a say. That makes it the least controlled measurement in the pipeline, and the one we act on. If it lies, everything downstream inherits the lie. Sending a noisy benchmark to CI doesn’t fix the noise. It industrialises it. ...

Here is one benchmark that gave us two different answers for the same code. We’re going to take that run apart: see what benchstat can and can’t say about it, add the one number it doesn’t report, and decide how many runs it takes before a result earns any trust. Let’s call it the run that lied. The benchmark is BenchmarkMakeBuffer_Correct, run on a machine with sixteen CPU-bound background processes competing for the same cores. Here are two of its twenty runs (-16 is the GOMAXPROCS suffix), with the B/op and allocs/op columns dropped: ...

We’re going to chase down one benchmark result that looks suspiciously good. The function under test calls make([]byte, 64) every time it runs, and the benchmark reports 0.34 ns/op, 0 B/op, 0 allocs/op. That is almost three billion heap allocations per second on a laptop, and the allocator counted none of them. (It comes from a real run, more on that below.) Keep it in your pocket; we’ll follow it the whole way. ...

We’re going to write one small benchmark, run it, and read every word of what comes back. By the end you’ll know where ns/op comes from, what -count and -benchtime change, what the -16 is, and how to ask the compiler and the profiler what the benchmark really did. Here is the line we’ll be decoding: 1 BenchmarkMakeBuffer-16 99632604 18.53 ns/op 64 B/op 1 allocs/op This is a primer before part 1 of Why Your Go Benchmarks Are Lying, the written companion to my GopherCon UK talk, Why Your Go Benchmarks Are Lying. The series argues about numbers like the one above, so it helps to know how they are made first. If you have run go test -bench before, skim. ...