Benchmarking

A benchmark is a measurement system. It can report a precise number while measuring removed work, an unstable sample, or an uncontrolled machine. This talk argues you should trust a Go benchmark only after answering three questions: Is the compiler measuring real work? Is the sample stable enough? Is the difference large relative to the noise? We work through three layers. Compiler honesty covers dead-code elimination, sinks, constant folding, inlining, timer ordering, and the B.Loop construct, which manages the timer for you and keeps the loop body from being optimized away. Statistical interpretation covers repeated samples, benchstat, coefficient of variation, run-count discipline, and the p-hacking traps that inflate false positives. Environment control covers both local machines and CI: Linux frequency and isolation controls, perflock, benchdiff, and the bare-metal runners that shared CI instances are not. ...

A benchmark bot once told me that one of my pull requests made a benchmark 6–9% slower. A same-machine comparison said the pull request made it faster. Both results were stable, and they disagreed. In this last part we’ll find out how that happens, and leave with three questions and three small Go tools that tell us how far to trust a number. A loose cable Physicists have been fooled the same way, at a much larger scale. In September 2011, the OPERA collaboration announced that muon neutrinos appeared to travel faster than the speed of light. Months of rechecking found nothing wrong. The root cause, eventually, was an improperly seated fibre-optic connector in the GPS timing chain, which introduced a ~73 ns bias that made neutrinos appear to arrive early (Science called it a loose cable). A second fault, an oscillator defect, pushed the other way and partly masked the first. Once both were corrected, the 2012 re-measurements showed neutrino speed consistent with the speed of light. ...

We’re going to take one benchmark and stop looking at it as a pair of numbers. First we’ll see why a pair misleads, then we’ll build a small time series and read it. Start with a pair that misled me. On dd-trace-go #4891, the benchmark bot’s comment now says BenchmarkOTLPProtoSize/1span is 7.6% to 8.0% faster than its baseline. Eleven days after #4891 was opened, on #4926, the bot said the same sub-benchmark was 8.1% to 8.5% slower. Both comments are correct arithmetic over two sets of runs, and the baselines differ: 05d9b8a on the first, 1e830f6 on the second. Same benchmark, opposite verdicts. 🔀 ...

Part 4 ended with a promise: the PR gate blocks only when benchstat says the delta is real and bigger than a floor we chose. A fair reader question is how. benchstat prints a table, and I checked that it exits 0 even when the table says +199.00% (two made-up result files, the pinned version, Go 1.27.1). Who, then, turns that table into a red check? We’ll build that job together. We’ll write one deliberately slow commit, let the job judge it, revert the commit, and let the job judge the revert. Two runs, two verdicts, both on a GitHub-hosted runner: ...

Say we put a benchmark gate on every pull request: if performance regresses, the check fails. After a week, something strange shows up. A PR that does nothing performance-sensitive fails the gate. Another PR, one that rewrites a hot path, sails through with a green check. Nobody is imagining it. Both outcomes are correct given the data, and the data is just wrong. A shared CI runner is a multi-tenant machine, so our benchmark lands on a host that also runs other teams’ compile jobs, Docker builds and test suites. A real 10% regression can vanish into that noise, and a phantom one can appear from nowhere. The gate isn’t measuring our code. It’s measuring the lottery of what happens to be running next door. 🎰 ...