# Go benchmarks that lie: reading, trusting, and gating them > [!summary] TL;DR > A Go benchmark number can be wrong three ways: the compiler optimized the work away, the machine noise drowned the signal, or the difference was too small to matter. The series asks three questions of every number: did the compiler let it measure real work, is the sample stable, is the difference large relative to the noise. The answers run from `ns/op` basics and the sink pattern through `benchstat` statistics, laptop and CI noise, to a pull request gate that fails only when benchstat calls the delta real. ## Start here 1. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Quickstart|Quickstart]]: What does each field of a `go test -bench` line mean, and how is the iteration count chosen? 2. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Compiler Honesty and Dead Code Elimination|Compiler honesty]]: Why does a benchmark of `make([]byte, 64)` report `0.34 ns/op` and `0 allocs/op`, and how do you fix it? 3. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Statistics and benchstat|Statistics and benchstat]]: The same code gave 39.39 and 27.54 ns/op; what do medians, `±` and CV say about which numbers to trust? 4. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local reproduction on a laptop]]: What do an idle Mac, a saturated one and a pinned container cost in noise? 5. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Laptop Said 230 Percent|Laptop said 230 percent]]: One pull request read ~230% overhead on a loaded laptop and +81% in CI; which number was real? 6. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI noise]]: Why do shared CI runners add noise, and what does a bare-metal runner buy? 7. [[resources/documentation/Go/Benchmarks/Go Benchmarks - AB Testing Is the Wrong Model for CI|AB testing is the wrong model for CI]]: Why does a per-PR A/B pair miss slow drift and fire on one noisy night? 8. [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|PR gate that fails]]: How do you build a GitHub Actions gate that fails only on real regressions? 9. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|Three questions before you trust a benchmark]]: What checks catch a CI regression that was really a speedup? ## Key points - `go test -run XXX -bench . -benchmem` prints `ns/op`, `B/op`, `allocs/op` per run. The testing package ramps `b.N` until the run fills `-benchtime` (default `1s`); the `-16` suffix is `GOMAXPROCS`. - A benchmark reporting `0.34 ns/op, 0 B/op, 0 allocs/op` for a function calling `make([]byte, 64)` measures an empty loop: once the call is inlined, dead-code elimination deletes the body. The sink pattern or `for b.Loop()` (Go 1.24+) keeps the work alive; the honest cost is about `15 ns/op` and `1 allocs/op`. - One run is one mean. With `-count=3` the slowest sample was about 35% slower than the fastest. - Same code and binary, 43% apart (39.39 and 27.54 ns/op): the machine shifted between regimes with sixteen CPU-bound processes running (the likely, unmeasured cause). Collect `-count=20`, summarize with `benchstat` (median plus 95% confidence interval). - CV = standard deviation / mean. Rule of thumb: under 2% reliable, 2 to 5% acceptable, 5 to 10% investigate the environment, over 10% do not trust the comparison. Idle host 4.75% CV; host with 16 background spinners 18.88%. - One benchmark, three conditions on an Apple M4 Max: idle 11.46 ns/op at 4.75% CV, with 16 spinners 34.97 ns/op at 18.88%, and in a container pinned to one vCPU with the same spinners 16.28 ns/op at 5.25%. - One otelc pull request read ~230% build overhead on the author's loaded laptop and +81% in CI. The scenario the change could not touch (`largeidle`, ~212%) moved as much, which pointed at the machine. CI's only real failure was `baseline` at +514.9% to +521.9%, attributed to the SDK dependency bump. - A shared CI runner is a multi-tenant machine. With SMT enabled, two CPU-bound tasks sharing a core showed a CV around 23%; on bare metal with SMT and frequency scaling disabled, runs drop below 0.25%. - A per-PR A/B compares two runs, but a benchmark history is a time series. In a synthetic sixty-night series (a Go program generated it, no real benchmark did), a 5% night-over-night gate fired twice on one noisy night and never on three 1.5% steps that added up to about 4.6%. - A working PR gate: build base and head binaries, run them interleaved (base, head, base, head) for 10 rounds at 200 ms, fail only when `benchstat` calls the delta real (p < 0.05) and at least +5%. `benchstat` exits 0 even on a +199.00% table, so the workflow parses its output. - Three questions before trusting a number: is the compiler measuring real work, is the sample stable (CV under ~5%, `-count=10` or more), is the difference large relative to the noise (`benchstat` p < 0.05)? Three CLIs in `kakkoyun/benchlab` (honestbench, benchgate, benchenv) check each one. - A CI bot flagged a dd-trace-go pull request as 6 to 9% slower; a same-machine A/B showed it faster. The likely cause was code layout, which alone can move results by ±10%. - Series numbers come from one machine unless stated: Apple M4 Max, Go 1.27.1, checked on 2 October 2026. The host's Go version was not recorded for the laptop three-conditions capture. ## FAQ ### Why does my Go benchmark report 0 allocs/op? Inlining let dead-code elimination delete the call: the loop is empty. Keep the result alive with a package-level sink, or use `for b.Loop()` (Go 1.24+), and always run with `-benchmem`. Details in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Compiler Honesty and Dead Code Elimination|Compiler Honesty and Dead Code Elimination]] ### What does the -16 suffix mean in benchmark output? It is `GOMAXPROCS`, dropped when the value is 1. A missing suffix in someone else's output means they ran with `GOMAXPROCS=1`. Field-by-field reading in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Quickstart|Go Benchmarks - Quickstart]] ### How many times should I run a Go benchmark? Ten is the practical floor, twenty for CI; below six samples `benchstat` prints `± ∞`. Summarize with `benchstat` and check CV in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Statistics and benchstat|Statistics and benchstat]] ### Can I trust a benchmark run on my laptop? Sometimes. An idle Mac gave 4.75% CV, sixteen background spinners 18.88%, and a container pinned to one vCPU 5.25% with the spinners still running. Good for catching obvious regressions during development, not for publication. The three conditions in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] ### Why did CI report a regression my laptop did not show? Either the shared runner is noisy (multi-tenant host, SMT contention can push CV near 23%), or the laptop was the noisy one: one pull request read ~230% overhead on a loaded laptop against +81% in CI, and an untouched scenario moving the same way pointed at the machine, not the diff. See [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] and [[resources/documentation/Go/Benchmarks/Go Benchmarks - Laptop Said 230 Percent|Laptop Said 230 Percent]] ### How do I gate benchmarks in a pull request? Build base and head binaries, run them interleaved for 10 rounds at 200 ms, fail only when `benchstat` calls the delta real (p < 0.05) and at least +5%. `benchstat` always exits 0, so parse its output. The full workflow in [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|PR Gate That Fails]] ### Is a per-PR benchmark A/B comparison enough? No. A benchmark history is a time series: in a synthetic sixty-night series (a Go program generated it, no real benchmark did), a 5% gate misses three 1.5% steps that add up to about 4.6% and fires on one noisy night. Change-point detection over the whole series does the opposite, per [[resources/documentation/Go/Benchmarks/Go Benchmarks - AB Testing Is the Wrong Model for CI|AB Testing Is the Wrong Model for CI]] ### How do I know a benchmark number is real? Check three things: the compiler measured real work, the sample is stable, the difference is large relative to the noise. The three checks, and the CLIs that run them, in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|Three Questions Before You Trust a Benchmark]] ## Related ## Sources - [Why Your Go Benchmarks Are Lying (GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/)