# Three questions before you trust a Go benchmark result
> [!summary] TL;DR
> A CI bot flagged dd-trace-go PR #4891 as 6–9% slower; a same-machine A/B showed it was faster, and the likely cause was code layout, a systematic error that looks like signal. Before trusting a number, check three things: the compiler is measuring real work, the sample is stable, and the difference is large relative to the noise. Three small CLIs in benchlab (honestbench, benchgate, benchenv) check each one.
## Key points
- In June 2026 the dd-trace-go benchmark bot said PR #4891 made BenchmarkOTLPProtoSize 6–9% slower than main. A same-machine comparison said the pull request made it faster. Both results were stable, and they disagreed.
- The same-machine A/B on an Apple M4 Max (darwin/arm64), compared with benchstat: main 883.3 ns/op (1 span) and 7115 ns/op (10 spans); #4891 840.7 ns/op and 6775 ns/op. Faster.
- Local CV was under 0.1%, so the 6–9% gap was not ordinary variance.
- The timed loop only calls proto.Size(tracesData) on a struct assembled entirely before the loop. It never calls ContextWithSpan, SpanFromContext, or any code the PR modified.
- The likely mechanism is code layout: changing context.go shifted function addresses in the test binary, which moved the hot proto.Size loop relative to cache-line and branch-target-buffer boundaries.
- A later symbol comparison shows proto.MarshalOptions.size and impl.(*MessageInfo).sizePointer each moved by 96 bytes, and the benchmark closure by 160 (amd64 binaries, indicative only; no hardware-counter measurement ties the shift to the delta).
- Emery Berger's Performance Matters (Strange Loop 2019) puts code layout alone at ±10%.
- The bot's history fits: from 12 June each update of its comment said four regressions, 6–9% at every span count; on 19 June, after main had moved, the same comment said four improvements of 6–8%. A stable bias that changed sign when main moved.
- The resolution was nothing: no code change for the benchmark. A speculative fix would have been chasing shadows.
- The same benchmark tripped again on #4926, eleven days after #4891 was opened, with +6.5–8.5% on the same four sub-benchmarks. It was dismissed as a known false positive: a code-layout artifact, with a local A/B of about +0.3%.
- Question 1: is the compiler measuring real work? Verify a sink pattern, no discarded results, allocs/op > 0 when allocation is expected.
- Question 2: is my sample stable enough? Verify CV < ~5% and at least -count=10.
- Question 3: is the difference large relative to the noise? benchstat p-value < 0.05, an effect that matters, A/B on the same machine, CI used for detection.
- Each question gates the next: a benchmark the compiler optimised away answers question 2 with noise, and a noisy environment makes question 3 unanswerable whatever the sample size.
- The three CLIs live in kakkoyun/benchlab at v0.1.0, one stdlib-only Go module (go 1.24). Versions, links and commands checked on 2 October 2026.
## The CI regression that was a speedup
In June 2026 a change touching context.go in the ddtrace/tracer package landed as [dd-trace-go #4891](https://github.com/DataDog/dd-trace-go/pull/4891), compile-time instrumentation plumbing. Shortly after the push, the benchmark bot commented that BenchmarkOTLPProtoSize was 6–9% slower than main. Disclosure: the author works at Datadog, which maintains dd-trace-go, so the story is about the employer's CI bot. The lesson holds for any repository.
The entire timed loop of the benchmark:
```go
// The entire timed loop inside BenchmarkOTLPProtoSize:
for b.Loop() {
proto.Size(tracesData)
}
```
That is a protobuf size computation on a struct assembled entirely before the loop. The [real benchmark](https://github.com/DataDog/dd-trace-go/blob/1a0c5e19b611f34f14c14531304973c89d2fb055/ddtrace/tracer/otlp_writer_bench_test.go#L144) still calls b.ResetTimer() there, which b.Loop makes redundant. It never calls ContextWithSpan, SpanFromContext, or any code the PR modified, so the diff had no believable path to the number.
Running the benchmark repeatedly on the same machine gave a coefficient of variation under 0.1%, so a 6–9% gap was not ordinary variance. The same-machine A/B on an Apple M4 Max (darwin/arm64), compared with benchstat, gave these medians:
| Build | 1 span | 10 spans |
|-------|--------|----------|
| main | 883.3 ns/op | 7115 ns/op |
| #4891 | 840.7 ns/op | 6775 ns/op |
# 4891 was faster. CI had flagged a regression, and the same-machine A/B showed the opposite
The likely mechanism is code layout. Changing context.go shifted function addresses in the test binary, which moved the hot proto.Size loop relative to cache-line and branch-target-buffer boundaries. At the sub-microsecond scale of the one-span case, a small alignment shift can swing a result by a few percent in either direction, enough to flip the verdict from improvement to regression. Emery Berger's [Performance Matters](https://www.youtube.com/watch?v=r-TLSBdHe1A) (Strange Loop 2019) puts code layout alone at ±10%.
The bot's own history on the PR fits the layout reading. From 12 June, each update of its comment said four regressions, 6–9% at every span count. On 19 June, after main had moved, the same comment said four improvements of 6–8% for the same benchmark. That is not noise around zero: the bias was stable and changed sign when main moved. The resolution was nothing. No code change for the benchmark; a speculative fix to quiet it would have been chasing shadows.
These benchmarks ran on shared CI runners at the time and have since moved to dedicated bare-metal machines, which takes the noisy neighbours out of the picture. It does not take code layout out, and neither does a same-machine A/B, since the two binaries still differ in layout. CI said +6 to 9% and the local machine about −5% for a benchmark whose loop the PR never touched; a sign that flips between machines points at layout more than at code. Neither machine is authoritative by default; the flip is the tell to go and look at layout.
The same benchmark tripped again on [#4926](https://github.com/DataDog/dd-trace-go/pull/4926), eleven days after #4891 was opened, with +6.5–8.5% on the same four sub-benchmarks. That flag was dismissed on sight as a known false positive: a code-layout artifact, with a local A/B of about +0.3%.
## A systematic error can look like signal
In September 2011, the OPERA collaboration [announced](https://web.archive.org/web/20140222165941/http://press-archived.web.cern.ch/press-archived/PressReleases/Releases2011/PR19.11E.html) that muon neutrinos appeared to travel faster than the speed of light. Months of rechecking found nothing wrong. The root cause was an improperly seated fibre-optic connector in the GPS timing chain, which introduced a ~73 ns bias that made neutrinos appear to arrive early ([Science](https://doi.org/10.1126/science.335.6072.1027) called it a loose cable). A second fault, an oscillator defect, pushed the other way and partly masked the first. Once both were corrected, the 2012 re-measurements showed neutrino speed consistent with the speed of light.
A systematic measurement error can hide in plain sight, look exactly like signal, and survive review by people far more careful than we are. A number can be reproducible and still be directionally wrong, and a gate that is directionally wrong blocks good changes and waves bad ones through. Go benchmarks have cables too: the compiler, the statistics, and the machine with its OS scheduler.
## Three questions to verify before merging
| # | Question | What to verify |
| --- | -------- | ------------- |
| 1 | Is the compiler measuring real work? | Sink pattern present; no discarded results; allocs/op > 0 when allocation is expected |
| 2 | Is my sample stable enough? | CV < ~5%; at least -count=10 |
| 3 | Is the difference large relative to the noise? | benchstat p-value < 0.05 and an effect that matters; environment diagnosed; A/B on the same machine, since a sub-10% micro delta can be code layout; CI used for detection, not as the primary measurement |
Each question gates the next. A benchmark the compiler has optimised away answers question 2 with noise, and a noisy environment makes question 3 unanswerable whatever the sample size. A checklist nobody runs is decoration, so each question gets a tool. The [FOSDEM 2026 post on measuring software performance](https://kakkoyun.me/posts/fosdem-2026-measuring-software-performance/) is the prerequisite.
## The benchlab CLIs
The three CLIs are in [benchlab](https://github.com/kakkoyun/benchlab/tree/v0.1.0), one per question. Everything here describes v0.1.0, the tag pinned for the talk on 12 August 2026. They share one stdlib-only Go module (go 1.24). Pin the tag, because unreleased work on main changes some of these flags:
```bash
go install github.com/kakkoyun/benchlab/cmd/
[email protected]
```
### honestbench: question 1
It walks *_test.go files with go/ast and flags results discarded after computation (dead-code elimination candidates), missing sink patterns, StopTimer/StartTimer misordering, and b.N loops that should migrate to testing.B.Loop, introduced in Go 1.24. It exits 1 on findings, so it works as a CI gate:
```bash
honestbench -r ./...
```
-r recurses into subdirectories, -json prints machine-readable output, and -q prints findings only. Exit codes: 0 for clean, 1 for findings, 2 for an error. Run it before reading a single ns/op, because a finding on a b.N loop means the benchmark probably measures something other than what you think. Caveat: v0.1.0 also reports discarded results inside for b.Loop() bodies, where the [testing package](https://pkg.go.dev/testing#B.Loop) documents that call results are kept alive. That includes the proto.Size loop from the story above. Treat a finding there as a prompt to look, not as proof.
### benchgate: question 2
It runs benchmarks N times, computes the coefficient of variation per benchmark, and fails if any exceeds a threshold. It can also diff against a saved baseline through benchstat, which must be on your PATH (see [[resources/documentation/Go/Benchmarks/Go Benchmarks - Statistics and benchstat|Statistics and benchstat]]):
```bash
benchgate -pkg ./... -count 10 -cv-threshold 5.0
```
-pkg (default ./...) and -bench (default .) pick what to run, -count (default 10) says how many times, and -cv-threshold (default 5.0, in percent) sets the bar. To capture a baseline on the current branch and compare after a change, run it twice:
```bash
benchgate -pkg ./... -count 10 -save old.txt
# make your change
benchgate -pkg ./... -count 10 -baseline old.txt
```
The second run calls benchstat old.txt <new-output> for you and prints the comparison. The exit code comes from the CV check alone, so a +40% delta on a quiet sample still exits 0; gating on the delta is a step you write. A gate at 5% catches environments too noisy for a reliable A/B before time is wasted interpreting numbers. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] shows what SMT and frequency scaling do to CV.
### benchenv: question 3
It diagnoses the measurement environment: SMT state, CPU frequency governor, Turbo Boost, system load average, and which of perflock, benchstat and benchdiff are installed. It works across platforms and degrades gracefully on macOS, where sysfs controls are unavailable. Its only flag is -json:
```bash
benchenv
```
One run, an Apple M4 Max (darwin/arm64, Go 1.27.1) with benchenv from v0.1.0, on 2026-10-02. The load-average warning depends on what else the machine was doing, so yours will differ:
```text
benchenv: benchmarking environment diagnosis (darwin/arm64, 16 CPUs)
[unavailable] SMT control — macOS does not expose SMT control via sysfs — use a Linux machine or bare-metal CI runner for publication-quality numbers
[unavailable] CPU frequency governor — macOS does not expose a CPU frequency governor — use a Linux machine or bare-metal CI runner for publication-quality numbers
[unavailable] Turbo Boost — macOS does not expose Turbo Boost control from user space — use a Linux machine or bare-metal CI runner for publication-quality numbers
[warn] load average — close background applications before benchmarking
[unavailable] thermal pressure — macOS thermal state is not accessible from user space — watch for CPU throttling on sustained benchmark runs
[warn] perflock not installed — go install github.com/aclements/perflock@latest
[ok] benchstat installed — benchstat found on PATH
[warn] benchdiff not installed — go install github.com/willabides/benchdiff/cmd/benchdiff@latest
[ok] GOMAXPROCS / NumCPU — NumCPU=16 GOMAXPROCS=16
Summary: 2 ok, 3 warn, 4 unavailable. Fix warnings before trusting benchmark numbers.
```
Every [warn] line is a noise source or a missing tool, and the [unavailable] lines are macOS declining to say. The install hints print @latest; pin perflock and benchdiff the way [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] does. Run benchenv once at the start of a benchmarking session, fix the warnings, then run benchgate, then compare with benchstat.
## The minimum viable discipline
If you keep one thing from this series: run benchmarks ten times, not once. The whole loop needs only the Go toolchain and benchstat:
```bash
# Baseline on the current branch
go test -bench=. -benchmem -count=10 ./... | tee old.txt
# After your change
go test -bench=. -benchmem -count=10 ./... | tee new.txt
# Compare
benchstat old.txt new.txt
```
benchstat then tells you whether the difference clears the noise. That is the floor. For anything you report, use -count=20 (plus -benchtime=2s on a noisy Mac), and for CI baselines consider a fixed count (-benchtime=Nx) over part 4's time-based 2s and 5s. Around that loop go all three tools: honestbench -r ./... before reading any numbers, benchenv on any new machine or CI runner, and benchgate -cv-threshold 5.0 as a gate that fails early when the environment is too noisy for a reliable signal. Three tools, under an hour to wire up, for any Go project.
## FAQ
### What are the three questions before trusting a Go benchmark?
Is the compiler measuring real work, is my sample stable enough, and is the difference large relative to the noise? Each question gates the next: compiler-optimised code answers question 2 with noise, and a noisy environment makes question 3 unanswerable.
### Why did CI report a regression that was actually a speedup?
On dd-trace-go #4891 the CI bot said 6–9% slower while a same-machine A/B said faster. The timed loop never called code the PR modified. The likely cause was code layout: the change shifted function addresses, and the sign flipped when main moved. The resolution was no change at all.
### How much can code layout change a Go benchmark result?
Emery Berger's Performance Matters talk puts code layout alone at ±10%. In the #4891 case the evidence is circumstantial: the hot functions moved by 96 bytes and the benchmark closure by 160, but those are amd64 binaries, not necessarily what CI ran, and no hardware-counter measurement ties the shift to the delta.
### What does honestbench check?
It walks *_test.go files with go/ast and flags discarded results, missing sink patterns, StopTimer/StartTimer misordering, and b.N loops that should migrate to testing.B.Loop. Exit codes: 0 clean, 1 findings, 2 error. Caveat: v0.1.0 also reports discarded results inside for b.Loop() bodies, where the testing package keeps results alive, so treat a finding there as a prompt to look, not proof.
### What does benchgate check?
It runs benchmarks N times and fails if any benchmark's coefficient of variation exceeds -cv-threshold (default 5.0, in percent). The exit code comes from the CV check alone, so a +40% delta on a quiet sample still exits 0; gating on the delta is a step you write.
### What does benchenv check?
SMT state, CPU frequency governor, Turbo Boost, system load average, and whether perflock, benchstat and benchdiff are installed. Its only flag is -json. Fix the [warn] lines before trusting benchmark numbers.
### How many times should I run a Go benchmark?
Ten is the floor, not once. For anything you report, use -count=20, plus -benchtime=2s on a noisy Mac.
### Should CI be the primary benchmark measurement?
No. Use CI for detection, as the smoke alarm, and A/B on the same machine for measurement, since a sub-10% micro delta can be code layout. In the #4891 story the A/B on one machine took the machine out of the story, with CI as the smoke alarm rather than the judge.
## Related
- [[resources/documentation/Go/Benchmarks/Go Benchmarks|Go Benchmarks]] (series hub)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Compiler Honesty and Dead Code Elimination|Compiler Honesty and Dead Code Elimination]] (question 1)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Statistics and benchstat|Statistics and benchstat]] (question 2)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] (question 3)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] (CI detection, not measurement)
## Sources
- [Why Your Go Benchmarks Are Lying (GopherCon UK 2026, GoLab 2026): talk](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/)
- [Why Your Go Benchmarks Are Lying: series](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/)
- [FOSDEM 2026: measuring software performance (prerequisite)](https://kakkoyun.me/posts/fosdem-2026-measuring-software-performance/)
- [OPERA press release, September 2011 (archived)](https://web.archive.org/web/20140222165941/http://press-archived.web.cern.ch/press-archived/PressReleases/Releases2011/PR19.11E.html)
- [Science: the loose cable](https://doi.org/10.1126/science.335.6072.1027)
- [dd-trace-go #4891](https://github.com/DataDog/dd-trace-go/pull/4891)
- [dd-trace-go #4926](https://github.com/DataDog/dd-trace-go/pull/4926)
- [BenchmarkOTLPProtoSize source](https://github.com/DataDog/dd-trace-go/blob/1a0c5e19b611f34f14c14531304973c89d2fb055/ddtrace/tracer/otlp_writer_bench_test.go#L144)
- [Performance Matters, Emery Berger (Strange Loop 2019)](https://www.youtube.com/watch?v=r-TLSBdHe1A)
- [testing.B.Loop documentation](https://pkg.go.dev/testing#B.Loop)
- [kakkoyun/benchlab at v0.1.0](https://github.com/kakkoyun/benchlab/tree/v0.1.0)
- [gopherconuk-26: the talk's demo results](https://github.com/kakkoyun/gopherconuk-26/tree/fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb/talks/go-benchmarks-lying)