# benchstat Go benchmarks: median, confidence interval, CV, p-value
> [!summary] TL;DR
> `BenchmarkMakeBuffer_Correct` gave 39.39 and 27.54 ns/op for the same code and binary, a 43% swing, because a loaded machine shifted between two regimes. Collect `-count=20`, summarize with `benchstat` (median plus 95% confidence interval), and check the machine separately with CV = standard deviation / mean. Idle host: 4.75% CV. Host with 16 background spinners: 18.88%. benchstat compared the two captures at `+230.34% (p=0.000 n=20)`.
## Key points
- `BenchmarkMakeBuffer_Correct-16` gave 39.39 and 27.54 ns/op for the same code and binary, eight runs apart: a 43% swing (runs 1 and 9 of twenty, Apple M4 Max, darwin/arm64, companion repository capture; `-16` is the `GOMAXPROCS` suffix).
- The machine ran sixteen CPU-bound background processes competing for the same cores. Scheduler preemption, cache evictions and stolen CPU time are the likely culprits; any single result is one draw from a distribution whose shape, center and spread you have not seen.
- Collect twenty runs (`-count=20 -benchtime=1s`) and summarize with `benchstat results.txt`, installed pinned at `v0.0.0-20260929162123-406019bb8b68`.
- benchstat reports the median with `±` as the 95% confidence interval around that median, as a percentage of it. Idle capture: `11.32n ± 5%`. The median ignores the outliers that a GC pause or a scheduler preemption creates; the arithmetic mean follows them up.
- Idle vs loaded: `11.32n ± 5%` against `37.40n ± 25%`, `+230.34% (p=0.000 n=20)`, about 3.3 times as long, with identical allocations between the two captures. When the p-value exceeds 0.05, benchstat prints `~` instead of a delta: no measurable difference with this sample size. The test is Mann-Whitney U on the samples (ranks, no bell curve needed), not the Welch's t-test from the FOSDEM post.
- The `±` intervals do not say whether the environment is stable enough to trust a comparison; they narrow as runs are added, CV does not. CV = σ/μ as a percentage, and `cv.awk` (run via `make cv`) computes mean, standard deviation and CV per benchmark from any `go test -bench` output.
- Idle host: 11.46 mean ns/op, 0.54 stddev, 4.75% CV. Sixteen background spinners: 34.97, 6.60, 18.88%. The loaded machine is three times slower and four times noisier; any A/B comparison collected there is uninterpretable.
- CV rule of thumb: under 2% results are reliable; 2 to 5% acceptable for most comparisons; 5 to 10% noisy, investigate the environment; over 10% fix the environment and do not trust comparisons. Noise shrinks with more runs; bias does not.
- The loaded twenty runs form two regimes: runs 1-7 land between 38 and 44 ns/op, run 8 is 37, runs 9-16 drop to 25-29, and the last four climb back (34, then 38-39). The mean 34.97 sits in the gap between the two clusters. Mann-Whitney assumes one distribution per sample, which these runs are not.
- Run counts: ten is the practical floor ("at least 10, ideally 20" per the benchstat docs); below six samples benchstat prints `± ∞`; twenty for a CI nightly suite or anything reported; Kalibera and Jones (ISMM 2013) argue for repetition counts based on measured variance; fifty or more for environments you cannot quiet down, though it keeps the same bias.
- Lehr's rule: resolving a change of Δ% at a CV of c% takes about 16·(c/Δ)² runs per side (about 80% power at α=0.05; assumes normal noise, and two regimes break it). At 5% CV a 5% change needs 16 to 20 runs; at 18.88%, more than 200.
- For CI baselines prefer a fixed count such as `-benchtime=1000000x` (a million iterations per run, no calibration) with `-count=20`; a calibration loop that reacts to momentary system load is variance nobody asked for. Pick N so each run lasts milliseconds, not microseconds.
- p-hacking (rerunning until you see a result you like) invalidates benchstat's p-values, which are calibrated for a pre-specified number of experiments: at the default α of 0.05 about one comparison in twenty shows a difference when nothing changed; across 40 benchmarks that is about two false flags per run. Pre-commit N, report `~` results, keep a ledger of flags that do not reproduce.
- A low p-value does not mean the difference matters: with 100 samples on a quiet machine (CV around 1% or less) a 0.3% change can clear p=0.05, which saves 600 ns on a 200 µs HTTP handler. With five samples a 15% regression might not reach significance, but 15% on a critical path deserves a look.
- This is part 2 of 5 in the Why Your Go Benchmarks Are Lying series. It asks the second of the three questions: is the sample stable enough to mean anything?
- Versions, links and commands checked on 2 October 2026.
## The run that lied: same code, two answers
`BenchmarkMakeBuffer_Correct` was run on a machine with sixteen CPU-bound background processes competing for the same cores. Two of its twenty runs (`B/op` and `allocs/op` columns dropped):
```text
BenchmarkMakeBuffer_Correct-16 41877204 39.39 ns/op
...
BenchmarkMakeBuffer_Correct-16 52521198 27.54 ns/op
```
Those are runs 1 and 9 of one capture on an Apple M4 Max (darwin/arm64) from the companion repository. Same code, same binary, eight runs apart, a 43% swing. Run it once, file the PR, and you could have reported either number. Scheduler preemption, cache evictions and stolen CPU time are the likely culprits: the benchmark measured what the machine was doing; it just was not what we thought we were measuring. Any single result is one draw from a distribution whose shape, center and spread we have not seen.
## From runs to a distribution with benchstat
The cure for a noisy measurement is more samples, not better luck. Collect twenty runs and let benchstat summarize them:
```bash
go test -bench=BenchmarkMakeBuffer_Correct -benchmem -count=20 -benchtime=1s . \
| tee results.txt
go install golang.org/x/perf/cmd/
[email protected]
benchstat results.txt
```
This is the benchstat output for the idle capture (same `-count=20 -benchtime=1s`, same M4 Max), trimmed to the `sec/op` table because the tool also prints `B/op` and `allocs/op`:
```text
goos: darwin
goarch: arm64
pkg: github.com/kakkoyun/gopherconuk-26/demo
cpu: Apple M4 Max
│ results/idle.txt │
│ sec/op │
MakeBuffer_Correct-16 11.32n ± 5%
```
Two numbers here. `11.32n` is the median of the 20 runs, and `± 5%` is how far the 95% confidence interval around that median reaches, as a percentage of it: a range built to contain the true median 95% of the time.
- Why the median: real latency data is rarely a bell curve. Brendan Gregg's frequency trails show disk I/O latency distributions that are combinations of bimodal and log-normal. A GC pause or a scheduler preemption can drag one benchmark run well above the rest, and the arithmetic mean follows it up. The median ignores those outliers.
### Comparing two captures
Comparison is where benchstat earns its keep: collect one file before a change and one after, then hand it both:
```bash
go test -bench=. -benchmem -count=20 -benchtime=1s . > old.txt
# make your change
go test -bench=. -benchmem -count=20 -benchtime=1s . > new.txt
benchstat old.txt new.txt
```
Here the "change" is the load, so the idle capture is compared against the loaded one (`sec/op` only; allocations were identical):
```text
│ results/idle.txt │ results/noisy.txt │
│ sec/op │ sec/op vs base │
MakeBuffer_Correct-16 11.32n ± 5% 37.40n ± 25% +230.34% (p=0.000 n=20)
```
The delta is +230.34%, so the loaded machine took about 3.3 times as long, and a p-value of effectively zero says the difference is real. When the p-value exceeds 0.05, benchstat prints `~` instead of a delta, which is the right answer: "no measurable difference with this sample size." Report it as such. The test underneath is Mann-Whitney U on the samples, which compares ranks and needs no bell curve, not the Welch's t-test from the FOSDEM post.
benchstat says the two machines differ. It does not say whether the loaded one is a place where any comparison means anything.
## What benchstat does not tell you: coefficient of variation
The `±` figures are 95% confidence intervals on the median. They hint at spread, but they do not say whether the environment is stable enough to trust a comparison run in it. The interval narrows as runs are added; CV does not. CV = σ/μ, as a percentage. Where benchstat answers "is A different from B?", CV answers "is this machine a reliable place to ask?" benchstat compares two distributions; it does not characterize the environment producing them, much as a referee does not inspect the pitch. A short awk script does that pass:
```bash
make cv
# or directly:
awk -f cv.awk results/idle.txt
awk -f cv.awk results/noisy.txt
```
`cv.awk` reads any `go test -bench` output and computes the mean, standard deviation and CV per benchmark. Over the idle and loaded captures (same `-count=20 -benchtime=1s` on the same M4 Max) it gives this table. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Part 3]] adds a third condition.
| Condition | Mean ns/op | Stddev | CV |
|-----------|-----------|--------|-----|
| Idle host | 11.46 | 0.54 | 4.75% |
| 16 background spinners | 34.97 | 6.60 | 18.88% |
The loaded machine is three times slower and four times noisier. A CV of 18.88% puts the standard deviation at nearly one-fifth of the mean, so we are measuring scheduling interference at least as much as the code. Any A/B comparison collected there is uninterpretable.
Rule of thumb for reading CV, not benchstat's:
| CV | Interpretation |
| ---- | --------------- |
| < 2% | Results are reliable |
| 2 to 5% | Acceptable for most comparisons |
| 5 to 10% | Noisy; investigate the environment |
| > 10% | Fix the environment; do not trust comparisons |
At 18.88% the loaded condition misses the 10% line by nearly a factor of two. More samples would not help, because the environment is biased and averaging does not remove a bias (noise shrinks with more runs; bias does not).
### Back to the run that lied
The mean of 34.97 ns/op looks unremarkable until we read the twenty runs. The first seven land between 38 and 44 ns/op. Run eight is 37. Runs nine through sixteen drop to 25 to 29, roughly a third lower. Then the last four climb back: 34, then 38 to 39. The mean sits in the gap between the two clusters, where only a couple of runs land; it describes a machine that was almost never in that state.
So run 1 (39.39) sits in the first cluster and run 9 (27.54) in the second. The machine shifted between two regimes during the capture, and we happened to sample one of each. Mann-Whitney assumes one distribution per sample, which these runs are not. One dot per measurement, the strip plot from the FOSDEM post, shows this at once; a boxplot would obscure it, and a mean buries it entirely.
Why the regimes differ was not measured. Scheduler interference is the likely explanation, though P/E-core placement or frequency scaling would look similar on this machine. More statistics cannot recover that; a pinned container is the start of the answer, and part 3 measures what isolation buys.
## How many runs are enough?
Ten is the practical floor: benchstat's own documentation says to run "at least 10, ideally 20". Below six samples it cannot compute a confidence interval at all and prints `± ∞`, so five is a sanity check and nothing more. Ten gives a reliable local A/B comparison, and 20 is for a CI nightly suite or anything we intend to report. Kalibera and Jones (ISMM 2013) argue for repetition counts based on measured variance, and 20 is a reasonable engineering compromise between rigor and machine time. Fifty or more is for environments we cannot quiet down: it tightens the interval and keeps the same bias, so for the run that lied it would only give a sharper picture of two moods.
As a rule of thumb, resolving a change of Δ% at a CV of c% takes about 16·(c/Δ)² runs per side. At 5% CV, a 5% change needs 16 to 20 runs; at the loaded machine's 18.88%, more than 200. This is Lehr's rule, 16σ²/Δ² per group for about 80% power at α=0.05. It assumes normal noise, and two regimes break it.
By default `-benchtime=1s` lets Go calibrate `b.N` to fill one second per run. For CI baselines prefer a fixed count such as `-benchtime=1000000x` (a million iterations per run, no calibration) with `-count=20`, because a calibration loop that reacts to momentary system load is variance nobody asked for. Pick N so each run lasts milliseconds, not microseconds.
## The p-hacking trap and effect size
The most common way to get a benchmark result you like is to keep running until you see one. That habit is p-hacking, and it invalidates the p-values benchstat reports, which are calibrated for a pre-specified number of experiments, not an open-ended search. benchstat's documentation calls rerunning until it reports a change a common statistical error: at the default α of 0.05, about one comparison in twenty shows a difference when nothing changed, whatever your `-count`. Across 40 benchmarks that is about two false flags per run: re-measure a flag with a pre-committed N, and keep a ledger of the ones that do not reproduce.
The discipline: decide N before running, run once, and report what benchstat says, `~` results included. If it matters enough to look again, a larger run is a new experiment: fix its N up front and label it a second look in the report. Rerunning after a disappointing p-value and keeping whichever result you like is p-hacking. "It only improved on the third attempt" is p-hacking. "It improved when I closed my editor" is environmental confounding, which is equally problematic.
A low p-value says the difference is unlikely to be zero. It does not say the difference matters. With 100 samples on a quiet machine (CV around 1% or less), a 0.3% change can clear p=0.05. On an HTTP handler running at 200 µs that saves 600 nanoseconds per request: statistically real, practically irrelevant. The inverse also holds: with five samples a 15% regression might not reach significance, but 15% on a critical path deserves a look anyway, and a larger pre-committed N can settle it as a new experiment.
Rule of thumb for acting on a delta, not benchstat's:
| Delta | p-value | Action |
| ------- | --------- | -------- |
| < 2% | any | No action needed |
| 2 to 10% | > 0.05 | Treat as no measurable difference; pre-commit to a larger N if it matters |
| 2 to 10% | < 0.05 | Investigate; may be real |
| > 10% | > 0.05 | Investigate; the noise may be hiding a real change |
| > 10% | < 0.05 | Real; act on it |
Regressions that creep in at 1 to 2% per commit never trip these thresholds; the FOSDEM post's change-point detection section covers them.
To recompute the CV table from the captured results in the companion repository:
```bash
git clone https://github.com/kakkoyun/gopherconuk-26
cd gopherconuk-26/talks/go-benchmarks-lying/demo
git checkout fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb
make cv # recompute the CV table from results/
```
`make cv` prints all three conditions, the two above and the pinned container from part 3. Re-running them is in [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|part 3]].
## FAQ
### How many times should I run a Go benchmark?
At least 10, ideally 20, per the benchstat documentation. Below six samples benchstat cannot compute a confidence interval and prints `± ∞`. Use 20 for a CI nightly suite or anything you intend to report.
### What is a good CV for Go benchmark comparisons?
Under 2% is reliable, 2 to 5% is acceptable for most comparisons, 5 to 10% means the environment is noisy and worth investigating, and above 10% you should fix the environment and not trust comparisons. More samples do not remove bias.
### What does the ± number in benchstat output mean?
It is the 95% confidence interval around the median, as a percentage of the median. It narrows as you add runs; it does not characterize the environment.
### Why did the same Go benchmark report 39 and 27 ns/op?
The machine shifted between two regimes during the capture: runs 1 to 7 landed at 38 to 44 ns/op and runs 9 to 16 at 25 to 29. A single run samples whichever regime is active; sixteen CPU-bound background processes were the load.
### What does ~ mean in benchstat output?
The p-value exceeded 0.05, so there is no measurable difference with this sample size. Report it as such instead of rerunning until a delta appears.
### Is a low p-value enough to act on a benchmark change?
No. With 100 samples on a quiet machine a 0.3% change can clear p=0.05, and 0.3% of a 200 µs handler is 600 ns per request. Check the delta size too: over 10% with p < 0.05 is real and worth acting on.
### What is p-hacking in benchmarking?
Rerunning until benchstat reports a difference you like. The p-values are calibrated for a pre-specified number of experiments; at α=0.05 about one comparison in twenty flags when nothing changed.
### Should I use -benchtime=1000000x for CI benchmarks?
Yes, with `-count=20`. A fixed iteration count avoids a calibration loop that reacts to momentary system load. Pick N so each run lasts milliseconds, not microseconds.
## Related
- [[resources/documentation/Go/Benchmarks/Go Benchmarks|Go Benchmarks]] (series hub)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Compiler Honesty and Dead Code Elimination|Compiler Honesty and Dead Code Elimination]] (part 1: whether the compiler lets the benchmark do real work)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] (part 3: what a pinned container buys)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] (part 4: the machine in CI)
## Field notes from agent sessions (2026-11-10)
Field notes from benchmark and memory investigations in agent sessions. They add code-layout noise, benchstat semantics, and memory-regression measurement.
- A CI benchmark delta that flips sign across runs is the signature of code-layout noise, not a real cost. A change flagged 6 to 9% slower touched nothing the timed loop measured. Local run-to-run variance stayed below 0.1%, and a same-machine A/B against the base branch showed the change was faster. Restructuring one file shifted function addresses and perturbed cache-line alignment in the hot loop.
- benchstat's per-benchmark figure is the median, not a geometric mean. The geometric mean appears only in the cross-benchmark summary row. Direct computation from the results file settled a draft that claimed otherwise.
- The Go 1.24+ benchmark pattern is `for b.Loop()` plus `b.ReportAllocs()`, instead of `for i := 0; i < b.N; i++`.
- Sample-size minimums must count measured samples, not stored records. A metric counted revisions with no measured duration, so a window with five revisions but one measured sample reported a 50% improvement and still passed a documented five-measurement minimum.
- Measure heap per case in its own process. Never compare `HeapSys` across sequential in-process iterations.
- For memory-regression measurement, build the test binaries explicitly so peak RSS lands on the test process itself. Run four sequential runs on a quiet host and report min, median and max. Single-run numbers moved a headline wall ratio from 2.3x to 1.7x once a clean baseline existed.
- Subset benchmarks can invert a conclusion. Measuring two payload tests in isolation "refuted" a memory hypothesis that the full package confirmed at 5.1x peak RSS. The measured unit must be the unit CI actually runs.
### Observations
- [benchmarks] A benchmark delta that flips sign across runs is code-layout noise, not a real cost. #benchmarks (source: agent sessions, 2026-07)
- [stats] The per-benchmark figure in benchstat output is a median, not a geometric mean. #benchstat (source: agent sessions, 2026-08)
- [benchmarks] Go 1.24+ benchmarks use `for b.Loop()` plus `b.ReportAllocs()`. #go (source: agent sessions, 2026-07)
- [metrics] Sample-size minimums must count measured samples, not stored records. #metrics (source: agent sessions, 2026-09)
- [benchmarks] Measure heap per case in its own process and never compare `HeapSys` across in-process iterations. #memory (source: agent sessions, 2026-07)
- [testing] Build test binaries explicitly so peak RSS lands on the test process for memory-regression checks. #testing (source: agent sessions, 2026-09)
- [testing] Subset benchmarks can invert a conclusion. #testing (source: agent sessions, 2026-09)
## Sources
- [Why Your Go Benchmarks Are Lying (talk, GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/)
- [Why Your Go Benchmarks Are Lying (series)](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/)
- [Measuring Software Performance: Why Your Benchmarks Are Probably Lying (FOSDEM 2026 post)](https://kakkoyun.me/posts/fosdem-2026-measuring-software-performance/)
- [benchstat documentation](https://pkg.go.dev/golang.org/x/
[email protected]/cmd/benchstat)
- [Brendan Gregg: Frequency Trails, outliers](https://www.brendangregg.com/FrequencyTrails/outliers.html)
- [Kalibera and Jones: Measuring Benchmarking Impact (ISMM 2013)](https://dl.acm.org/doi/10.1145/2464157.2464160)
- [gopherconuk-26 repository](https://github.com/kakkoyun/gopherconuk-26)
- [Demo at commit fdce88cc](https://github.com/kakkoyun/gopherconuk-26/tree/fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb/talks/go-benchmarks-lying/demo)