# Go benchmark CI noise: shared runners, bare metal, and the nightly suite
> [!summary] TL;DR
> A shared CI runner is a multi-tenant machine, so a real 10% regression can vanish into the noise and a phantom one can appear from nowhere. With SMT enabled, two CPU-bound tasks sharing one core showed a coefficient of variation around 23%; on bare metal with SMT and dynamic frequency scaling disabled, the FOSDEM runs drop below 0.25%. Build two tiers: a fast PR gate that blocks only when benchstat calls the delta significant, plus a nightly full suite on a self-hosted bare-metal runner.
## Key points
- A shared CI runner is a multi-tenant machine: other teams' compile jobs, Docker builds and test suites run on the same host. The gate measures the lottery of what happens to run next door, not your code.
- Three mechanisms compound: contention with another tenant's CPU-bound job for memory bandwidth and the last-level cache (LLC, shared across the cores of a die) and for execution units on a sibling hardware thread; dynamic frequency scaling; and hypervisor pauses.
- Dynamic frequency scaling (DFS, Turbo Boost on Intel) adjusts the CPU clock based on thermal load and power headroom, and a shared VM cannot disable it from inside the guest. A benchmark that ran at 3.5 GHz in one CI run may run at 3.1 GHz the next.
- The hypervisor can pause the vCPU to serve another tenant's interrupt; from inside the guest that looks like the benchmark stalled.
- With SMT (Simultaneous Multithreading, marketed as Hyper-Threading on Intel) enabled, two CPU-bound tasks sharing a core (DFS off) showed a CV around 23%; with SMT disabled, CV dropped roughly a hundredfold, and the tasks ran faster too.
- Turning DFS off cut CV roughly tenfold (0.383% to 0.041%). Absolute runtime goes up because the CPU runs at base frequency instead of boosting, but the measurement is stable: slow and steady beats fast and moody.
- A pinned container on a macOS developer machine bottoms out near 5.25% CV; a 5% CV puts a 5% real regression inside the noise floor of any single run. On bare metal with SMT and DFS disabled, the FOSDEM runs drop below 0.25%.
- PR gate: five minutes or less on a pinned runner, a curated subset (benchmarks that have regressed before, or that the PR directly exercises), `-count=6 -benchtime=2s` as the lowest count that works. benchstat needs six samples to print a 95% confidence interval at all; its docs recommend at least ten.
- The gate's baseline comes from `main`, kept in a GitHub Actions cache rather than an artifact, which by default is only visible to the run that produced it. The gate blocks only when benchstat reports a delta instead of `~` (p < 0.05) and that delta is larger than a floor you choose.
- benchstat exits 0 whatever it finds, so a step must read its `vs base` column and fail the job on a significant delta. The Go wiki warns that data taken far apart in time, even on the same hardware, can result in a lot of noise; building base and head in one job and interleaving them is stronger.
- Nightly suite: on a schedule against dedicated bare-metal hardware, `-count=20 -benchtime=5s`, full environment controls at the runner level, results in a time-series store (Bencher or Nyrkiö), compared against a rolling 30-day window, with change-point detection flagging regressions, alerts to Slack or email, and nightly data never blocks a PR.
- AWS EC2 bare-metal instances (`m5.metal`, `m7i.metal-24xl`) are dedicated physical hosts with no hypervisor layer, at roughly $4-5/hour on demand for Linux in us-east-1. Hetzner rents dedicated servers with full root access. An `m5.metal` running two hours nightly costs around $280/month, so a dedicated server can pay for itself within months.
- `taskset -c 0` makes `GOMAXPROCS` 1, so the nightly workflow suits single-threaded benchmarks; parallel ones need a wider CPU set.
- GitHub advises that self-hosted runners should almost never be used for public repositories: anyone who can open a pull request can run code on them, and this workflow uses `sudo`. On a private repository, put the gate behind an environment with required reviewers.
- Versions, links and commands checked on 2 October 2026.
## Why shared runners lie
Three mechanisms compound. Another tenant's CPU-bound job contends with yours for memory bandwidth and last-level cache, and for execution units if it lands on a sibling hardware thread. DFS adjusts the CPU clock based on thermal load and power headroom, and a shared VM cannot disable it from inside the guest. The hypervisor can even pause your vCPU to serve another tenant's interrupt, which from inside the guest looks like the benchmark stalled.
The [FOSDEM post](https://kakkoyun.me/posts/fosdem-2026-measuring-software-performance/) put numbers on what controlling these variables buys. Its experiments ran on a dedicated AWS `m5.metal` instance, with slides and results in [igoragoli/fosdem-2026-software-performance](https://github.com/igoragoli/fosdem-2026-software-performance/tree/2cea1c2a854119f48d124daf2a8157480cd4450a). With SMT (Simultaneous Multithreading, also marketed as Hyper-Threading on Intel; two hardware threads share one physical core's execution resources) enabled, two CPU-bound tasks sharing a core (DFS off) showed a coefficient of variation (CV, the standard deviation as a share of the mean) around 23%. With it disabled, the same two tasks ran on separate cores:
| Configuration | Mean | CV |
| --- | --- | --- |
| SMT enabled, task 1 | 1537.64 ± 367.29 ms | 23.887% |
| SMT enabled, task 2 | 1536.88 ± 366.84 ms | 23.869% |
| SMT disabled, task 1 | 737.37 ± 0.32 ms | 0.044% |
| SMT disabled, task 2 | 737.93 ± 1.74 ms | 0.235% |
Turning SMT off cut CV by roughly a hundredfold, and the tasks ran faster too, because they stopped fighting over one core's execution units. Next the clock. With SMT already disabled on the same machine, the [slides](https://github.com/igoragoli/fosdem-2026-software-performance/blob/2cea1c2a854119f48d124daf2a8157480cd4450a/presentation.md) show DFS on and off for a single task:
| Configuration | Mean | CV |
| --- | --- | --- |
| DFS on, 1 task | 533.97 ± 2.046 ms | 0.383% |
| DFS off, 1 task | 738.18 ± 0.306 ms | 0.041% |
The DFS-off row is the SMT-disabled configuration again (SMT and DFS off), hence the same ~738 ms. For comparison with a setup you might already have: [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Part 3]] showed that a pinned container on a macOS developer machine bottoms out near 5.25% CV. A 5% CV puts a 5% real regression inside the noise floor of any single run, and resolving it takes more samples than a PR gate can afford. On bare metal with SMT and DFS disabled, the FOSDEM runs drop below 0.25%, well below any regression worth caring about.
## Two tiers that work: PR gate and nightly suite
A PR gate needs to finish in minutes, while thorough statistics need many samples over a stable environment. A nightly suite can run for an hour on controlled hardware, but most teams cannot require that hardware on every PR. No single job serves both, so build two.
Pattern A, the PR gate, runs on every pull request and catches unambiguous regressions before they merge. It targets five minutes or less on a pinned runner, running a curated subset: benchmarks that have regressed before, or that the PR directly exercises. Start with `-count=6 -benchtime=2s`, the lowest count that works, because [benchstat](https://pkg.go.dev/golang.org/x/
[email protected]/cmd/benchstat) needs six samples to print a 95% confidence interval at all, and its docs recommend at least ten. The baseline comes from `main`, kept in a GitHub Actions cache rather than an artifact, which by default is only visible to the run that produced it. The gate blocks only when benchstat reports a delta instead of `~` (p < 0.05) and that delta is larger than a floor you choose.
Two caveats. benchstat exits 0 whatever it finds, so a step must read its `vs base` column and fail the job on a significant delta. A cached baseline was also measured earlier, maybe elsewhere: [data taken far apart in time, even on the same hardware, can result in a lot of noise](https://go.dev/wiki/PerformanceMonitoring), says the Go wiki, so building base and head in one job and interleaving them is stronger; [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|A PR gate that actually fails]] shows that job catching a slow commit. On GitHub-hosted runners the gate is weaker, not worthless: a smoke alarm, as [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|part 5]] says, not the judge.
A raw percentage threshold ("block on a >5% change") is dangerous in both directions: it blocks a 5.1% improvement and passes a 4.9% regression. With benchstat's significance test alongside it, `~ (p=0.3)` should not block and `+8.00% (p=0.001 n=6)` should.
Pattern B, the nightly full suite, runs on a schedule against dedicated bare-metal hardware and watches for the slow regressions that accumulate across many PRs. It can afford `-count=20 -benchtime=5s`, with the full environment controls applied at the runner level. Results go into your own time-series store (Bencher or Nyrkiö, from the survey below), compared against a rolling 30-day window, with change-point detection flagging regressions. Alerts go to Slack or email, and nightly data never blocks a PR. A change that costs 1% per PR, landing five times over a sprint, will not trip a per-PR gate, but change-point detection on the time series can; [[resources/documentation/Go/Benchmarks/Go Benchmarks - AB Testing Is the Wrong Model for CI|A/B is the wrong model for CI]] builds a small detector to show it.
## Bare metal versus VM
The controls are the Linux toolbox from [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|part 3]]: SMT off, the performance governor with turbo off, and `taskset`. Together they remove the three largest sources of benchmark variance, but only where they are physically real. Inside a shared VM the hypervisor manages SMT, so the guest cannot disable it, and frequency controls in a guest, where they exist at all, do not change the physical CPU clock. `echo off > /sys/devices/system/cpu/smt/control` may succeed (it offlines the guest's own sibling vCPUs) without changing how the hypervisor schedules them. A control that succeeds and changes nothing is the worst kind: it makes you feel better and measures the same.
Where to find a machine with real controls: AWS EC2 bare-metal instances (`m5.metal`, `m7i.metal-24xl`) are dedicated physical hosts with no hypervisor layer, at roughly $4-5/hour on demand for Linux in us-east-1. Hetzner rents dedicated servers with full root access. Past an hour or so of nightly runtime, one persistent bare-metal machine is likely to beat cloud instances on cost: an `m5.metal` running two hours nightly costs around $280/month, so a dedicated server, rented or owned, can pay for itself within months.
## Nightly GitHub Actions workflow, verbatim
This workflow runs on a self-hosted bare-metal runner and applies the controls before the benchmarks. The critical detail is `runs-on: [self-hosted, linux, x64, bare-metal]`: GitHub adds `self-hosted`, `linux` and `x64` when you register the runner, `bare-metal` is a custom label you add, and the combination routes only to those runners. Writing `ubuntu-latest` here would silently route to shared VMs and defeat the entire exercise.
```yaml
# .github/workflows/benchmark-nightly.yml
name: Nightly Benchmarks
on:
schedule:
- cron: '0 2 * * *' # 2am UTC
jobs:
benchmark:
runs-on: [self-hosted, linux, x64, bare-metal]
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
- name: Restore baseline cache
uses: actions/cache/restore@v6
with:
path: bench-baseline.txt
key: bench-baseline-${{ github.run_id }}
restore-keys: |
bench-baseline-
- name: Apply environment controls
run: |
echo off | sudo tee /sys/devices/system/cpu/smt/control >/dev/null
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo >/dev/null
- name: Run benchmarks
shell: bash # runs with pipefail, so a failing go test fails the step
run: |
taskset -c 0 go test \
-bench=. -benchmem \
-count=20 -benchtime=5s \
./... | tee bench-new.txt
- name: Compare with baseline
run: |
if [ -f bench-baseline.txt ]; then
go run golang.org/x/perf/cmd/
[email protected] \
bench-baseline.txt bench-new.txt
else
echo "No cached baseline yet. This run establishes one."
fi
- name: Promote new baseline
run: cp bench-new.txt bench-baseline.txt
- name: Save baseline cache
if: always()
uses: actions/cache/save@v6
with:
path: bench-baseline.txt
key: bench-baseline-${{ github.run_id }}
```
Top to bottom, it restores last night's baseline, applies the sysfs controls, pins the run to CPU 0, compares with benchstat, then promotes tonight's results as the new baseline. `taskset -c 0` makes `GOMAXPROCS` 1, so it suits single-threaded benchmarks; parallel ones need a wider CPU set. It never undoes the controls, fine for a box that only runs benchmarks; on a shared machine, add an `if: always()` step that writes `on` to the SMT control. It only prints the comparison; the time-series store and the alerting come from the tools in the survey below. The PR gate uses the same structure with a dedicated (but lighter) pinned runner in `runs-on`, a lower count and benchtime to fit the time budget, and only the curated benchmark subset. One warning before copying any of this: GitHub advises that [self-hosted runners should almost never be used for public repositories](https://docs.github.com/en/actions/reference/security/secure-use), because anyone who can open a pull request can run code on them, and this workflow uses `sudo`. On a private repository, put the gate behind an environment with required reviewers.
The comparison step leans on [golang.org/x/perf](https://pkg.go.dev/golang.org/x/
[email protected]), the Go team's official performance measurement toolchain; its tools read the standard `go test -bench` output with no adapters or code changes. benchstat computes the median, a non-parametric confidence interval for the median, and a Mann-Whitney U-test across two result sets. It prints `~` when the difference is indistinguishable from noise, and otherwise a delta with its p-value and sample count. That makes it the primary tool for A/B comparisons, locally and in CI:
```bash
go install golang.org/x/perf/cmd/
[email protected]
benchstat bench-baseline.txt bench-new.txt
```
The version pin matches the one in the workflow, and the old results go first, so the delta reads as the new run against the baseline. The `benchsave` sibling uploads results to a `perfdata` server, but its default one is the Go project's, behind an interactive Google sign-in that publicly records your email address; unattended CI needs its own store. benchstat is a point-in-time comparison: it keeps no time series and alerts on no drift. What CI adds is the historical baseline, the automated gate and the cross-PR comparison that no human maintains by hand.
## Tool survey and recommendations (12 August 2026)
The tools that tackle continuous Go benchmarking differ in statistical model, hosting model, and how much infrastructure they need. The field as of the talk (12 August 2026):
| Tool | Go native | Hosting | Statistics | PR gate | Maintained |
| --- | --- | --- | --- | --- | --- |
| **[bencher.dev](https://bencher.dev)** | Yes | Both | t-test, z-score, IQR, log-normal, percentage | First-class | Yes |
| **[`github-action-benchmark`](https://github.com/benchmark-action/github-action-benchmark/tree/v1.22.1)** | Yes | GitHub Pages | Percentage threshold | Yes | Yes (v1.22.1, May 2026) |
| **gobenchdata** | Yes | GitHub Pages | User-defined expression | Yes | Maintenance-mode (last release Jan 2023) |
| **cob** | Yes | None (ephemeral) | Percentage threshold | Via exit code | Borderline (Oct 2023) |
| **chronologer** | No, uses [hyperfine](https://github.com/sharkdp/hyperfine), not `go test` | None (local HTML) | None | No | Minimal |
| **[Nyrkiö](https://github.com/nyrkio/change-detection/tree/v2.0.2) / [Apache Otava](https://otava.apache.org/)** | Yes | Both | E-divisive change-point detection | Yes | Yes (Nyrkiö 2.0.0, Feb 2026) |
| **codespeed** (Django; not CodSpeed) | No, custom uploader required | Self-hosted (Django) | Visualization only | No | Unmaintained (last release Feb 2019) |
| **golang.org/x/perf benchstat** | Yes (official) | perf.golang.org (Go team only) | Median + CI, significance test | None built-in | Yes |
[CodSpeed](https://codspeed.io/docs/benchmarks/go) is not in the table: its Go integration is in early development and supports only wall-time measurement, not the CPU simulation it offers elsewhere. One of these deserves a warning label: **cob** has a subtle footgun, it internally runs `git reset`, so all changes must be committed before running it, which rules it out for pre-commit hooks or any CI step with a dirty working tree.
Which to pick depends on who "we" are:
- **Small OSS project on GitHub.** Use `benchmark-action/github-action-benchmark`. It needs no external accounts and stores results on GitHub Pages. The default 200% threshold is too loose to be useful. The `alert-threshold` input is a ratio of current to previous, so tighten it to `110%`-`120%` (a 10-20% slowdown) and add a benchstat step alongside it, because the action has no concept of significance. With `-count=10` the benchstat step gives honest numbers in the log, but the action treats each sample line as a separate result and does not average them.
- **Team with a dedicated CI runner.** Use **bencher.dev self-hosted**, on a noise-controlled machine. Bencher's [pricing page](https://bencher.dev/pricing/) says you can run the self-hosted binary without a Bencher Plus license, at the feature level of the Cloud Free plan, so check it against your private-project needs. Configure the `t_test` threshold model; unlike `z_score`, its docs do not ask for 30 historical samples. Wire `--error-on-alert` to block PRs. The `go_bench` adapter needs no changes to your benchmark code, though it reports only the mean.
- **Large org wanting change-point detection.** Use **Nyrkiö with Apache Otava** as the detection backend. The [e-divisive approach, as applied at MongoDB](https://arxiv.org/abs/2003.00584) finds persistent shifts across the series instead of comparing each run with a fixed baseline, and adapts to each benchmark's noise through its significance test. Apache Otava entered incubation in November 2024. Nyrkiö 2.0.0 (February 2026) added Nyrkiö Runners for GitHub, which its release notes describe as configured for stable, repeatable benchmark results. For fully on-premises operation, Apache Otava can run against a self-hosted time-series store.
In every scenario, keep benchstat for local comparisons and PR descriptions. Of the tools here, only benchstat gives a confidence interval and a p-value for a direct comparison of two sets of samples, so trust it when the question is whether a change is real.
## FAQ
### Why do Go benchmark results differ between CI runs?
The shared CI runner is a multi-tenant machine: other teams' compile jobs, Docker builds and test suites contend for memory bandwidth, last-level cache and execution units, DFS shifts the CPU clock (3.5 GHz in one run can become 3.1 GHz in the next), and the hypervisor can pause the vCPU to serve another tenant's interrupt. The gate measures the runner's neighbours, not just your code.
### How much noise does a shared CI runner add to benchmarks?
The FOSDEM experiments on a dedicated `m5.metal` showed a CV around 23% for two CPU-bound tasks sharing a core with SMT enabled and DFS off. With SMT disabled (tasks on separate cores) CV fell to 0.044% and 0.235%, roughly a hundredfold cut, and the tasks ran faster.
### Why can't I disable SMT or turbo inside a cloud VM?
Inside a shared VM the hypervisor manages SMT, so the guest cannot disable it, and frequency controls in a guest, where they exist at all, do not change the physical CPU clock. `echo off > /sys/devices/system/cpu/smt/control` may succeed by offlining the guest's own sibling vCPUs without changing how the hypervisor schedules them.
### What count and benchtime should a PR gate and a nightly suite use?
The PR gate starts with `-count=6 -benchtime=2s`, the lowest count that works, because benchstat needs six samples to print a 95% confidence interval at all (its docs recommend at least ten). The nightly suite on bare metal can afford `-count=20 -benchtime=5s`.
### How do I fail a CI job when benchstat finds a regression?
benchstat exits 0 whatever it finds, so a step must read its `vs base` column and fail the job when the delta is significant (not `~`, p < 0.05) and larger than the floor you choose. Block only on that combination; a raw percentage threshold blocks a 5.1% improvement and passes a 4.9% regression.
### Which benchmark CI tool should I pick?
A small OSS project: `benchmark-action/github-action-benchmark` with `alert-threshold` tightened to `110%`-`120%` plus a benchstat step. A team with a dedicated runner: bencher.dev self-hosted with the `t_test` model and `--error-on-alert`. A large org wanting change-point detection: Nyrkiö with Apache Otava. Keep benchstat everywhere for local comparisons; only it gives a confidence interval and a p-value for two sets of samples.
### Does the nightly suite block pull requests?
No. Nightly data never blocks a PR. It runs `-count=20 -benchtime=5s` on dedicated bare-metal hardware, feeds a time-series store compared against a rolling 30-day window with change-point detection, and sends alerts to Slack or email. It exists to catch slow regressions that accumulate across many PRs, like a 1% cost landing five times over a sprint.
## Related
- [[resources/documentation/Go/Benchmarks/Go Benchmarks|Go Benchmarks]] (series hub)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] (the Linux environment-control toolbox)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|PR Gate That Fails]] (base and head interleaved in one job)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - AB Testing Is the Wrong Model for CI|AB Testing Is the Wrong Model for CI]] (the change-point detector)
- [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|Three Questions Before You Trust a Benchmark]]
## Sources
- [Why Your Go Benchmarks Are Lying (GopherCon UK 2026, GoLab 2026): talk](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/)
- [Why Your Go Benchmarks Are Lying: series](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/)
- [FOSDEM post: measuring software performance](https://kakkoyun.me/posts/fosdem-2026-measuring-software-performance/)
- [igoragoli/fosdem-2026-software-performance: experiments and results](https://github.com/igoragoli/fosdem-2026-software-performance/tree/2cea1c2a854119f48d124daf2a8157480cd4450a)
- [FOSDEM slides](https://github.com/igoragoli/fosdem-2026-software-performance/blob/2cea1c2a854119f48d124daf2a8157480cd4450a/presentation.md)
- [benchstat on pkg.go.dev](https://pkg.go.dev/golang.org/x/
[email protected]/cmd/benchstat)
- [golang.org/x/perf on pkg.go.dev](https://pkg.go.dev/golang.org/x/
[email protected])
- [Go wiki: PerformanceMonitoring](https://go.dev/wiki/PerformanceMonitoring)
- [GitHub docs: secure use of self-hosted runners](https://docs.github.com/en/actions/reference/security/secure-use)
- [bencher.dev](https://bencher.dev)
- [bencher.dev pricing](https://bencher.dev/pricing/)
- [benchmark-action/github-action-benchmark at v1.22.1](https://github.com/benchmark-action/github-action-benchmark/tree/v1.22.1)
- [sharkdp/hyperfine](https://github.com/sharkdp/hyperfine)
- [nyrkio/change-detection at v2.0.2](https://github.com/nyrkio/change-detection/tree/v2.0.2)
- [Apache Otava](https://otava.apache.org/)
- [CodSpeed Go benchmarks docs](https://codspeed.io/docs/benchmarks/go)
- [E-divisive change-point detection as applied at MongoDB](https://arxiv.org/abs/2003.00584)