# How to build a Go benchmark PR gate that actually fails
> [!summary] TL;DR
> One GitHub Actions job builds the base and head benchmark binaries, runs them interleaved (base, head, base, head) for 10 rounds at 200 ms, and fails when benchstat calls the delta real (p < 0.05) and at least +5%. benchstat exits 0 even on a +199.00% table, so an awk pass over its CSV output makes the verdict. The slow commit went red at +198.57% (p=0.000 n=10); its revert went green with ~ (p=0.782 n=10).
## Key points
- The gate blocks only when benchstat says the delta is real (p < 0.05) and at least MIN_DELTA_PCT = 5%. A faster result is ignored.
- benchstat exits 0 even when its table says +199.00% (checked with two made-up result files, the pinned version, Go 1.27.1), so the workflow parses benchstat's CSV output with awk.
- One job builds both binaries and runs them on the same hosted VM, interleaved: base, head, base, head. ROUNDS=10 is both the ABAB round count and the sample count per side (n=10, the floor the benchstat docs recommend).
- BENCHTIME is 200 ms; the whole job took about 35 seconds. Part 4's nightly suite can afford -count=20 -benchtime=5s on controlled hardware; this gate cannot, on every push.
- benchgate is the noise gate: exit 1 when any benchmark's coefficient of variation is above CV_THRESHOLD=10. benchstat is the regression gate.
- Verdict exit codes: 0 pass, 1 regression, 2 tool or setup error, 3 unstable sample. A regression wins over "unstable"; each finding becomes an ::error annotation on the PR.
- The slow commit routes every addition through a helper marked //go:noinline, roughly 3x slower on purpose.
- The demo PR targets a branch that already holds the workflow and the benchmark, because a PR against main has no base-side benchmark.
- Run 1: Ints-4 went from 2.579µs to 7.700µs, +198.57% (p=0.000 n=10), benchgate CV 3.1%, red check (exit 1), on an AMD EPYC 7763 runner.
- Run 2, a minute later on the same kind of hosted runner: ~ (p=0.782 n=10), CV 3.4%, exit 0.
- benchgate v0.1.0 -baseline only prints the benchstat output; the exit code never looks at it. benchgate runs its own go test on the head tree, one after another, so it measures how noisy the runner was, not the interleaved samples.
- The ± column in the benchstat table is the better noise indicator for the comparison itself.
- A hosted gate cannot promise to catch a 5–10% regression; with n=10 on a shared runner, a change that small can flap: pass once, fail the next time.
- Tools: benchstat v0.0.0-20260929162123-406019bb8b68, benchgate v0.1.0 from kakkoyun/benchlab. The workflow and both runs live on PR #5, a closed demo. actionlint had no complaints. Versions, links and commands checked on 2 October 2026; the runs were captured on 4 October 2026.
## Why base and head share one job
[[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Part 4]] keeps the baseline from main in a cache, which suits a pinned runner. On a hosted runner, a cached baseline was measured on whatever VM ran the earlier job, and nothing promises that VM is the one judging your PR.
This gate builds both binaries in one job and runs them on the same VM, taking turns: base, head, base, head. That interleaving (ABAB) spreads slow drift evenly over both sides. It does not remove noisy neighbours: ubuntu-latest is a shared machine, and SMT or frequency scaling cannot be turned off from inside it.
## The bench-gate workflow, verbatim
The whole workflow: one job builds the base and head binaries, runs them ABAB, compares with benchstat, checks sample stability with benchgate, and fails on a regression that benchstat calls real.
```yaml
# A PR gate that actually fails: companion to part 4 of the Go benchmarks series.
#
# One job builds the base and head benchmark binaries, runs them interleaved
# (ABAB), and fails on a regression that benchstat calls real. benchgate rules
# out an unstable sample first.
name: bench-gate
on:
pull_request:
permissions:
contents: read
env:
BENCHSTAT_VERSION: v0.0.0-20260929162123-406019bb8b68
BENCHGATE_VERSION: v0.1.0
BENCH_PKG: ./internal/sum
# ROUNDS is both the number of ABAB rounds and the sample count per side (n).
ROUNDS: "10"
BENCHTIME: 200ms
# Block only when benchstat says the delta is real (p < 0.05) AND at least this large.
MIN_DELTA_PCT: "5"
# benchgate fails the sample when any benchmark's coefficient of variation is above this.
CV_THRESHOLD: "10"
jobs:
bench-gate:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
# pull_request checks out the merge commit, so HEAD^1 is the base tip and
# HEAD is what would land. Depth 2 is enough to reach both.
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
fetch-depth: 2
persist-credentials: false
- uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6
with:
go-version: "1.27.1"
cache: false
- name: Install benchstat and benchgate
run: |
export GOBIN="$RUNNER_TEMP/bin"
go install "golang.org/x/perf/cmd/benchstat@${BENCHSTAT_VERSION}"
go install "github.com/kakkoyun/benchlab/cmd/benchgate@${BENCHGATE_VERSION}"
- name: Build base and head benchmark binaries
run: |
if [ "$(git rev-list --parents -n 1 HEAD | wc -w)" -ne 3 ]; then
echo "::error::HEAD is not a merge commit; expected the pull_request merge ref"
exit 2
fi
mkdir -p "$RUNNER_TEMP/bench"
git worktree add --detach "$RUNNER_TEMP/base" HEAD^1
echo "base: $(git rev-parse --short HEAD^1) head: $(git rev-parse --short HEAD)"
go test -c -o "$RUNNER_TEMP/bench/head.test" "$BENCH_PKG"
(cd "$RUNNER_TEMP/base" && go test -c -o "$RUNNER_TEMP/bench/base.test" "$BENCH_PKG")
- name: Run base and head interleaved (ABAB)
run: |
pkgdir="${BENCH_PKG#./}"
: > "$RUNNER_TEMP/bench/base.txt"
: > "$RUNNER_TEMP/bench/head.txt"
for round in $(seq 1 "$ROUNDS"); do
for side in base head; do
root="$RUNNER_TEMP/base"
if [ "$side" = head ]; then root="$GITHUB_WORKSPACE"; fi
echo "round $round: $side"
(cd "$root/$pkgdir" && "$RUNNER_TEMP/bench/$side.test" \
-test.run='^