# Why A/B testing is the wrong model for benchmark CI > [!summary] TL;DR > A per-PR benchmark A/B compares two runs, but a benchmark's history is a time series. On dd-trace-go #4891 the bot said `BenchmarkOTLPProtoSize/1span` was 7.6% to 8.0% faster than its baseline; eleven days later, on #4926, it said the same sub-benchmark was 8.1% to 8.5% slower, because the baseline moved from `05d9b8a` to `1e830f6`. A 5% night-over-night gate fires twice on one noisy night and never on three 1.5% steps that add up to about 4.6%; change-point detection over the whole series does the opposite. ## Key points - On [dd-trace-go #4891](https://github.com/DataDog/dd-trace-go/pull/4891#issuecomment-4690941739), the benchmark bot's comment said `BenchmarkOTLPProtoSize/1span` was 7.6% to 8.0% faster than its baseline. Eleven days after #4891 was opened, on [#4926](https://github.com/DataDog/dd-trace-go/pull/4926#issuecomment-4779621296), the bot said the same sub-benchmark was 8.1% to 8.5% slower. - Both comments are correct arithmetic over two sets of runs; the baselines differ: `05d9b8a` on the first, `1e830f6` on the second. Same benchmark, opposite verdicts. - The bot's own comment: "This is an A/B test comparing a candidate commit's performance against that of a baseline commit." The pair quietly assumes the baseline is a fixed, neutral reference. It is not: `main` moves, and the binary's layout and the machine's mood move with it. - On #4891 alone the sign changed between bot updates; a pair cannot show that, because it has one point on each side. - Disclosure: the author works at Datadog, which maintains dd-trace-go. - The sixty-night series is synthetic: a Go program generated it, no real benchmark did. It is a pretend 1000 ns/op function measured once a night with 0.5% Gaussian noise; three "PRs" land at nights 20, 35 and 50, each adding 1.5%; on night 42 a noisy neighbour adds 8% for one night only. - The 5% gate (each night vs the one before, standing in for a PR check against one baseline run) fires twice, both times on the noisy neighbour: night 42 looks like a regression (+9.3%), night 43 like a recovery (-8.3%). - The gate never fires on the three real steps, because 1.5% is well under its threshold, yet together they made the benchmark about 4.6% slower. It blames the innocent night and lets the guilty ones through. - The toy change-point detector does binary segmentation: find the night with the largest median shift, and if that beats a threshold, cut there and repeat on both halves. Its output lands within two nights of each planted step (21, 33, 49 vs 20, 35, 50) and ignores the night-42 spike. - The toy reader does not use E-Divisive. [MongoDB engineers](https://arxiv.org/abs/2003.00584) moved to an E-Divisive means detector after threshold-based detection, and report that it "dramatically dropped our false positive rate". - The Go project's rule: never report performance numbers in isolation, only relative to some baseline, because comparing performance data taken far apart in time, even on the same hardware, can result in a lot of noise that goes unaccounted for. - Before merge, [SlowBot](https://go.dev/wiki/SlowBots) `perf_vs_parent` and `perf_vs_tip` run a pair on purpose. After merge, the [performance dashboard](https://perf.golang.org/dashboard/) gives continuous monitoring of benchmark performance for every commit; the tip-of-tree baseline is always the latest overall release of Go, so on every minor release the baseline shifts. - Keep the nightly series, use pairs for "did this change do it", and write down the alarms that turned out to be nothing. - Versions, links and commands checked on 2 October 2026. ## The pair that flipped: dd-trace-go #4891 and #4926 The bot says what it does in its own comment: "This is an A/B test comparing a candidate commit's performance against that of a baseline commit." For a question about one change, that is fair, and [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|A PR gate that actually fails]] builds one. The pair quietly assumes the baseline is a fixed, neutral reference. It is not. `main` moves, and the binary's layout and the machine's mood move with it. Change the baseline and the same benchmark gets a different verdict, as #4891 and #4926 show; on #4891 alone the sign changed between bot updates ([[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|part 5]] tells the #4891 story in full, including the CI regression that turned out to be a speedup). A pair cannot show that, because it has one point on each side. ## Sixty nights of one benchmark (synthetic) Everything in this section is synthetic: a Go program generated it, and no real benchmark did. The benchmark is a pretend 1000 ns/op function measured once a night, with 0.5% Gaussian noise. Three times, at nights 20, 35 and 50, a "PR" lands and adds 1.5%. On night 42 a noisy neighbour adds 8% for one night only. We read it two ways: a 5% gate comparing each night with the one before, standing in for a PR check against one baseline run, and a change-point detector. ```go package main import ( "fmt" "math" "math/rand/v2" "slices" ) const ( nights = 60 minLen = 5 threshold = 6.0 ) func main() { // Synthetic: 1000 ns/op, 0.5% noise, three +1.5% steps, one noisy night. rng := rand.New(rand.NewPCG(1, 2)) series := make([]float64, nights) for i := range series { level := 1000.0 for _, step := range []int{20, 35, 50} { if i >= step { level *= 1.015 } } series[i] = level * (1 + 0.005*rng.NormFloat64()) } series[42] *= 1.08 for i := 1; i < nights; i++ { if d := series[i]/series[i-1] - 1; math.Abs(d) > 0.05 { fmt.Printf("night %2d vs night %2d: %+.1f%% <- a 5%% gate fires\n", i, i-1, 100*d) } } fmt.Println("change points (night where a new level starts):") for _, cp := range split(series, 0, nights) { fmt.Printf(" night %d\n", cp) } } ``` The first loop is the per-PR model. Here is the first half of the output, from Go 1.27.1: ```text night 42 vs night 41: +9.3% <- a 5% gate fires night 43 vs night 42: -8.3% <- a 5% gate fires ``` The gate fires twice, both times on the noisy neighbour: night 42 looks like a regression, night 43 like a recovery. It never fires on the three real steps, because 1.5% is well under its threshold, yet together they made the benchmark about 4.6% slower. The gate blames the innocent night and lets the guilty ones through. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Part 4]] said a change costing 1% per PR won't trip a per-PR gate. Here are the numbers. ## Reading the series: a toy change-point detector Change-point detection scans the whole history for places where the results shift, and stays quiet about everything else. [MongoDB engineers](https://arxiv.org/abs/2003.00584) moved to an E-Divisive means detector after threshold-based detection, and report that it "dramatically dropped our false positive rate". The toy reader is much cruder and does *not* use E-Divisive. It does binary segmentation: find the night with the largest median shift, and if that beats a threshold, cut there and repeat on both halves. Here is the rest of the program: ```go // split does binary segmentation: cut at the strongest shift, recurse on both halves. func split(x []float64, lo, hi int) []int { best, bestT := -1, 0.0 for k := lo + minLen; k <= hi-minLen; k++ { if t := math.Abs(shift(x[lo:k], x[k:hi])); t > bestT { best, bestT = k, t } } if bestT < threshold { return nil } return append(append(split(x, lo, best), best), split(x, best, hi)...) } // shift is the difference of medians in units of its standard error, with // the noise scale taken from the median absolute deviation so that a single // spike cannot inflate it. func shift(a, b []float64) float64 { ma, mb := median(a), median(b) dev := make([]float64, 0, len(a)+len(b)) for _, e := range a { dev = append(dev, math.Abs(e-ma)) } for _, e := range b { dev = append(dev, math.Abs(e-mb)) } sigma := 1.4826 * median(dev) return (mb - ma) / (sigma * math.Sqrt(1/float64(len(a))+1/float64(len(b)))) } func median(x []float64) float64 { s := slices.Sorted(slices.Values(x)) return (s[(len(s)-1)/2] + s[len(s)/2]) / 2 } ``` `split` tries every cut at least `minLen` nights from either end and keeps the strongest. The `threshold` of 6 is a knob the author picked by eye for this data, and a real deployment has to set its own. The last part of the output: ```text change points (night where a new level starts): night 21 night 33 night 49 ``` It finds three level changes and ignores the night-42 spike. The planted steps were at 20, 35 and 50, so it lands within two nights of each. On a real series the output is a list of candidate nights, and the next job is `git log` between them. [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Part 4]] lists tools that ship change-point detection. ## How the Go project reads its own history The Go wiki's [PerformanceMonitoring](https://go.dev/wiki/PerformanceMonitoring) page states the principle plainly: "We never report performance numbers in isolation, and only relative to some baseline", because "comparing performance data taken far apart in time, even on the same hardware, can result in a lot of noise that goes unaccounted for". Before merge, [SlowBot](https://go.dev/wiki/SlowBots) `perf_vs_parent` and `perf_vs_tip` run a pair on purpose. After merge, the [performance dashboard](https://perf.golang.org/dashboard/) gives "continuous monitoring of benchmark performance for every commit", and because the tip-of-tree baseline "is always the latest overall release of Go", "on every minor release of Go, the baseline shifts". Read that next to the opening pair: the sign on #4891 flipped when `main` moved, and Go's dashboard puts baseline changes where you can see them. ## A ledger for alarms that weren't Bot comments will keep arriving, and some will be old friends. A mute regex silences a benchmark and forgets why. Add a ledger entry per dismissed alarm, in the repository, next to the benchmark. This one is illustrative, written from the public bot comments and part 5: ```text BenchmarkOTLPProtoSize (all four sub-benchmarks) seen: dd-trace-go #4891 (12 Jun), #4926 (23 Jun) symptom: 6-9% in one direction; sign flipped when main moved cause: code layout, suspected, not proven (see part 5) check: same-machine A/B of base vs head before anyone edits code ``` `seen` turns one surprise into a count, `cause` says how sure we are, and `check` names the decisive test, so the next person doesn't improvise one. To run the detector yourself: the two Go blocks above are one program, paste the second under the first in a `main.go`, run `go mod init cp` and `go run .`. The author ran that with Go 1.27.1 and got the output shown. Change the seed or `threshold` and watch the detector miss a step or invent one. Then ask what the baseline was in your own bot's last three comments. ## FAQ ### Why did my benchmark bot call the same benchmark faster and slower? On dd-trace-go #4891 the bot said `BenchmarkOTLPProtoSize/1span` was 7.6% to 8.0% faster than its baseline (`05d9b8a`); eleven days later on #4926 it said the same sub-benchmark was 8.1% to 8.5% slower than `1e830f6`. Both are correct arithmetic; the baselines differ, and `main` moved between them. ### Why is a per-PR A/B the wrong model for CI benchmarks? A pair assumes the baseline is a fixed, neutral reference. `main` moves, and the binary's layout and the machine's mood move with it, so the same benchmark gets a different verdict when the baseline changes. A benchmark's history is a time series, and a pair has one point on each side. ### What does a 5% threshold gate miss? In the synthetic sixty-night series it fired twice, both times on a one-night 8% neighbour spike (+9.3% then -8.3%), and never on three 1.5% steps that together made the benchmark about 4.6% slower. It blames the innocent night and lets the guilty ones through. ### What is change-point detection on benchmark history? It scans the whole history for places where the results shift and stays quiet about everything else, instead of comparing each run with a fixed baseline. The toy version does binary segmentation over the median shift; a real deployment has to set its own threshold, a knob the author picked by eye for this data. ### How does the Go project track benchmark performance over time? Before merge, SlowBot `perf_vs_parent` and `perf_vs_tip` run a pair on purpose. After merge, the performance dashboard at perf.golang.org gives continuous monitoring for every commit, with a tip-of-tree baseline that is always the latest overall Go release, so the baseline shifts on every minor release. The Go wiki's rule: never report performance numbers in isolation. ### How do I run the change-point demo myself? The two Go blocks are one program: paste the second under the first in a `main.go`, run `go mod init cp` and `go run .`. It was run with Go 1.27.1 to produce the output shown. Change the seed or `threshold` and watch the detector miss a step or invent one. ### Should I write anything down when I dismiss a benchmark alarm? Yes, a ledger entry in the repository, next to the benchmark: which PRs it was seen on, the symptom (percentage and direction, sign flips), the cause and how proven it is, and the decisive check (a same-machine A/B of base vs head before anyone edits code). A mute regex silences a benchmark and forgets why. ### Is the sixty-night series real benchmark data? No. It is synthetic: a Go program generated it. A pretend 1000 ns/op function measured once a night with 0.5% Gaussian noise, three +1.5% steps at nights 20, 35 and 50, and a one-night +8% noisy neighbour at night 42. ## Related - [[resources/documentation/Go/Benchmarks/Go Benchmarks|Go Benchmarks]] (series hub) - [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] (the nightly suite and tool survey) - [[resources/documentation/Go/Benchmarks/Go Benchmarks - PR Gate That Fails|PR Gate That Fails]] (the per-PR gate this note argues against leaning on alone) - [[resources/documentation/Go/Benchmarks/Go Benchmarks - Three Questions Before You Trust a Benchmark|Three Questions Before You Trust a Benchmark]] ## Sources - [Why Your Go Benchmarks Are Lying (GopherCon UK 2026, GoLab 2026): talk](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/) - [Why Your Go Benchmarks Are Lying: series](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/) - [dd-trace-go #4891: benchmark bot comment (7.6-8.0% faster)](https://github.com/DataDog/dd-trace-go/pull/4891#issuecomment-4690941739) - [dd-trace-go #4926: benchmark bot comment (8.1-8.5% slower)](https://github.com/DataDog/dd-trace-go/pull/4926#issuecomment-4779621296) - [E-divisive change-point detection as applied at MongoDB](https://arxiv.org/abs/2003.00584) - [Go wiki: PerformanceMonitoring](https://go.dev/wiki/PerformanceMonitoring) - [Go wiki: SlowBots](https://go.dev/wiki/SlowBots) - [Go performance dashboard](https://perf.golang.org/dashboard/)