# Go build overhead 230% on my laptop, 81% in CI: which number is real? > [!summary] TL;DR > One otelc pull request produced two honest numbers: ~230% build overhead for the `multi` scenario on the author's loaded laptop, +81% in CI. The scenario the change could not have touched (`largeidle`, ~212%) moved by the same amount, which pointed at the machine, not the diff. CI's only real failure was `baseline` at +514.9% to +521.9%, attributed to the SDK dependency bump. The question that told the two apart cost nothing to ask: what could this change not have touched? ## Key points - Pull request [#643](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/643) consolidated a pile of dependency bumps, among them `go.opentelemetry.io/otel` core to v1.44.0. On 3 July the overhead check ran on the author's machine and in CI, and the two disagreed. - The overhead formula is `(otelc − plain) / plain`, so 230% means the instrumented build took about 3.3 times as long as the plain build. - The check's ceiling is 150% for every scenario except `baseline`, which gets 500%. - The readings: `multi` ~230% on the laptop against +81% in CI; `largeidle` ~212% against +105%; `baseline` not recorded on the laptop against +514.9%, then +521.9% on a rerun. - The PR edited `multi`'s own `go.mod` and `go.sum` and nothing in `largeidle`'s module, yet `largeidle` read ~212%, about as far over its ceiling as `multi`. When an untouched scenario moves like a touched one, suspect the machine before the diff. - otelc always injects the OpenTelemetry SDK initialization package (per the [#650](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/650) description), so an SDK bump can reach every scenario. CI settles it: `largeidle` sat at +105% on CI with the SDK bump already in, a hundred points under the laptop's reading. - `minBuildTime` runs a full rebuild (`go build -a`) three times and keeps the fastest, but all three plain builds run before all three otelc builds. Load that arrives between the two halves lands entirely in the percentage. - A machine that is uniformly slower cancels out of a ratio: in the CI rerun, `baseline` plain went from 2.146s to 3.432s and otelc from 13.197s to 21.344s, both up about 60%, and the overhead moved only from +514.9% to +521.9%. - CI's `baseline` failure was real. The author's comment on the PR and the #650 description both put it down to the SDK bump: `baseline` compiles a program with one `fmt` call, so the fixed cost of compiling the SDK dominates its percentage. - #650 merged on 6 July and raised `baseline`'s ceiling from 500% to 550%. The maintainers loosened the limit rather than hold back the dependency bump. - The overhead job declares `runs-on: ubuntu-latest`. That label names an image; it does not say how busy the host was. The workflow's other job, the CodSpeed walltime run, names a bare-metal machine instead. - Sourcing caveats: the laptop numbers are first-hand recollection (the tildes), not recorded in the PR; the CI job logs expired by 2 October (GitHub answers HTTP 410), so everything from CI comes from PR comments; it was one run per side, from different machines, and the identical commit is not guaranteed. - Disclosure: the author works at Datadog and is one of otelc's maintainers. Versions, links and commands were checked on 2 October 2026. ## What the otelc overhead check measures [otelc](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation) instruments Go programs at compile time, through `go build -toolexec`. Every package in a build spawns an otelc process, even when no rule matches it. The [benchmarking doc at v1.1.0](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/v1.1.0/docs/benchmarking.md) calls that overhead "the primary UX metric to track", and the check times three scenarios: | Scenario | What it builds | | --- | --- | | `baseline` | stdlib only (`fmt`), no rule matches | | `multi` | `net/http`, gRPC, `database/sql`, Redis and client-go, with 6–8 instrumented packages | | `largeidle` | heavy stdlib plus many third-party libraries, zero instrumented packages | The overhead is `(otelc − plain) / plain`, so 230% means the instrumented build took about 3.3 times as long. The ceiling is 150% for every scenario except `baseline`, which gets 500%. Here is the heart of the [test as it stood when #643 merged](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/5911ac776cd58d0ba98d57eb3084e522bb0ddb44/test/bench/overhead_test.go): ```go plain := minBuildTime(t, dir, plainArgs, name) otelcTime := minBuildTime(t, dir, otelcArgs, name) pct := (otelcTime - plain) / plain * 100 ``` `minBuildTime` runs a full rebuild (`go build -a`) three times and keeps the fastest, a common way to cut noise. The order matters. All three plain builds run first, then all three otelc builds. If the machine changes its mood between the two halves, the percentage absorbs the whole difference. This is a macrobenchmark: a whole build timed end to end, unlike the microbenchmarks in the rest of the series. It did not fit the [GopherCon UK talk](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/), and it sits between [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Part 3 (the laptop)]] and [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Part 4 (CI)]] in the [series](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/). ## Two numbers for one change: laptop vs CI On 3 July the author ran the overhead check on their own machine, which had been running parallel builds and integration tests all session. CI ran the same check that day. | Scenario | Laptop | CI | Ceiling | | --- | --- | --- | --- | | `multi` | ~230% | +81% | 150% | | `largeidle` | ~212% | +105% | 150% | | `baseline` | not recorded | +514.9%, then +521.9% on a rerun | 500% | Read cold, the laptop column says the bump broke two scenarios. The CI column says only `baseline` fails. The sourcing is asymmetric: the CI numbers and the PR discussion are public, while the laptop numbers are the author's recollection of the terminal, hence the tildes. It was one run on each side, from different machines, and the author cannot promise both ran the identical commit. ## The scope tell: the scenario that could not have moved Ask what the change could have touched. The PR edited `multi`'s own `go.mod` and `go.sum`, and nothing in `largeidle`'s module. Yet `largeidle` came out at ~212%, about as far over its ceiling as `multi`. When a scenario the change didn't touch moves by the same amount as one it did, suspect the machine before the diff. The scenario was not a perfectly clean control. The description of the later fix, [#650](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/650), says otelc always injects the OpenTelemetry SDK initialization package, so an SDK bump can reach every scenario, `largeidle` included. CI settles it: `largeidle` sat at +105% on CI, a hundred points under the laptop's reading, with the SDK bump already in. Whatever the bump cost, it doesn't explain 212%. The author waited for CI instead of hunting for a regression in code that hadn't changed, never reran the check on a quiet machine, and didn't log CPU pressure, so "load did it" is an attribution. It fits every number above; it can't be proven. The rerun numbers show which load the ratio absorbs. [In the rerun on the PR](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/643#issuecomment-4875740809), `baseline` went from 2.146s plain and 13.197s with otelc to 3.432s and 21.344s. Both times grew by about 60%, and the overhead moved from +514.9% to +521.9%. A machine that is uniformly slower cancels out of a ratio, which is why CI's failure repeated so steadily. What doesn't cancel is load that arrives between the plain half and the otelc half, or that hits one build harder than the other. A full `-a` rebuild fans out across every core, so parallel builds on the same machine are a likely source. That part is a guess too. Next time a number looks alarming, ask the cheap question first: what scenario in this run could not have changed? If it moved too, the diff is innocent until proven otherwise. Then check how the test orders its halves, since a ratio only protects you from noise that lands on both sides. The laptop said 230% and CI said 81%; the 230% said more about the machine than about the pull request, and the question that showed it cost nothing to ask. ## Why baseline failed in CI CI's failure was real. Only `baseline` crossed its ceiling, and [the author's comment on the PR](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/643#issuecomment-4875740809) and the [#650](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/650) description both put it down to the SDK bump. The reason `baseline` feels it most: it compiles a program with one `fmt` call, so a fixed cost for compiling the SDK dominates its percentage. #650 merged on 6 July and raised that ceiling from 500% to 550%. The maintainers loosened the limit rather than hold back the dependency bump. Judging a real measurement is a different job from catching a lying one. ## CI is a second opinion, not the truth At the commit where #643 merged, the [overhead job](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/5911ac776cd58d0ba98d57eb3084e522bb0ddb44/.github/workflows/benchmark.yaml) declares `runs-on: ubuntu-latest`. That label names an image. It doesn't tell you how busy the host was. The workflow's other job, the CodSpeed walltime run, names a bare-metal machine instead. What else was running next to the overhead check is unknown. What CI gave was a second measurement with different noise, and the two disagreed in a pattern that could be read. Shared runners add their own kind of weather, which is [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Part 4]]'s territory. Two readings and a scope argument beat one reading and a hunch. ## FAQ ### Why do my benchmark numbers differ between my laptop and CI? The two machines have different noise. Here the laptop had been running parallel builds and integration tests all session, and the check runs all plain builds before all otelc builds, so load between the halves lands entirely in the percentage. CI measured the same change at +81% where the laptop read ~230%. ### Which number should I trust: local benchmark or CI? Neither is the truth. CI gave a second measurement with different noise, and the disagreement between the two was readable: the untouched scenario moved on the laptop only. Two readings and a scope argument beat one reading and a hunch. ### How do I tell if a benchmark regression is my code or the machine? Ask what scenario in the run could not have changed. The PR edited `multi`'s `go.mod` and `go.sum` and nothing in `largeidle`'s module, yet `largeidle` moved by the same amount as `multi`. When that happens, suspect the machine before the diff. ### Why doesn't a slower machine change an overhead percentage? The overhead is a ratio. A machine that is uniformly slower makes both halves slower by the same factor, which cancels out: in the CI rerun both plain and otelc times grew about 60% and the overhead moved only from +514.9% to +521.9%. Load that arrives between the plain half and the otelc half does not cancel. ### What was the otelc overhead check in PR #643? It times three full builds (`baseline`, `multi`, `largeidle`), keeps the fastest of three `go build -a` runs per side, and computes `(otelc − plain) / plain * 100`. The ceiling is 150% per scenario, 500% for `baseline`. ### Why did baseline fail CI in otelc PR #643? The SDK bump. otelc always injects the OpenTelemetry SDK initialization package, and `baseline` compiles a program with one `fmt` call, so the fixed cost of compiling the SDK dominates its percentage: +514.9%, then +521.9% on a rerun. #650 raised that ceiling from 500% to 550% rather than hold back the dependency bump. ### Should I rerun a suspicious benchmark on a quiet machine? That is the check this story never did. The author never reran the overhead check on a quiet machine and never logged CPU pressure, so "load did it" stayed an attribution that fit every number but could not be proven. ### What does runs-on: ubuntu-latest mean for benchmark reliability? It names an image, not a host. It does not tell you how busy the machine was when your benchmark ran. The same workflow's CodSpeed walltime job names a bare-metal machine instead. ## Related - [[resources/documentation/Go/Benchmarks/Go Benchmarks|Go Benchmarks]] (series hub) - [[resources/documentation/Go/Benchmarks/Go Benchmarks - Local Reproduction on a Laptop|Local Reproduction on a Laptop]] (Part 3: the laptop, microbenchmarks) - [[resources/documentation/Go/Benchmarks/Go Benchmarks - Benchmark CI Noise|Benchmark CI Noise]] (Part 4: shared runners and CI noise) ## Sources - [Why Your Go Benchmarks Are Lying (talk, GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/why-your-go-benchmarks-are-lying/) - [Why Your Go Benchmarks Are Lying (series)](https://kakkoyun.me/series/why-your-go-benchmarks-are-lying/) - [otelc: opentelemetry-go-compile-instrumentation](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation) - [otelc PR #643](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/643) - [Rerun numbers, comment on PR #643](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/643#issuecomment-4875740809) - [otelc PR #650](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/pull/650) - [overhead_test.go at the #643 merge commit](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/5911ac776cd58d0ba98d57eb3084e522bb0ddb44/test/bench/overhead_test.go) - [benchmark.yaml workflow at the #643 merge commit](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/5911ac776cd58d0ba98d57eb3084e522bb0ddb44/.github/workflows/benchmark.yaml) - [otelc benchmarking doc at v1.1.0](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/v1.1.0/docs/benchmarking.md)