Engineers overrun estimates all the time. But this wasn’t a planning failure, it was something I’d spent my whole adult life recognizing too late: hyperfocus. The kind where time doesn’t slow down, it simply stops existing. My coffee went cold. I missed lunch. I blew past my 6pm hard stop with more working code than I’d shipped in the previous three days combined.

The twist: I was working with Claude Code. And I hadn’t been writing much code at all.

I was first told I had ADHD as a teenager. My mother, a pharmacist, disagreed; not with the diagnosis exactly, but with what the diagnosis implied: medication. So that chapter closed as quickly as it opened. I went on to do what a lot of undiagnosed ADHD people do: I built a career by outrunning my own brain. Long (inefficient) hours, hyperfocus weaponized as a coping mechanism, elaborate external systems to compensate for the executive dysfunction I didn’t have a name for yet.

In 2021, in my early thirties, I got the diagnosis again. Properly this time, with medication, with a therapist, with actual context for the previous three decades. It felt less like a revelation and more like finally seeing the subtitles to a film I’d been watching my whole life.

Late diagnosis doesn’t fix anything. But it does give you a map.

So here I am in 2026, holding this map, and the territory has changed completely.

AI coding agents arrived and immediately did something strange for my brain: they fit. Not perfectly. But better than almost any other tool I’d used.

Think about how executive dysfunction actually works. The bridge between “I know what to do” and “I am doing it” is missing or unreliable. A structural gap in the signaling, not laziness. You can stare at a task for forty minutes, completely aware of every step required, completely unable to begin the first one. The friction of starting is enormous.

These tools lowered that friction to almost nothing.

“Hey, I want to explore this problem. Start here.” You don’t have to have the architecture worked out. You don’t have to have the perfect name for the function yet. The act of beginning (which ADHD makes so hard) becomes nearly trivial. The conversation is the scaffold. The model fills in the gaps you’d normally spend executive resources on. You can start badly, vaguely, and iterate forward.

This is not how most software development tools work. Most tools assume you’ve already started.

But here’s the part I didn’t see coming.

ADHD brains are famously bad at dopamine regulation. The reason tasks are hard to start is the same reason they’re hard to stop: we don’t have a reliable internal signal that says enough. The dopamine hit from making progress is chemically addictive for a brain that doesn’t produce it reliably otherwise. Normally, the friction of the work itself acts as a natural brake: it gets hard, you pause, time passes, you come up for air. ¹

When the friction is gone, the brake is gone too.

Steve Yegge wrote about this in February as a vampire dynamic: AI makes you 10x more productive (his number, not mine), companies capture all of that 10x, and the worker burns out at 10x speed. That’s a real structural problem. But for ADHD people, the vampire was already inside the house. I don’t need anyone to extract my productivity at scale. I’ll do it to myself. I already do it to myself. The exhaustion isn’t new, the acceleration is.

What I need is a new way to answer a question my brain has always been bad at: when is enough?

Here’s what I’ve learned, imperfectly, in the past few years (feels like a decade).

Hyperfocus is a superpower with an invoice attached. AI amplifies it in both directions. A task that would have taken hours of overhead to warm up now lets me get to the interesting part immediately, and the interesting part is exactly where my brain thrives. I have lost more evenings, more “just thirty minutes” that became five hours. The superpower doesn’t come with a volume knob. You have to install one yourself.

The invoice used to be paid differently. In my twenties I worked without a stop signal — long hours, contract gigs stacked onto the weekends, no life outside the work. Every two or three weeks the bill would arrive all at once: a Sunday I couldn’t get out of bed. That doesn’t work anymore. There’s a kid now, and a 6pm hard cutoff, and the invoice can’t be settled in one Sunday-shaped lump. Which means I have to see it building, sooner.

Time blindness hasn’t gone anywhere (my wife has a thing or two to say about this). Timers help. What actually works for me is externalizing time the same way I externalize memory: make it visible, make it loud, make it impossible to ignore. The agent tools won’t do this for you. They have no concept of your dinner getting cold.

Executive dysfunction loves these tools, and that relief is also a trap. When starting is easier, I start more things. More half-finished contexts, more “I’ll just switch over to this for a second”, more fragmented attention. One bottleneck removed reveals the one behind it. I’m faster at beginning; I’m not automatically better at finishing.

None of this is a reason not to use these tools. It’s a reason to use them with a clearer picture of your own brain.

What the map tells me now (the one I waited until my thirties to find) is that my brain was never broken. It was mismatched: to school structures, to open-plan offices, to 9am standups, to the way most software development was expected to happen. I built compensating systems for all of it and called the effort “discipline.”

Some of those compensating systems are obsolete now. The friction I trained myself to push through is genuinely gone. The new question is: what does discipline mean when the friction is gone?

I don’t fully know yet. I suspect it means learning to stop in a world that no longer stops you.

After thirty-something years of a brain that felt like it was fighting itself, having a problem that’s about too much flow instead of not enough is a strange and mostly welcome inversion. I’ll take it.

But I’m also tired.

Above the fold: a Whoop recovery score of 62 (fine), a Wakatime row showing 4h 21m on the OpenTelemetry Go compile-time instrumentation repo yesterday, three GitHub PRs waiting on review, and a Things 3 task that has been sliding forward for nine days: Review that thing from X. Each previous day the task migrated, and the journal entry from that day is one click away. The reason it kept slipping is in those entries somewhere.

The system did not make the task less unpleasant. It made the unpleasantness traceable. That is the smaller of the two things this setup buys me. The larger one is the synthesis layer: an LLM reads what I capture and writes connections back into the vault without editing the original notes.

This is what I built, plus the one design choice I think makes the whole thing work. I am going to lead with the boring parts on purpose.

The 80% that does the work

Three workhorses. None of them are interesting.

The first is PARA (Projects, Areas, Resources, Archives). It gives roughly 4,000 Markdown files in an Obsidian vault an obvious home, which is its only job. People who prefer Johnny Decimal, ACCESS, or their own taxonomies are correct that PARA has rough edges. They are also wrong that the rough edges matter. The point of PARA is to stop you from re-litigating where things go.

The second is Readwise. It syncs highlights from Kindle, Reader, Twitter (yes, still Twitter for me), and any podcast clipping I tag into curation/readwise/. I do not curate at this layer. Everything lands. A highlight from a book I read in 2023 is still there.

The third is obsidian-git. It commits the vault every 70 minutes and pushes every 7. I have not thought about backups in months.

The Whoop score, the Wakatime row, and the PR queue in that morning’s daily note came from a Python CLI I wrote called pkm-tool. It pulled GitHub, Jira, Wakatime, Whoop, Apple Calendar, Things 3, and Google Docs into one report. The experiment worked, but it also turned my daily note into a small integration platform I had to maintain.

I no longer use it. Today I prefer small deterministic scripts, Obsidian cron jobs, and official connectors where they exist. A handful of small capture jobs now assemble the note instead of one aggregator.

That is the substrate. Other people have described it before. If the post ended here, it would not be worth your time.

The one design choice

The part I care about most is the boundary between source material and synthesis.

The LLM never edits a source.

Most AI-PKM tools work by chewing on your notes and offering to rewrite, tag, or restructure them. Copilot for Obsidian edits selected text in-editor. Notion AI is happy to rewrite a page. The implicit contract is: your notes are a substrate the AI improves.

I do not want that. I want my raw captures to stay exactly as they were when they hit the disk. That includes Readwise highlights, meeting notes I scribbled in a hurry, and half-baked journal entries. Those are the historical record. If a synthesis is wrong, I want to look at the source and see why it went wrong, without finding that the LLM has already “improved” the source out from under me.

The vault has two halves with a wall between them. On one side are raw sources such as curation/, journal/, devlog/, and meetings/. On the other are pages written and maintained by an LLM (Claude Code in my case): zettelkasten/, resources/concepts/, decisions/, and reflections/.

The synthesis layer can be wrong, and I can regenerate it. The raw layer remains the source of truth.

This is the Karpathy LLM Wiki pattern applied to a PARA vault. The schema lives in system/wiki/SCHEMA.md and declares three operations the LLM is allowed to perform. Ingest reads a new source and writes synthesis pages without touching the source. Query answers a question and files the answer back as new wiki content if it produced new insight. Lint walks the synthesis graph looking for orphans, stale references, and contradictions to flag for me to resolve.

If I had to defend the architecture in one sentence: it lets knowledge accumulate across LLM sessions without letting the LLM rewrite the past. Almost every other choice in the setup followed from that.

Where it paid off

The retrieval engine is qmd, Tobi Lütke’s local-first CLI. It combines BM25, vector embeddings, and local models for query expansion and reranking without sending my notes to an API.

On July 21, while preparing a GopherCon UK talk, I ran:

1

qmd query "benchmark overhead auto-instrumentation Go latency allocations" -n 8

It found an older GopherCon proposal in creation/, a raw auto-instrumentation note imported from iCloud, a saved article about measuring Go performance, and work notes on auto-instrumentation and code origins. None of that material was new. I had forgotten some of those notes existed. The agent used those documents as the starting material for the talk instead of rebuilding the context from scratch.

On August 3, the synthesis side did something different. A connection pass updated resources/concepts/Go Programming Expertise.md from a July 24 daily note about Go 1.24 and 1.25 and an August 3 work devlog about benchmark internals. The page links the release-tracking habit to the compiler and benchmarking work behind the talk, cites both source notes, and labels its confidence as low.

That combination is why I trust the design: qmd finds old material, and the wiki records connections without changing the sources. I can follow every claim back to the note that produced it.

If you are starting from zero

If I rebuilt it, I would do these in this order.

Get retrieval working before synthesis. Run qmd or another local search over whatever notes you have today. If you cannot reliably find what you have, the synthesis layer will not save you. Most of the value of a “second brain” is retrieval, not generation.

Do not sync everything immediately. Resist the urge to dump every Twitter favorite, web clip, and podcast highlight into one folder on day one. Start with one capture channel, such as books, articles you actually read, or your daily journal, and live with it until “I captured something today and I can find it later” feels routine. Add channels once that rhythm exists.

Be honest about what the wiki layer needs from you. The LLM Wiki pattern only improves if you actually run it. My ingestion logs are busy now, but the Readwise backlog is still waiting. The structure works. The practice is still catching up. If you are skeptical of “second brain” hype because most setups die at the synthesis layer, you are not wrong. Mine has not proven itself over the long term either.

The capture layer still has failure modes. The live jobs run inside Obsidian through Cron, Shell Commands, Templater, and connectors, so they depend on Obsidian being open. A failed capture can leave a section empty without making the note look broken. I still do not have one health check for the whole pipeline.

What I cannot tell you yet

When the LLM maintains the synthesis pages, and I read one of those pages six months from now, will it feel like my thinking or like a summary someone handed me?

The Karpathy pattern makes that trade: citations point back to raw sources, and the synthesis can be regenerated as the wiki changes. Tiago Forte describes progressive summarization as “opportunistic compression” that makes notes discoverable later. I am testing what changes when the summarizer is not me.

I am not yet sure I agree. The synthesis pages in my vault from the last month read fluently and cite real sources and make connections I would not have made unaided. They also do not always feel like me. Something about the shape of the paragraphs. Something about which details get kept and which get smoothed away.

I do not know how this resolves. It is possible that in six months the synthesis layer will feel like a colleague I trust, and that is fine. It is possible it will feel like reading someone else’s notes about my own life, and that is not fine. I am betting the answer is closer to the first, with limited evidence.

If you build something like this and end up on the other side of the question, I would genuinely like to hear about it.

The part I enjoyed arguing about most was blast radius. Manual instrumentation is precise and tedious, and it never finishes. eBPF buys you one agent covering every language on the box, at the cost of running code in the kernel, where a bad program can take the whole host down with it. Compile-time instrumentation rewrites the syntax tree behind go build through the -toolexec hook, which keeps failures scoped to one service and works anywhere Go compiles. People love this dark magic, which I find either reassuring or alarming depending on the day.

We also got into why a trace and a pprof profile answer different questions, how Orchestrion ended up under OpenTelemetry governance, and the least intimidating route into a SIG if you want to start contributing. Shay’s bicycle makes an appearance. I was speaking for myself rather than reading from a Datadog brochure, which I hope comes through.

Listen

Episode

• Cup o’ Go E167 — Quick GopherCon recap, and interview with Kemal of Datadog
  • Watch on YouTube
  • Transcript
• Apple Podcasts
• Spotify

Topics

• GopherCon US recap, and Austin’s talk on fixing the proposal process
• What APM measures, and how request tracing differs from pprof sampling
• Manual, eBPF, and compile-time instrumentation, and what each one costs you
• Orchestrion moving to OpenTelemetry, plus how to find your way into a SIG
• Checklocks, the gVisor static lock checker I keep recommending to people

Related

• talk: Instrument Go Without Changing a Single Line — GopherCon UK 2026
• talk: How to Instrument Go Without Changing a Single Line of Code — FOSDEM 2026
• talk: Unleashing the Go Toolchain — what -toolexec makes possible
• Auto-Instrumenting Go: From eBPF to USDT Probes
• OpenTelemetry Go Compile-time Instrumentation v1

The runbook starts with one boring question: is this Go service already running, or can I rebuild it? If the service has to stay untouched, OBI (OpenTelemetry eBPF Instrumentation) is the default. If I can rebuild it and need function-level spans, otelc (OpenTelemetry Go compile-time instrumentation) is the default. The rest of the decision tree is checking privileges, Go version, collector endpoint, and whether RED metrics are enough.

We’ll keep that loop in view the whole way (yes, that’s foreshadowing 👀) and see whether the series promise survives contact with it. This is part 6 of 6 in the How to Instrument Go Without Changing a Single Line of Code series. Part 5 looked at what Go itself might add, and parts 2 to 4 covered the tools (OBI, otelc, the eBPF profiler). Disclosure: I’m one of otelc’s maintainers, and zeroins, kubectl-obi and the skill in this part are my own projects. The skill and plugins below live in the zeroins repository.

The routing logic

The series follows three intervention points: build time, process start and the kernel. For a Go service, two of them are real routes. We reach for OBI when the service is already running (production, staging, Kubernetes, Docker Compose) and rebuilding or redeploying isn’t acceptable, when the fleet is polyglot, or when HTTP/gRPC RED metrics (rate, errors, duration) and library-level spans are all we need. We reach for otelc when we control the build (a laptop, CI or the release pipeline), want granular spans for specific functions or one slow path, and are willing to rebuild with otelc go build. If neither fits cleanly, the tiebreaker is one question: can this service be rebuilt and redeployed (otelc), or does it have to stay untouched (OBI)?

The profiler from part 4 isn’t a third route but a layer beside either one, adding whole-node CPU profiles. A practical stack is otelc in the build for semantic spans, OBI for service boundaries and other languages, and the profiler for the node; add a layer when its signal earns the cost. The flight recorder lives in your code, so it stays out of the runbook. Orchestrion is not in this tree, since the zeroins lookups cover only OBI and otelc (part 3 still points to it for Datadog APM).

Both routes need somewhere to send the output, such as an OTel Collector or Jaeger. OBI needs a BTF-enabled Linux kernel and a specific capability set (part 2 has the list), and otelc needs Go 1.25 or newer.

What happened to process start? We left it out on purpose. The OpenTelemetry host injector preloads agents for Java, Node.js, .NET, Ruby, and Python (disabled by default), not Go, and a pure-Go build is static, so LD_PRELOAD has nothing to attach to. The OpenTelemetry Operator’s inject-go annotation doesn’t change that: it adds a per-pod sidecar running the Go auto-instrumentation agent (go.opentelemetry.io/auto, not OBI), off by default, that needs a privileged root container. A fleet-wide injector can keep the other runtimes while Go services route to OBI, or to otelc if you can rebuild (part 1 has the background). Two routes remain. Let’s see what each one buys us, starting with the one that leaves our binary alone.

What OBI actually covers

OBI instruments 13 Go libraries out of the box, including net/http, gin, gRPC and database/sql (the list is in part 2). It attaches from outside the process: no LD_PRELOAD, no rebuild, and in DaemonSet mode no restart. It also skips services it sees publishing their own OpenTelemetry telemetry (exclude_otel_instrumented_services, on by default), so an OTel Java or Python agent shouldn’t double up; the docs describe this for OpenTelemetry SDK calls, so test a pod with a vendor agent first. On Kubernetes that DaemonSet comes from a Helm chart, and the defaults of chart 0.10.0 (open-telemetry/opentelemetry-ebpf-instrumentation) are enthusiastic: every namespace, privileged, exporting to a collector on the node IP (${HOST_IP}:4317 for traces, :4318 for metrics). Point it elsewhere before installing it on a cluster you care about.

The cost is span depth. Custom spans and business-logic events still need code changes when they are not at the HTTP/gRPC boundary. OBI observes what crosses library boundaries and can’t see inside our logic, so when the slow part is in our own functions, we need the other route.

What otelc covers

otelc takes the compile-time path: otelc go build is a drop-in for go build. It instruments your code, its dependencies, and parts of the standard library at the AST level, and the injected spans cost the same as manually written OTel code. Here is the entire change: install the tool, swap the build command, run the binary with an endpoint and a service name.

1
2
3
4
5
6

go install go.opentelemetry.io/otelc/tool/cmd/otelc@v1.1.0
otelc go build -o ./myapp .

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_SERVICE_NAME=my-go-service \
./myapp

That is what “without a single line” means in practice: not one line of application source, one changed line in the build script. The endpoint says 4318 on purpose. otelc’s runtime exports OTLP over HTTP (http/protobuf) by default, and 4317 is the gRPC port. Point the default protocol at 4317 and you get malformed HTTP response errors instead of traces (wrong door, polite knock); to use 4317, set OTEL_EXPORTER_OTLP_PROTOCOL=grpc.

The tradeoff is build-time coupling: we can’t attach otelc to a running binary, or instrument services we are not rebuilding. We can put it in the release pipeline, though. The getting-started guide covers CI/CD and calls it stable and ready for production use as of v1.0.0. Two tools, two sets of costs, and a choice an impatient human will get wrong at 3 a.m. Let’s write it down so an agent can follow it.

Making the decision executable

A decision tree in a blog post doesn’t shorten anyone’s loop. As the talk puts it, an agent on its own still has to pick a mechanism and know what it costs. I put the routing logic into an agent skill called collect-go-telemetry, installed with npx skills add kakkoyun/zeroins --all. It encodes the decision tree and wraps it in a fixed workflow: preflight, coverage lookup, dry-run plan, confirmation gate, bounded attach, confirm traces, session audit. The tree is fixed; the context (which service, which libraries, which environment) is what an agent inspects.

The coverage step is the clever bit. Pasting OBI’s support matrix and otelc’s library catalog into context for every request is wasteful, so the skill uses per-library lookups (zeroins obi lookup <library>, zeroins otelc lookup <import-path>) that return only the relevant rows. They are offline, reading a snapshot embedded in the binary (OBI v0.10.0 and otelc v1.0.1 in zeroins v0.2.0), so they change only when zeroins does. Treat documentation as a retrieval problem, not a context-stuffing problem. The skill can now choose and look things up, but on Kubernetes something still has to do the attaching.

zeroins and kubectl-obi: the Kubernetes path

That something is kubectl-obi, one of two experimental kubectl plugins in zeroins, which I wrote for the talk. go install puts an executable named kubectl-obi on your PATH, and kubectl discovers it as kubectl obi. It deploys privileged eBPF observers and is not a production installer, so the first thing we do is read the dry-run. It shells out to helm and kubectl (zeroins asks for helm v4), and --endpoint is required. Here is the lifecycle, dry-run first:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

go install github.com/kakkoyun/zeroins/cmd/...@v0.2.0

# Review the plan first (changes nothing)
kubectl obi attach --dry-run --endpoint=https://otel-collector.observability.svc:4318

# Instrument everything on every node, detach automatically after 15 minutes
kubectl obi attach --duration=15m --endpoint=https://otel-collector.observability.svc:4318

# Check what's being instrumented
kubectl obi status --all-namespaces

# Stop instrumenting
kubectl obi detach

The endpoint must be an OTLP HTTP(S) base URL, so no /v1/traces suffix. In DaemonSet mode the plugin installs the pinned OBI chart (0.10.0, OBI v0.10.0, the release pinned for the talk on 13 August 2026) into the obi-system namespace with Helm. Sidecar mode also exists; it patches one deployment and restarts it, so the “no restart” promise holds only for the DaemonSet. Sidecar mode needs its own kubectl obi detach <deployment> --mode=sidecar; a bare detach removes only the DaemonSet. The whole thing is experimental: CI runs unit tests and Helm contract checks, and the real-eBPF integration test on a disposable K3s cluster last passed shortly before the v0.2.0 tag.

Handing an agent a command that attaches privileged eBPF programs to a cluster is exactly as fun as it sounds. Hence the guardrails.

Guardrails before attach

The messy operational checks deserve names: kernel version, capabilities, collector endpoint, rebuild access, Go version, and span depth. I still would not let a skill silently attach eBPF to production, so the skill doesn’t. It produces a plan first (attach --dry-run prints the cluster context, the privileges, the exact commands and the rendered values), then stops at a confirmation gate until a human approves. Exploratory attaches take --duration and detach themselves, and zeroins sessions list and zeroins sessions reap show and clean up what is still attached. Boring, reversible, logged. Observability needs more of that kind of magic trick. That’s the whole runbook, so let’s run the safe parts of it.

Try it

Nothing here touches a cluster. First, a local trace backend for the otelc example above, using Jaeger. Jaeger only accepts traces, so the metrics flush logs a 404 when the process shuts down. The traces arrive anyway:

1
2
3
4

docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:2.21.0

# build and run any small net/http service as shown in "What otelc covers", send a few requests, then:
curl -s localhost:16686/api/v3/services

The services list should include my-go-service. Then the zeroins side, which only looks things up and prints a plan:

1
2
3

zeroins obi lookup net/http
zeroins otelc lookup net/http
kubectl obi attach --dry-run --endpoint=https://otel-collector.observability.svc:4318

The two lookups say which route covers net/http, and the dry-run prints the plan a human would be asked to approve.

Versions, links and commands checked on 2 October 2026.

Where to go from here

Back to the loop. The bug that won’t reproduce locally used to cost a log line and a redeploy, then another when the first guess landed in the wrong place. Now the first move is the boring question from the top of this post. If the service has to stay untouched, the plan is one dry-run away and, in DaemonSet mode, nothing restarts. If the answer lives in our own functions, the question sends us to otelc, and the rebuild carries no source edit. Either way, we stop guessing where the log line goes.

That is the series. The talk page has the slides and links, and the FOSDEM 2026 post has the write-up of the earlier version of the talk. To help out, OBI, otelc and the eBPF profiler each have an OpenTelemetry SIG (see the community repository), and Ollygarden’s agent skills are a good next stop for more OpenTelemetry skills.

Two routes, one boring question, no source edits. Go break the loop. 🔧

A loose cable

Physicists have been fooled the same way, at a much larger scale. In September 2011, the OPERA collaboration announced that muon neutrinos appeared to travel faster than the speed of light. Months of rechecking found nothing wrong. The root cause, eventually, was an improperly seated fibre-optic connector in the GPS timing chain, which introduced a ~73 ns bias that made neutrinos appear to arrive early (Science called it a loose cable). A second fault, an oscillator defect, pushed the other way and partly masked the first. Once both were corrected, the 2012 re-measurements showed neutrino speed consistent with the speed of light.

A systematic measurement error can hide in plain sight, look exactly like signal, and survive review by people far more careful than we are. Our Go benchmarks have testing.B, a laptop, and background Chrome tabs. They have cables too: the compiler, the statistics, and the machine with its OS scheduler. We’ll follow one loose cable through this post. I’ll show you mine first, and then we’ll see which question would have caught it.

This is part 5 of 5 in the Why Your Go Benchmarks Are Lying series, the written companion to the GopherCon UK 2026 talk. Part 4 built benchmark CI that holds up (with the companions A PR gate that actually fails and A/B is the wrong model for CI); this part asks whether we should believe it. The FOSDEM 2026 post on measuring software performance is the prerequisite. The tools live in benchlab and the talk’s demo results in gopherconuk-26. Disclosure: I work at Datadog, which maintains dd-trace-go, so the story is about my employer’s CI bot. The lesson holds for any repository.

Enough physics. Here is my loose cable.

The CI regression that was a speedup

In June 2026 I pushed a change touching context.go in the ddtrace/tracer package, which landed as dd-trace-go #4891. The change was compile-time instrumentation plumbing. Shortly after the push, the benchmark bot commented that BenchmarkOTLPProtoSize was 6–9% slower than main.

My first instinct was to suspect my change. The better move is to read what the benchmark measures first, so here is its timed loop:

1
2
3
4

// The entire timed loop inside BenchmarkOTLPProtoSize:
for b.Loop() {
	proto.Size(tracesData)
}

That is a protobuf size computation on a struct assembled entirely before the loop (the real benchmark still calls b.ResetTimer() there, which b.Loop makes redundant). It never calls ContextWithSpan, SpanFromContext, or any code the PR modified, so the diff had no believable path to the number. Hold that thought, because the likely mechanism is stranger.

Did the regression show up on my own machine? Running the benchmark repeatedly there gave a coefficient of variation (CV) under 0.1%, so a 6–9% gap was not ordinary variance. I built main and #4891 on that same machine, an Apple M4 Max (darwin/arm64), and compared them with benchstat. The table shows the medians:

#4891 was faster. CI had flagged a regression, and the same-machine A/B showed the opposite.

The likely mechanism is code layout. Changing context.go shifted function addresses in the test binary, which moved the hot proto.Size loop relative to cache-line and branch-target-buffer boundaries. At the sub-microsecond scale of the one-span case, a small alignment shift can swing a result by a few percent in either direction, enough to flip the verdict from “improvement” to “regression”. Emery Berger’s Performance Matters (Strange Loop 2019) puts code layout alone at ±10%.

The evidence is circumstantial. A later comparison of symbol addresses in linux/amd64 test binaries, built from the base commit and from the PR’s merge commit, shows the hot protobuf size functions (proto.MarshalOptions.size, impl.(*MessageInfo).sizePointer) each moved by 96 bytes, and the benchmark closure by 160. Those are amd64 binaries, not necessarily what CI ran, so the layout is indicative only, and no hardware-counter measurement ties the shift to the delta.

The bot’s own history on the PR fits the layout reading. From 12 June, each update of its comment said four regressions, 6–9% at every span count. On 19 June, after main had moved, the same comment said four improvements of 6–8% for the same benchmark. That is not noise around zero. The bias was stable and changed sign when main moved, the same shape as OPERA: a systematic error that looks like signal.

The resolution: nothing. No code change for the benchmark. A speculative “fix” to quiet it would have been chasing shadows.

At the time, these benchmarks ran on shared CI runners. We have since moved them to dedicated bare-metal machines, which takes the noisy neighbours out of the picture. It doesn’t take code layout out, and neither does a same-machine A/B, since the two binaries still differ in layout. It removes the machine. CI said +6 to 9% and my machine about −5% for a benchmark whose loop the PR never touched; a sign that flips between machines points at layout more than at code. Neither machine is authoritative by default; the flip is the tell to go and look at layout. The same benchmark tripped again on #4926, eleven days after #4891 was opened, with +6.5–8.5% on the same four sub-benchmarks. That time the flag was dismissed on sight as a known false positive: a code-layout artifact, with a local A/B of about +0.3%.

OPERA and #4891 teach the same thing. A number can be reproducible and still be directionally wrong, and a gate that is directionally wrong blocks good changes and waves bad ones through. One loose cable is bad luck. Knowing which cables to check is not, and the series gave us three.

Three questions

Each earlier part went after one way a Go benchmark can mislead us. Side by side, they make a checklist to run before merging anything on a hot path:

Each question gates the next. A benchmark the compiler has optimised away answers question 2 with noise, and a noisy environment makes question 3 unanswerable whatever the sample size. A checklist nobody runs is decoration, so each question gets a tool.

Wire it up this afternoon

The three CLIs from the talk are in benchlab, my own project, one per question. Everything here describes v0.1.0, the tag pinned for the talk on 12 August 2026. They share one stdlib-only Go module (go 1.24). Pin the tag, because unreleased work on main changes some of these flags:

1

go install github.com/kakkoyun/benchlab/cmd/...@v0.1.0

We’ll take them in question order, starting with the compiler.

honestbench

honestbench answers question 1. It walks *_test.go files with go/ast and flags results discarded after computation (dead-code elimination candidates), missing sink patterns, StopTimer/StartTimer misordering, and b.N loops that should migrate to testing.B.Loop, introduced in Go 1.24. It exits 1 on findings, so it works as a CI gate:

1

honestbench -r ./...

-r recurses into subdirectories, -json prints machine-readable output and -q prints findings only. Exit codes are 0 for clean, 1 for findings and 2 for an error. We run it before reading a single ns/op, because a finding on a b.N loop means the benchmark probably measures something other than what we think.

One caveat: v0.1.0 also reports discarded results inside for b.Loop() bodies, where the testing package documents that call results are kept alive. That includes the proto.Size loop from my story. Treat a finding there as a prompt to look, not as proof. (Yes, my own linter flags my own story. Tools are measurements too.) Once the compiler is honest, the next question is whether the sample is.

benchgate

benchgate answers question 2. It runs benchmarks N times, computes the coefficient of variation (CV) per benchmark, and fails if any exceeds a threshold. It can also diff against a saved baseline through benchstat, which must be on your PATH (see part 2):

1

benchgate -pkg ./... -count 10 -cv-threshold 5.0

-pkg (default ./...) and -bench (default .) pick what to run, -count (default 10) says how many times, and -cv-threshold (default 5.0, in percent) sets the bar. -baseline takes a saved file to diff against, -save writes one, and -json prints machine-readable output. A gate at 5% catches environments too noisy for a reliable A/B before we waste time interpreting numbers. Part 4 shows what SMT and frequency scaling do to CV.

To capture a baseline on the current branch and compare after a change, we run it twice:

1
2
3

benchgate -pkg ./... -count 10 -save old.txt
# make your change
benchgate -pkg ./... -count 10 -baseline old.txt

The second run calls benchstat old.txt <new-output> for us and prints the comparison. The exit code comes from the CV check alone, so a +40% delta on a quiet sample still exits 0; gating on the delta is a step you write. A stable sample still says nothing about the machine that produced it, which is what the third tool is for.

benchenv

benchenv helps with question 3. It diagnoses the measurement environment: SMT state, CPU frequency governor, Turbo Boost, system load average, and which of perflock, benchstat and benchdiff are installed. It works across platforms and degrades gracefully on macOS, where sysfs controls are unavailable. Its only flag is -json.

1

benchenv

Here is one run, on one machine: an Apple M4 Max (darwin/arm64, Go 1.27.1) with benchenv from v0.1.0, on 2026-10-02. The load-average warning depends on what else the machine was doing, so yours will differ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

benchenv: benchmarking environment diagnosis (darwin/arm64, 16 CPUs)

  [unavailable]   SMT control — macOS does not expose SMT control via sysfs — use a Linux machine or bare-metal CI runner for publication-quality numbers
  [unavailable]   CPU frequency governor — macOS does not expose a CPU frequency governor — use a Linux machine or bare-metal CI runner for publication-quality numbers
  [unavailable]   Turbo Boost — macOS does not expose Turbo Boost control from user space — use a Linux machine or bare-metal CI runner for publication-quality numbers
  [warn]          load average — close background applications before benchmarking
  [unavailable]   thermal pressure — macOS thermal state is not accessible from user space — watch for CPU throttling on sustained benchmark runs
  [warn]          perflock not installed — go install github.com/aclements/perflock@latest
  [ok]            benchstat installed — benchstat found on PATH
  [warn]          benchdiff not installed — go install github.com/willabides/benchdiff/cmd/benchdiff@latest
  [ok]            GOMAXPROCS / NumCPU — NumCPU=16 GOMAXPROCS=16

Summary: 2 ok, 3 warn, 4 unavailable. Fix warnings before trusting benchmark numbers.

Every [warn] line is a noise source or a missing tool, and the [unavailable] lines are macOS declining to say. The install hints print @latest; pin perflock and benchdiff the way part 3 does. We run benchenv once at the start of a benchmarking session, fix the warnings, then run benchgate, then compare with benchstat.

The minimum viable discipline

If we keep one thing from this series, it should be this: run benchmarks ten times, not once. The whole loop needs only the Go toolchain and benchstat:

1
2
3
4
5
6
7
8

# Baseline on the current branch
go test -bench=. -benchmem -count=10 ./... | tee old.txt

# After your change
go test -bench=. -benchmem -count=10 ./... | tee new.txt

# Compare
benchstat old.txt new.txt

benchstat then tells us whether the difference clears the noise. That is the floor. For anything you report, use -count=20 (plus -benchtime=2s on a noisy Mac, part 3), and for CI baselines consider a fixed count (-benchtime=Nx, part 2) over part 4’s time-based 2s and 5s. Around that loop go all three tools: honestbench -r ./... before reading any numbers, benchenv on any new machine or CI runner, and benchgate -cv-threshold 5.0 as a gate that fails early when the environment is too noisy for a reliable signal.

Three tools, under an hour to wire up, for any Go project. Time to check them against the cable from the start.

Versions, links and commands checked on 2 October 2026.

Where to go from here

Back to the loose cable. Run #4891 through the questions. Question 1 passes: the loop sizes a real protobuf message. Question 2 passes too, with a CV under 0.1%, and that is the trap, because a stable sample is not the same as a right one. Question 3 is the one that failed. The CI delta said slower, the A/B on one machine said faster, and the sign flipped when main moved. The A/B on one machine took the machine out of the story, with CI as the smoke alarm rather than the judge.

That is the series. The talk page has the slides and links, the CLIs are in benchlab, and the demo results are in gopherconuk-26.

Go find your loose cable. Preferably before anyone calls a press conference.

Start with a pair that misled me. On dd-trace-go #4891, the benchmark bot’s comment now says BenchmarkOTLPProtoSize/1span is 7.6% to 8.0% faster than its baseline. Eleven days after #4891 was opened, on #4926, the bot said the same sub-benchmark was 8.1% to 8.5% slower. Both comments are correct arithmetic over two sets of runs, and the baselines differ: 05d9b8a on the first, 1e830f6 on the second. Same benchmark, opposite verdicts. 🔀

This is a companion to the Why Your Go Benchmarks Are Lying series, between part 4 and part 5. Part 5 tells the #4891 story in full; here we ask what it looks like over time. Disclosure: I work at Datadog, which maintains dd-trace-go.

A pair carries an assumption

The bot says what it does in its own comment: “This is an A/B test comparing a candidate commit’s performance against that of a baseline commit.” For a question about one change, that is fair, and A PR gate that actually fails builds one.

The pair quietly assumes the baseline is a fixed, neutral reference. It isn’t. main moves, and the binary’s layout and the machine’s mood move with it. Change the baseline and the same benchmark gets a different verdict, as above; on #4891 alone the sign changed between bot updates (part 5). A pair can’t show that, because it has one point on each side.

What does the same benchmark look like when we keep every night instead of comparing two? Let’s make one up.

Sixty nights of one benchmark

Everything in this section is synthetic: a Go program generated it, and no real benchmark did. The benchmark is a pretend 1000 ns/op function measured once a night, with 0.5% Gaussian noise. Three times, at nights 20, 35 and 50, a “PR” lands and adds 1.5%. On night 42 a noisy neighbour adds 8% for one night only. We’ll read it two ways: a 5% gate comparing each night with the one before, standing in for a PR check against one baseline run, and a change-point detector we’ll write after that.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

package main

import (
	"fmt"
	"math"
	"math/rand/v2"
	"slices"
)

const (
	nights    = 60
	minLen    = 5
	threshold = 6.0
)

func main() {
	// Synthetic: 1000 ns/op, 0.5% noise, three +1.5% steps, one noisy night.
	rng := rand.New(rand.NewPCG(1, 2))
	series := make([]float64, nights)
	for i := range series {
		level := 1000.0
		for _, step := range []int{20, 35, 50} {
			if i >= step {
				level *= 1.015
			}
		}
		series[i] = level * (1 + 0.005*rng.NormFloat64())
	}
	series[42] *= 1.08

	for i := 1; i < nights; i++ {
		if d := series[i]/series[i-1] - 1; math.Abs(d) > 0.05 {
			fmt.Printf("night %2d vs night %2d: %+.1f%%  <- a 5%% gate fires\n", i, i-1, 100*d)
		}
	}

	fmt.Println("change points (night where a new level starts):")
	for _, cp := range split(series, 0, nights) {
		fmt.Printf("  night %d\n", cp)
	}
}

The first loop is the per-PR model. Here is the first half of the output, from Go 1.27.1:

1
2

night 42 vs night 41: +9.3%  <- a 5% gate fires
night 43 vs night 42: -8.3%  <- a 5% gate fires

The gate fires twice, both times on the noisy neighbour: night 42 looks like a regression, night 43 like a recovery. It never fires on the three real steps, because 1.5% is well under its threshold, yet together they made the benchmark about 4.6% slower. The gate blames the innocent night and lets the guilty ones through. Part 4 said a change costing 1% per PR won’t trip a per-PR gate. Here are the numbers.

Reading the series

Change-point detection scans the whole history for places where the results shift, and stays quiet about everything else. MongoDB engineers moved to an E-Divisive means detector after threshold-based detection, and report that it “dramatically dropped our false positive rate”.

Our toy reader is much cruder and does not use E-Divisive. It does binary segmentation: find the night with the largest median shift, and if that beats a threshold, cut there and repeat on both halves. Here is the rest of the program:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// split does binary segmentation: cut at the strongest shift, recurse on both halves.
func split(x []float64, lo, hi int) []int {
	best, bestT := -1, 0.0
	for k := lo + minLen; k <= hi-minLen; k++ {
		if t := math.Abs(shift(x[lo:k], x[k:hi])); t > bestT {
			best, bestT = k, t
		}
	}
	if bestT < threshold {
		return nil
	}
	return append(append(split(x, lo, best), best), split(x, best, hi)...)
}

// shift is the difference of medians in units of its standard error, with
// the noise scale taken from the median absolute deviation so that a single
// spike cannot inflate it.
func shift(a, b []float64) float64 {
	ma, mb := median(a), median(b)
	dev := make([]float64, 0, len(a)+len(b))
	for _, e := range a {
		dev = append(dev, math.Abs(e-ma))
	}
	for _, e := range b {
		dev = append(dev, math.Abs(e-mb))
	}
	sigma := 1.4826 * median(dev)
	return (mb - ma) / (sigma * math.Sqrt(1/float64(len(a))+1/float64(len(b))))
}

func median(x []float64) float64 {
	s := slices.Sorted(slices.Values(x))
	return (s[(len(s)-1)/2] + s[len(s)/2]) / 2
}

split tries every cut at least minLen nights from either end and keeps the strongest. The threshold of 6 is a knob I picked by eye for this data, and a real deployment has to set its own. The last part of the output:

1
2
3
4

change points (night where a new level starts):
  night 21
  night 33
  night 49

It finds three level changes and ignores the night-42 spike. The planted steps were at 20, 35 and 50, so it lands within two nights of each. On a real series the output is a list of candidate nights, and the next job is git log between them. Part 4 lists tools that ship change-point detection.

How the Go project reads its own

The Go wiki’s PerformanceMonitoring page states the principle plainly: “We never report performance numbers in isolation, and only relative to some baseline”, because “comparing performance data taken far apart in time, even on the same hardware, can result in a lot of noise that goes unaccounted for”. Before merge, SlowBot perf_vs_parent and perf_vs_tip run a pair on purpose. After merge, the performance dashboard gives “continuous monitoring of benchmark performance for every commit”, and because the tip-of-tree baseline “is always the latest overall release of Go”, “on every minor release of Go, the baseline shifts”. Read that next to our opening pair: the sign on #4891 flipped when main moved, and Go’s dashboard puts baseline changes where you can see them.

A ledger for alarms that weren’t

Bot comments will keep arriving, and some will be old friends. A mute regex silences a benchmark and forgets why. I’d add a ledger entry per dismissed alarm, in the repository, next to the benchmark. This one is illustrative, written from the public bot comments and part 5:

1
2
3
4
5

BenchmarkOTLPProtoSize (all four sub-benchmarks)
  seen:    dd-trace-go #4891 (12 Jun), #4926 (23 Jun)
  symptom: 6-9% in one direction; sign flipped when main moved
  cause:   code layout, suspected, not proven (see part 5)
  check:   same-machine A/B of base vs head before anyone edits code

seen turns one surprise into a count, cause says how sure we are, and check names the decisive test, so the next person doesn’t improvise one.

Back to the one benchmark

The pair told us −8% on one PR and +8% on another. Our sixty nights told us which night was a neighbour and which were the code. Keep the nightly series from part 4, use pairs for “did this change do it”, and write down the alarms that turned out to be nothing.

Versions, links and commands checked on 2 October 2026.

Try it

The two Go blocks above are one program: paste the second under the first in a main.go, run go mod init cp and go run .. I ran that with Go 1.27.1 and got the output shown. Change the seed or threshold and watch the detector miss a step or invent one. Then ask what the baseline was in your own bot’s last three comments.

A benchmark is a story told over time. Stop reading only the last sentence. 📈

We’ll build that job together. We’ll write one deliberately slow commit, let the job judge it, revert the commit, and let the job judge the revert. Two runs, two verdicts, both on a GitHub-hosted runner:

1
2

Ints-4   2.579µ ± 1%   7.700µ ± 1%  +198.57% (p=0.000 n=10)
Ints-4   2.579µ ± 2%   2.576µ ± 1%  ~ (p=0.782 n=10)

The first line is the slow commit, and the second is its revert. This is a companion to the Why Your Go Benchmarks Are Lying series, between part 4 and part 5, the written companion of my GopherCon UK talk. GitHub deletes Actions logs after a retention period, so this post carries the key log lines itself. The benchgate tool is kakkoyun/benchlab at v0.1.0, and the workflow and both runs live on PR #5, a closed demo.

The slow commit

The patient is a small benchmark, BenchmarkInts, which sums a slice of integers. The slow commit routes every addition through a helper marked //go:noinline, so each total = add(total, x) becomes a real function call. I made it slow on purpose, and by a lot: roughly three times.

A PR against main has no benchmark on its base side, so the base binary couldn’t even build. The demo PR targets a branch that already holds the workflow and the benchmark, and the diff under test is the slow commit alone. First run, slow commit. Second run, I push the revert to the same PR.

Now the judge. Where should it run, and against what?

Why base and head share a job

Part 4 keeps the baseline from main in a cache, which suits a pinned runner. On a hosted runner, a cached baseline was measured on whatever VM ran the earlier job, and nothing promises that VM is the one judging your PR. This gate builds both binaries in one job and runs them on the same VM, taking turns: base, head, base, head. That interleaving (ABAB) spreads slow drift evenly over both sides. It does not remove noisy neighbours, and we’ll come back to that.

Here is the whole workflow. Read the comments, then we’ll walk through it step by step.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

# A PR gate that actually fails: companion to part 4 of the Go benchmarks series.
#
# One job builds the base and head benchmark binaries, runs them interleaved
# (ABAB), and fails on a regression that benchstat calls real. benchgate rules
# out an unstable sample first.
name: bench-gate

on:
  pull_request:

permissions:
  contents: read

env:
  BENCHSTAT_VERSION: v0.0.0-20260929162123-406019bb8b68
  BENCHGATE_VERSION: v0.1.0
  BENCH_PKG: ./internal/sum
  # ROUNDS is both the number of ABAB rounds and the sample count per side (n).
  ROUNDS: "10"
  BENCHTIME: 200ms
  # Block only when benchstat says the delta is real (p < 0.05) AND at least this large.
  MIN_DELTA_PCT: "5"
  # benchgate fails the sample when any benchmark's coefficient of variation is above this.
  CV_THRESHOLD: "10"

jobs:
  bench-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      # pull_request checks out the merge commit, so HEAD^1 is the base tip and
      # HEAD is what would land. Depth 2 is enough to reach both.
      - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
        with:
          fetch-depth: 2
          persist-credentials: false

      - uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6
        with:
          go-version: "1.27.1"
          cache: false

      - name: Install benchstat and benchgate
        run: |
          export GOBIN="$RUNNER_TEMP/bin"
          go install "golang.org/x/perf/cmd/benchstat@${BENCHSTAT_VERSION}"
          go install "github.com/kakkoyun/benchlab/cmd/benchgate@${BENCHGATE_VERSION}"

      - name: Build base and head benchmark binaries
        run: |
          if [ "$(git rev-list --parents -n 1 HEAD | wc -w)" -ne 3 ]; then
            echo "::error::HEAD is not a merge commit; expected the pull_request merge ref"
            exit 2
          fi
          mkdir -p "$RUNNER_TEMP/bench"
          git worktree add --detach "$RUNNER_TEMP/base" HEAD^1
          echo "base: $(git rev-parse --short HEAD^1)  head: $(git rev-parse --short HEAD)"
          go test -c -o "$RUNNER_TEMP/bench/head.test" "$BENCH_PKG"
          (cd "$RUNNER_TEMP/base" && go test -c -o "$RUNNER_TEMP/bench/base.test" "$BENCH_PKG")

      - name: Run base and head interleaved (ABAB)
        run: |
          pkgdir="${BENCH_PKG#./}"
          : > "$RUNNER_TEMP/bench/base.txt"
          : > "$RUNNER_TEMP/bench/head.txt"
          for round in $(seq 1 "$ROUNDS"); do
            for side in base head; do
              root="$RUNNER_TEMP/base"
              if [ "$side" = head ]; then root="$GITHUB_WORKSPACE"; fi
              echo "round $round: $side"
              (cd "$root/$pkgdir" && "$RUNNER_TEMP/bench/$side.test" \
                -test.run='^$' -test.bench=. -test.benchtime="$BENCHTIME" -test.count=1) \
                >> "$RUNNER_TEMP/bench/$side.txt"
            done
          done

      - name: Compare with benchstat
        run: |
          cd "$RUNNER_TEMP/bench"
          benchstat="$RUNNER_TEMP/bin/benchstat"
          "$benchstat" base.txt head.txt
          "$benchstat" -format csv base.txt head.txt > compare.csv
          # Rows are "name,base,CI,head,CI,delta,P". Gate on the sec/op table only; the delta
          # column holds "~" when benchstat sees noise, so only a signed percentage can match.
          awk -F, -v floor="$MIN_DELTA_PCT" '
            $1 == "" && NF > 1 { gate = ($2 == "sec/op"); next }
            gate && $1 != "geomean" && $4 != "" { compared++ }
            gate && $1 != "geomean" && $6 ~ /^\+[0-9.]+%$/ && substr($6, 2) + 0 >= floor {
              printf "%s %s (%s)\n", $1, $6, $7 > "regressions.txt"
            }
            END { print compared + 0 > "compared.count" }
          ' compare.csv
          touch regressions.txt

      - name: Check sample stability with benchgate
        run: |
          cd "$RUNNER_TEMP/bench"
          # Exit codes: 0 stable, 1 unstable (CV over threshold), 2 error. Record it
          # instead of failing here, so the verdict step can report everything at once.
          rc=0
          (cd "$GITHUB_WORKSPACE" && "$RUNNER_TEMP/bin/benchgate" \
            -pkg "$BENCH_PKG" -count "$ROUNDS" -benchtime "$BENCHTIME" \
            -cv-threshold "$CV_THRESHOLD") || rc=$?
          echo "$rc" > benchgate.rc

      - name: Gate verdict
        run: |
          cd "$RUNNER_TEMP/bench"
          compared="$(cat compared.count)"
          benchgate_rc="$(cat benchgate.rc)"
          # Exit codes: 0 pass, 1 regression, 2 tool error, 3 unstable sample.
          if [ "$compared" -eq 0 ]; then
            echo "::error title=Nothing compared::benchstat found no benchmark present in both base and head"
            exit 2
          fi
          status=0
          case "$benchgate_rc" in
            0) ;;
            1)
              echo "::error title=Unstable sample::benchgate exit 1: a benchmark's CV is above ${CV_THRESHOLD}%; the run cannot support a verdict"
              status=3
              ;;
            *)
              echo "::error title=benchgate failed::benchgate exit ${benchgate_rc}"
              exit 2
              ;;
          esac
          if [ -s regressions.txt ]; then
            while read -r line; do
              echo "::error title=Benchmark regression::${line}"
            done < regressions.txt
            status=1
          fi
          echo "verdict: exit ${status}"
          exit "$status"

Walking through it

The env block holds every number we might argue about later. ROUNDS is both the number of ABAB rounds and the sample count per side, so n=10. MIN_DELTA_PCT and CV_THRESHOLD are the two thresholds: a regression floor and a noise ceiling.

The checkout step uses fetch-depth: 2. On pull_request, GitHub checks out a merge commit, so HEAD^1 is the tip of the base branch and HEAD is what would land. Depth 2 reaches both. The build step refuses to continue if HEAD doesn’t have two parents, and it prints both short hashes, which is how the logs told me the comparison was the right one (base: 021b688 head: fb2ae9c in run 1). It then runs go test -c once per side, so we have two frozen test binaries and no rebuild between rounds.

The ABAB loop runs each binary ten times with -test.count=1, appending to base.txt and head.txt. The flags are a budget. Ten samples per side is the floor the benchstat docs recommend, and 200 ms is plenty for a benchmark that takes a few microseconds, so the whole job took about 35 seconds. Part 4’s nightly suite can afford -count=20 -benchtime=5s on controlled hardware, and this gate cannot afford that on every push.

The compare step is where the exit code gets made. It runs benchstat twice, once for the human-readable table and once with -format csv. The awk script reads rows shaped like name,base,CI,head,CI,delta,P, looks only at the sec/op table, and keeps a row only if its delta is a signed +N% with N at or above the floor. When benchstat sees noise, the delta column holds ~, so noise can never match. A faster result starts with a minus and is ignored too. The gate counts compared rows and writes the survivors to regressions.txt.

Then benchgate. At tag v0.1.0, -baseline only prints the benchstat output and the exit code never looks at it. The exit code answers a different question: is the sample stable enough to trust? It exits 1 when any benchmark’s coefficient of variation is above the threshold. That makes benchgate the noise gate and benchstat the regression gate. Also, benchgate runs its own go test on the head tree, one after another, so it measures how noisy the runner was, not the interleaved samples. The ± column in the benchstat table is the better noise indicator for the comparison itself.

Last, the verdict step folds everything into one exit code: 0 pass, 1 regression, 2 tool or setup error (nothing compared, benchgate failed, not a merge commit), 3 unstable sample. A regression wins over “unstable”, and each finding becomes an ::error annotation on the PR. Enough reading. Let’s watch it judge.

Run 1: the slow commit

Run 1 failed, and it failed in the right step. The benchstat step printed this (the log also names the CPU, an AMD EPYC 7763):

1
2
3

       │  base.txt   │               head.txt               │
       │   sec/op    │   sec/op     vs base                 │
Ints-4   2.579µ ± 1%   7.700µ ± 1%  +198.57% (p=0.000 n=10)

Ints went from 2.579 µs to 7.700 µs, and the ± columns are small. Then the stability check:

1
2
3
4
5

benchgate: go test -bench=. -count=10 -benchtime=200ms ./internal/sum

  BenchmarkInts                                 mean=  7816.3 ns/op  cv=  3.1%  ✓

VERDICT: PASS — all 1 benchmarks within CV threshold 10.0%

A CV of 3.1% is under the 10% ceiling, so the sample was trusted. The verdict step did the rest:

1
2
3

##[error]Ints-4 +198.57% (p=0.000 n=10)
verdict: exit 1
##[error]Process completed with exit code 1.

The ##[error] line is the annotation from the awk output, and the last line is the red check. Was it a fluke? Let’s judge the revert.

Run 2: the revert

The revert makes the head tree identical to the base, so run 2 is a null test: there is nothing to find, and the gate should find nothing.

1
2
3

       │  base.txt   │           head.txt            │
       │   sec/op    │   sec/op     vs base          │
Ints-4   2.579µ ± 2%   2.576µ ± 1%  ~ (p=0.782 n=10)

~ means benchstat could not tell the sides apart, and benchgate reported cv= 3.4% ✓. The verdict step closed with:

1

verdict: exit 0

The gate went red for the slow commit and green for the revert, on the same kind of hosted runner, a minute apart.

What this does not prove

Part 4’s argument stands: ubuntu-latest is a shared machine, and I can’t turn off SMT or frequency scaling from inside it. These runs don’t show that the runner is quiet. They show that a 3x change is far outside whatever noise was there. I’d call a hosted gate weaker than one on bare metal, not worthless. It catches the obvious regressions, and it can’t promise to catch a 5–10% one. With n=10 on a shared runner, a change that small can flap: pass once, fail the next time. A/B is the wrong model for CI looks at what one pair per PR misses.

The CV numbers give a partial answer to a question readers asked, whether -cv-threshold 5 always fails on hosted runners. In these two runs it would not have (3.1% and 3.4%). I set 10, and two runs are a sample of two. If a noisy run does trip the ceiling, the job fails closed with exit 3, a red check that is not a regression. I saw that path only on my laptop, where noise tripped it. I never saw exit 2 on GitHub.

The runs are also a single benchmark on one machine type, captured on 4 October 2026. If you adopt this, calibrate BENCHTIME, ROUNDS and the two thresholds on your own benchmarks first. The gate is only worth what that calibration says.

Versions, links and commands checked on 2 October 2026; the CI runs shown were captured on 4 October 2026.

Try it

Copy the workflow into a repository with a benchmark package, point BENCH_PKG at it, and push a branch holding the workflow and the benchmark. Open a PR against that branch with a deliberately slow commit, watch the check go red, then push a revert and watch it go green. That is exactly what I did in the demo, and actionlint had no complaints about the file.

A red check only means something if we trust what it compares. The slow commit was easy to catch: at about 3x, nothing about the hardware could hide it. The dangerous deltas are small ones that look real, and a green or red check that is really measuring code layout. Part 5 opens with a CI regression that turned out to be a speedup. 🎟️

This is part 5 of 6 in the How to Instrument Go Without Changing a Single Line of Code series, the written companion to my GopherCon UK 2026 talk. Part 4 left us in the kernel with a profiler. Now we step outside the three intervention points and knock on the runtime’s door four times (the FOSDEM 2026 post covers the last knock at more length).

Disclosure: I work at Datadog, which maintains Orchestrion, and I’m one of otelc’s maintainers. The poc_flight_recorder and poc_usdt forks below are my own experiments.

Flight recording shipped in Go 1.25

Let’s start with the door that opened. golang/go#63185, the flight recorder proposal, is closed. It shipped.

Go 1.25 includes a circular buffer tracer in runtime/trace, in the spirit of Java Flight Recorder (which has been around for a while, so allow Go a small victory lap). It records Go’s execution trace, not OpenTelemetry spans: goroutine, syscall and GC events, read with go tool trace, not Jaeger. Instead of streaming trace data to a file, it keeps the last N seconds in memory and snapshots on demand. Here’s a sketch of asking for it, imports left out. The API is a config struct, not setter methods:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{
	MinAge:   2 * time.Second,
	MaxBytes: 64 * 1024 * 1024,
})

if err := fr.Start(); err != nil {
	log.Fatal(err)
}

// Later, on a slow request or error:
var buf bytes.Buffer
if _, err := fr.WriteTo(&buf); err != nil {
	log.Println("flight recorder:", err)
}
// buf now holds at least the last 2 seconds of trace data
// (MinAge is a lower bound; MaxBytes wins if they conflict)

Start begins recording and WriteTo copies the buffer out when something goes wrong. Notice that every line of it lives in our service’s code. Hold that thought.

OBI (OpenTelemetry eBPF Instrumentation) tells you something is slow; flight recording tells you what the runtime was doing at the time. “On demand” has a catch: the recorder traces continuously, so you pay for tracing and for the in-memory buffer all the time (the Go blog post puts a busy service at around 10 MB/s of trace data, and suggests a MinAge of about 2x the event window). Only writing and keeping the data is on demand: wire WriteTo to your error handler or a debug HTTP endpoint (net/http/pprof-style; only one goroutine may call WriteTo at a time, and a concurrent call returns an error), and you keep a trace only when something goes wrong.

I pushed on this API too. My poc_flight_recorder fork adds an EventFilter bitmask (FilterHTTP, FilterSQL, FilterNet and friends) with SetEventFilter, SetSampleRate, W3C TraceContext parsing, and HTTPServerRequest and HTTPClientRequest spans to runtime/trace, aiming at always-on distributed tracing. That’s an aspiration. I don’t expect the Go team to extend the flight recorder this way. The FOSDEM post has the longer write-up.

The runtime handed us a way to watch itself. The libraries built on it are another story, and the HTTP client is where it hurts.

The httptrace gap that breaks HTTP/2 spans

golang/go#75654 is an open proposal for a hook called GotResponseEnd on httptrace.ClientTrace. No hook reliably tells the client, on every protocol, that a response body has been fully consumed. PutIdleConn is the workaround OTel Go uses, but the standard library documents that hook as not used for HTTP/2, so on HTTP/2 OTel Go’s http.receive client span is never ended. That’s tracked in OpenTelemetry Go contrib issue #4876, which was still open in October 2026. In the sources I read, otelc’s net/http client hook (v1.1.0) doesn’t reference httptrace, and OBI watches net/http with uprobes instead, so neither should hit this, though I haven’t tested HTTP/2 spans from either.

As filed, the fix is simple. This is the proposal’s pseudo-code, not something httptrace has today:

1
2
3
4

// Add to httptrace.ClientTrace:
GotResponseEnd func(err error)
// Fires exactly once per request when resp.Body.Read returns io.EOF,
// a non-nil error, or the body is closed early.

One callback, fired once. The shape is still being argued over, with commenters asking about responses with no body and bodies the application never reads, and there’s no acceptance decision yet.

That’s one hook a library can’t get. The next ask is about how the tools from part 3 get between us and the compiler.

Why compile-time instrumentation has rough edges

Orchestrion and otelc (OpenTelemetry Go compile-time instrumentation) intercept the compilation pipeline with -toolexec, as we saw in part 3. golang/go#69887 is Romain Marcadier’s proposal to improve it. Two of its gaps explain why these tools have sharper edges than they should.

First, a -toolexec tool can’t influence the build cache per package. The only hook point is intercepting the -V=full version probe, which works at the level of the complete build, so the issue describes excessively frequent cache invalidation for untransformed packages. otelc changes every tool’s version line, so instrumented builds never share cache entries with plain go build; its own cache keeps warm rebuilds fast. My guest post shows what the cache does to -toolexec tools, and what that costs. Second, the toolchain doesn’t tell -toolexec tools the full build arguments, so Orchestrion crawls its own process tree looking for a parent go build invocation and parses its arguments. That’s the workaround you write when the API won’t give you what you need.

Did the Go team agree? Two members replied, leaning different ways. Michael Matloob said dedicated support for source-rewriting tools would add a lot of complexity to the go command and that he doesn’t think the costs are worth it. Austin Clements called it a useful problem report and line-information handling a known weakness, and suggested go list plus -overlay as a way around the build-graph changes. As of October 2026 it’s still at Incoming on the proposals project. A neighbouring ask, a hook on goroutine start (golang/go#73798), was closed as not planned on the day of filing, with a first reply that pointed at toolexec rewrites.

Both asks are about things Go already has. Our last stop is something it doesn’t: probes.

USDT: an experiment toward tracepoints

USDT (User Statically-Defined Tracing) probes are named sites compiled into a binary as NOP instructions, which do nothing. When a consumer such as bpftrace, perf, or a custom eBPF program attaches, the NOP is replaced with an INT3 interrupt and the kernel delivers the event, the same kernel path as uprobes.

What does that buy us over a bare uprobe? Not speed: a USDT site is a uprobe on a NOP, so it still costs a trap per hit, and my fork computes its arguments even with nothing attached, which I haven’t benchmarked. It buys less work for the tool, which reads site addresses and argument locations from the note instead of hunting for every RET (no uretprobes, which Go’s moving stacks can’t survive), and a probe name that can outlive refactors; What a uprobe costs, and what USDT buys has the sources and the numbers.

The Go runtime doesn’t ship USDT probes. The closest thing to a discussion is in the notes for the 2024-12-19 performance and diagnostics sync, recorded in golang/go#57175, where Felix Geisendörfer asked whether the Go team had considered USDT support. That’s a brainstorm, not a plan, and I’m not aware of a proposal filed since.

Asking nicely only goes so far, so I built one. My poc_usdt fork, pinned at commit c739f4e, adds USDT probes to net/http, database/sql, crypto/tls, and net via a go tool usdt subcommand. Once we build a binary from the fork, we can list its probes and generate a bpftrace script:

1
2
3
4
5
6
7

$ go tool usdt list ./myserver
PROVIDER   NAME                  ADDRESS     ARGUMENTS
net_http   server_request_start  0x63296c    8@%rsi -8@%r8 8@%rdx -8@%r9
net_http   server_request_end    0x631c5c    -4@%ecx

$ go tool usdt bpftrace ./myserver > trace.bt
$ sudo bpftrace trace.bt

That listing is abridged from the fork’s README.

On Linux, a binary built with the fork and the internal linker gets standard library instrumentation automatically, with no SDK import and no application code changes. (On darwin the probes compile to nothing, and external linking currently drops them.) The probes are standard .note.stapsdt ELF notes, so readelf -n can read them. bpftrace’s usdt: attach doesn’t find the probes in a static Go binary, so the script go tool usdt bpftrace generates uses uprobe: probes at the probe addresses instead.

ARM64 argument parsing in bpftrace also has issues with the probe argument notation the fork emits. Nothing here is proposed upstream.

The idea I’d argue for is bigger than USDT: generic tracepoints. Safe, named instrumentation points that the standard library and runtime expose on purpose, with stable arguments external tools can consume. USDT is one way to build them. It needs more work from me (the open items above) and the Go runtime team would have to accept it.

That’s all four doors. Only one of them opened. 🚪

Observing isn’t propagating

Tracepoints let a tool see a request. Distributed tracing also has to carry trace headers from the incoming connection to the outgoing ones. Without threading context.Context by hand, that needs some goroutine-associated storage, and it must not lose what the request’s context already carries: deadlines, cancellation, and values the application attached. Part 1 showed tools hacking a field into g for this. My fork’s WithSpan and SpanFromContext show the gap from the other side: they ride on context.Context, so the context still has to be threaded. Context across goroutines and connections walks that gap with a small program. I have no solution. This is the unsolved part.

What is already usable

Back to our question: what could the runtime hand us? Shipped: flight recording, on Go 1.25 or newer. Experimental, and mine: the tracing fork and the USDT probes. Unsolved: propagation without threading the context.

Remember the thought we held on to? The answer that shipped wants a few lines in our service, and the probes that need no application changes exist only in my fork. For a series about not changing a single line, that’s the gap in a sentence. It’s narrowing, just slowly.

Versions, links and commands checked on 2 October 2026.

Up next

Part 6, Making zero-touch Go observability agent-actionable, pulls the tools together into a decision rule and a skill an agent can run. The runtime may get there on its own schedule. Part 6 is for those of us with a service to debug today.

Nobody is imagining it. Both outcomes are correct given the data, and the data is just wrong. A shared CI runner is a multi-tenant machine, so our benchmark lands on a host that also runs other teams’ compile jobs, Docker builds and test suites. A real 10% regression can vanish into that noise, and a phantom one can appear from nowhere. The gate isn’t measuring our code. It’s measuring the lottery of what happens to be running next door. 🎰

I’ll call this the machine that changes its mind, and it will keep turning up as we go. Better statistics can’t compensate for an unstable environment, so we’ll change the environment.

This is part 4 of 5 in the Why Your Go Benchmarks Are Lying series, the written companion to the GopherCon UK 2026 talk. Part 3 kept one machine honest. Here we answer the series’ third question, whether the difference is large relative to the noise, once the machine isn’t ours. We’ll start with a number from FOSDEM, two CPU-bound tasks sharing a core with a coefficient of variation around 23%, to see what shared runners do to a measurement. Then we’ll split the gate into a fast PR check and a slow nightly suite, wire the nightly one into GitHub Actions on bare metal, and pick a tool for the history. Part 5 closes the series with a checklist.

The prerequisite is the FOSDEM post, which covers environment control in language-agnostic terms. Its experiments ran on a dedicated AWS m5.metal instance, with slides and results in igoragoli/fosdem-2026-software-performance, and we reuse those numbers here. What exactly makes a shared machine change its mind?

Why shared runners lie

Three mechanisms compound. Another tenant’s CPU-bound job contends with ours for memory bandwidth and last-level cache (LLC), which is shared across the cores of a die, and for execution units if it lands on a sibling hardware thread. Dynamic frequency scaling (DFS, or Turbo Boost on Intel) adjusts the CPU clock based on thermal load and power headroom, and a shared VM cannot disable it from inside the guest, so a benchmark that ran at 3.5 GHz in one CI run may run at 3.1 GHz the next because a neighbouring workload raised the thermal floor.

Then there’s the hypervisor. Even without tenant competition, it can pause our vCPU to serve another tenant’s interrupt, and from inside the guest that looks like our benchmark stalled.

The FOSDEM experiments put numbers on what controlling these variables buys. Start with SMT Simultaneous Multithreading, also marketed as Hyper-Threading on Intel. Two hardware threads share one physical core’s execution resources. . With it enabled, two CPU-bound tasks sharing a core (DFS off) showed a coefficient of variation (CV, the standard deviation as a share of the mean) around 23%. With it disabled, the same two tasks ran on separate cores:

Turning SMT off cut CV by roughly a hundredfold, and the tasks ran faster too, because they stopped fighting over one core’s execution units. Next the clock. With SMT already disabled on the same machine, the slides show DFS on and off for a single task:

The DFS-off row is the SMT-disabled configuration again (SMT and DFS off), hence the same ~738 ms. Turning DFS off cut CV roughly tenfold. Absolute runtime goes up, because the CPU runs at base frequency instead of boosting, but the measurement is stable: slow and steady beats fast and moody.

How do those numbers compare with a setup we might already have? Part 3 showed that a pinned container on a macOS developer machine bottoms out near 5.25% CV. A 5% CV puts a 5% real regression inside the noise floor of any single run, and resolving it takes more samples than a PR gate can afford. On bare metal with SMT and DFS disabled, the FOSDEM runs drop below 0.25%, well below any regression worth caring about.

The machine can be tamed, then, but only if we own the whole box. What we run on it comes next, because a PR gate and a nightly history want opposite things.

Two patterns that actually work

A PR gate needs to finish in minutes, while thorough statistics need many samples over a stable environment. A nightly suite can run for an hour on controlled hardware, but most teams can’t require that hardware on every PR. No single job serves both, so we build two.

Pattern A: PR gate

The gate runs on every pull request and catches unambiguous regressions before they merge. It targets five minutes or less on a pinned runner, running a curated subset: benchmarks that have regressed before, or that the PR directly exercises. I’d start with -count=6 -benchtime=2s, the lowest count that works, because benchstat needs six samples to print a 95% confidence interval at all, and its docs recommend at least ten. The baseline comes from main, kept in a GitHub Actions cache rather than an artifact, which by default is only visible to the run that produced it. The gate blocks only when benchstat reports a delta instead of ~ (p < 0.05) and that delta is larger than a floor we choose.

Two caveats. benchstat exits 0 whatever it finds, so a step must read its vs base column and fail the job on a significant delta. A cached baseline was also measured earlier, maybe elsewhere: data taken far apart in time, even on the same hardware, can result in a lot of noise, says the Go wiki, so building base and head in one job and interleaving them is stronger; A PR gate that actually fails shows that job catching a slow commit. On GitHub-hosted runners the gate is weaker, not worthless: a smoke alarm, as part 5 says, not the judge.

Pattern B: nightly full suite

The nightly suite runs on a schedule against dedicated bare-metal hardware, and it watches for the slow regressions that accumulate across many PRs. It can afford -count=20 -benchtime=5s, with the full environment controls applied at the runner level. Results go into our own time-series store (Bencher or Nyrkiö, for example, from the survey below), compared against a rolling 30-day window, with change-point detection flagging regressions. Alerts go to Slack or email, and nightly data never blocks a PR.

A change that costs 1% per PR, landing five times over a sprint, won’t trip a per-PR gate, but change-point detection on the time series can; A/B is the wrong model for CI builds a small detector to show it. The nightly suite is only as trustworthy as the box under it, though, so let’s see what that box needs.

Bare metal versus VM

The controls are the ones from part 3: SMT off, the performance governor with turbo off, and taskset. Together they remove the three largest sources of benchmark variance, but only where they are physically real. Inside a shared VM the hypervisor manages SMT, so the guest cannot disable it, and frequency controls in a guest, where they exist at all, do not change the physical CPU clock. And echo off > /sys/devices/system/cpu/smt/control may succeed (it offlines the guest’s own sibling vCPUs) without changing how the hypervisor schedules them. A control that succeeds and changes nothing is the worst kind: it makes us feel better and measures the same.

Where do we find a machine with real controls? AWS EC2 bare-metal instances (m5.metal, m7i.metal-24xl) are dedicated physical hosts with no hypervisor layer, at roughly $4-5/hour on demand for Linux in us-east-1. Hetzner rents dedicated servers with full root access. Past an hour or so of nightly runtime, one persistent bare-metal machine is likely to beat cloud instances on cost: an m5.metal running two hours nightly costs around $280/month, so a dedicated server, rented or owned, can pay for itself within months.

Suppose we have the box. How do we point a workflow at it, and only at it?

GitHub Actions wiring

The nightly workflow below runs on a self-hosted bare-metal runner and applies the controls before the benchmarks. The critical detail is runs-on: [self-hosted, linux, x64, bare-metal]. GitHub adds self-hosted, linux and x64 when we register the runner; bare-metal is a custom label we add, and the combination routes only to those runners. Writing ubuntu-latest here would silently route to shared VMs and defeat the entire exercise, which is a lot of YAML to end up exactly where we started.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# .github/workflows/benchmark-nightly.yml
name: Nightly Benchmarks
on:
  schedule:
    - cron: '0 2 * * *'  # 2am UTC

jobs:
  benchmark:
    runs-on: [self-hosted, linux, x64, bare-metal]
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0

      - uses: actions/setup-go@v7
        with:
          go-version-file: go.mod

      - name: Restore baseline cache
        uses: actions/cache/restore@v6
        with:
          path: bench-baseline.txt
          key: bench-baseline-${{ github.run_id }}
          restore-keys: |
            bench-baseline-

      - name: Apply environment controls
        run: |
          echo off | sudo tee /sys/devices/system/cpu/smt/control >/dev/null
          echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null
          echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo >/dev/null

      - name: Run benchmarks
        shell: bash  # runs with pipefail, so a failing go test fails the step
        run: |
          taskset -c 0 go test \
            -bench=. -benchmem \
            -count=20 -benchtime=5s \
            ./... | tee bench-new.txt

      - name: Compare with baseline
        run: |
          if [ -f bench-baseline.txt ]; then
            go run golang.org/x/perf/cmd/benchstat@v0.0.0-20260929162123-406019bb8b68 \
              bench-baseline.txt bench-new.txt
          else
            echo "No cached baseline yet. This run establishes one."
          fi

      - name: Promote new baseline
        run: cp bench-new.txt bench-baseline.txt

      - name: Save baseline cache
        if: always()
        uses: actions/cache/save@v6
        with:
          path: bench-baseline.txt
          key: bench-baseline-${{ github.run_id }}

Top to bottom, it restores last night’s baseline, applies the sysfs controls, pins the run to CPU 0, compares with benchstat, then promotes tonight’s results as the new baseline. taskset -c 0 makes GOMAXPROCS 1, so it suits single-threaded benchmarks; parallel ones need a wider CPU set. It never undoes the controls, fine for a box that only runs benchmarks; on a shared machine, add an if: always() step that writes on to the SMT control. It only prints the comparison. The time-series store and the alerting come from the tools in the survey below.

The PR gate uses the same structure with a dedicated (but lighter) pinned runner in runs-on, a lower count and benchtime to fit the time budget, and only the curated benchmark subset. One warning before copying any of this: GitHub advises that self-hosted runners should almost never be used for public repositories, because anyone who can open a pull request can run code on them, and this workflow uses sudo. On a private repository, put the gate behind an environment with required reviewers.

The workflow leans on benchstat and promises a time series it doesn’t keep. Where do those come from? The Go team ships the first one.

The golang.org/x/perf toolchain

golang.org/x/perf is the Go team’s official performance measurement toolchain, and its tools read the standard go test -bench output with no adapters or code changes. benchstat is the one we’ve been leaning on. It computes the median, a non-parametric confidence interval for the median, and a Mann-Whitney U-test across two result sets. It prints ~ when the difference is indistinguishable from noise, and otherwise a delta with its p-value and sample count. That makes it the primary tool for A/B comparisons, locally and in CI:

1
2

go install golang.org/x/perf/cmd/benchstat@v0.0.0-20260929162123-406019bb8b68
benchstat bench-baseline.txt bench-new.txt

The version pin matches the one in the workflow, and the old results go first, so the delta reads as the new run against the baseline. The benchsave sibling uploads results to a perfdata server, but its default one is the Go project’s, behind an interactive Google sign-in that publicly records your email address. Unattended CI needs its own store.

benchstat is a point-in-time comparison. It keeps no time series and alerts on no drift. The tools from part 3 carry over; the controls need bare metal. What CI adds is the historical baseline, the automated gate and the cross-PR comparison that no human maintains by hand. The storage layer is what’s left to choose, and the field is crowded.

Tool survey and a recommendation

The tools that tackle continuous Go benchmarking differ in statistical model, hosting model, and how much infrastructure they need. Here is the field as of the talk (12 August 2026):

CodSpeed is not in the table: its Go integration is in early development and supports only wall-time measurement, not the CPU simulation it offers elsewhere. One of these deserves a warning label. cob has a subtle footgun: it internally runs git reset, so all changes must be committed before running it, which rules it out for pre-commit hooks or any CI step with a dirty working tree.

Which would I pick? That depends on who “we” are.

Small OSS project on GitHub. Use benchmark-action/github-action-benchmark. It needs no external accounts and stores results on GitHub Pages. The default 200% threshold is too loose to be useful. The alert-threshold input is a ratio of current to previous, so tighten it to 110%-120% (a 10-20% slowdown) and add a benchstat step alongside it, because the action has no concept of significance. With -count=10 the benchstat step gives honest numbers in the log, but the action treats each sample line as a separate result and does not average them.

Team with a dedicated CI runner. Use bencher.dev self-hosted, on a noise-controlled machine. Bencher’s pricing page says you can run the self-hosted binary without a Bencher Plus license, at the feature level of the Cloud Free plan, so check it against your private-project needs. Configure the t_test threshold model; unlike z_score, its docs do not ask for 30 historical samples. Wire --error-on-alert to block PRs. The go_bench adapter needs no changes to your benchmark code, though it reports only the mean.

Large org wanting change-point detection. Use Nyrkiö with Apache Otava as the detection backend. The e-divisive approach, as applied at MongoDB finds persistent shifts across the series instead of comparing each run with a fixed baseline, and adapts to each benchmark’s noise through its significance test. Apache Otava entered incubation in November 2024. Nyrkiö 2.0.0 (February 2026) added Nyrkiö Runners for GitHub, which its release notes describe as configured for stable, repeatable benchmark results. For fully on-premises operation, Apache Otava can run against a self-hosted time-series store.

In every scenario, keep benchstat for local comparisons and PR descriptions. Of the tools here, only benchstat gives a confidence interval and a p-value for a direct comparison of two sets of samples, so trust it when the question is whether a change is real. Which brings us back to the gate we started with.

Back to the machine that changes its mind

Remember the gate that failed a harmless PR and waved a hot-path rewrite through? It faithfully reported a machine that kept changing its mind. With the setup we just built, the nightly suite runs on bare metal with SMT off, the performance governor set and turbo off, and the PR gate blocks only when benchstat says the delta is real and larger than a floor we chose. The lottery next door is out of the draw. 🎟️

Versions, links and commands checked on 2 October 2026.

Up next

Part 5, Three Questions Before You Trust a Benchmark, closes the series: a CI regression that turned out to be a speedup, and a three-question checklist backed by three small CLIs. Even on perfect hardware, a benchmark can still lie to us. Part 5 shows how.

This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 4 and Part 5. The series accompanies my GopherCon UK talk. Part 1 showed tools hacking a field into g. Here we look at what that field holds, what it can’t, and who does the rest.

Disclosure: I work at Datadog and I’m one of the maintainers of otelc, the OpenTelemetry compile-time instrumentation tool.

What falls out of a goroutine

Our request arrives with a context: a trace ID stored as a value, a one-second deadline, and, because the client hung up, a cancellation. The program below hands that context to one goroutine and context.Background() to another. The second is what a function ends up with when its signature has no ctx parameter and nothing else to go on. Each goroutine prints what it can learn about the request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

package main

import (
	"context"
	"fmt"
	"time"
)

type traceKey struct{}

// inGoroutine runs the report in a new goroutine and waits for it.
func inGoroutine(name string, ctx context.Context) {
	done := make(chan struct{})
	go func() {
		defer close(done)
		_, hasDeadline := ctx.Deadline()
		fmt.Printf("%-12s value=%v deadline=%v err=%v\n",
			name, ctx.Value(traceKey{}), hasDeadline, ctx.Err())
	}()
	<-done
}

func main() {
	ctx := context.WithValue(context.Background(), traceKey{}, "trace-abc123")
	ctx, cancel := context.WithTimeout(ctx, time.Second)
	cancel() // the client hung up

	inGoroutine("request ctx:", ctx)
	inGoroutine("fresh ctx:", context.Background())
}

Go 1.27.1 prints this (one run, one machine, though the output doesn’t depend on timing):

1
2

request ctx: value=trace-abc123 deadline=true err=context canceled
fresh ctx:   value=<nil> deadline=false err=<nil>

The first goroutine knows the lot because we handed it the context. The second knows nothing: no value, no deadline, and ctx.Err() is nil even though the client left. Threading ctx through by hand fixes it, and that’s the code change this series is trying to avoid. A tool that can’t ask us to do that needs somewhere else to keep the request.

Job one: copy a slot when a goroutine starts

Orchestrion (Datadog’s -toolexec tool) keeps that somewhere else in a __dd_gls_v2 field on runtime.g, which part 1 walks through. Its aspect adds the field and clears it in goexit1, and neither rule touches the code that starts a goroutine, so the slot does not follow go. The TODO part 1 quotes from the context stack is still there at v2.9.1. I read both files at v2.0.0 and v2.9.1, and they match.

otelc copies on go. The runtime rule at v1.1.0 (the instrumentation/runtime/ directory is identical at v1.0.1, the series pin) injects a deferred function into newproc1, the runtime function that builds a new goroutine, to copy the two fields it adds to g, otel_trace_context and otel_baggage_container. Re-indented the way gofmt would print it, the injected code is:

1
2
3
4

defer func() {
	_unnamedRetVal0.otel_trace_context = propagateOtelContext(callergp.otel_trace_context)
	_unnamedRetVal0.otel_baggage_container = propagateOtelContext(callergp.otel_baggage_container)
}()

_unnamedRetVal0 is the new goroutine and callergp is its creator. propagateOtelContext returns nil for nil, calls Clone() when the value has one, and otherwise hands over the same reference. The one Clone in the SDK instrumentation is on the span stack: it keeps the innermost live span and drops the rest, so the child starts with a stack of one. The baggage slot is copied too, but in the v1.1.0 tree the only write I found is a reset to nil, so I can’t tell you what it carries today.

Reading it back, a hook on trace.SpanFromContext swaps in the goroutine’s span when the context has no valid one. What crosses a go statement is the parent’s current span, and a span is not the request.

Job two: keep the request’s own context

Our program already showed the difference. A context.Context carries values, a deadline and a cancellation signal, and the goroutine with the fresh context lost all three. Give that goroutine an otelc-style slot and, going by the hook’s code, SpanFromContext(context.Background()) returns the right parent span, so the trace tree stays whole. ctx.Err() is still nil and the deadline is still missing. The trace repaired itself and the request didn’t.

My opinion: tools have been solving two problems as one. “Which span is my parent?” is a question a goroutine slot can answer. “Is my caller still waiting, and what did it attach to the request?” needs the context itself. The context package says values are for “request-scoped data that transits processes and APIs”, which is precisely the data a span-only copy drops.

Let’s leave the process and follow the request onto the wire.

Job three: incoming connection to outgoing connection

At library level the job is spelled out in code. The otelhttp handler extracts trace headers from the incoming request into its context. The transport injects them into the outgoing request from r.Context(). Every link in that chain needs the context threaded. As I read it, that is the gap the SpanFromContext fallback from job one is built for.

OBI (OpenTelemetry eBPF Instrumentation) does it from outside the process, and here it helps to separate two modes. The v0.10.0 design doc describes kernel-side propagation, off by default (OTEL_EBPF_BPF_CONTEXT_PROPAGATION). With headers it injects a Traceparent header into plaintext HTTP, and with tcp it adds a TCP option to any TCP traffic. On the way in, it parses whichever one arrived. The security page (unversioned) lists CAP_NET_ADMIN for the trace-context-propagation programs.

For Go there is a second mode. The same design doc says a uprobe tries bpf_probe_write_user to write the header straight into Go’s HTTP buffer and leaves the header to the kernel-side injector if that fails. The security page lists CAP_SYS_ADMIN for “library-level Go trace-context propagation”.

How does OBI know which outgoing call belongs to which incoming request? It hooks the same function otelc does. Two uprobes on runtime.newproc1 record each new goroutine’s parent in a BPF map, keyed by goroutine address instead of a field in g. When an HTTP client call starts, OBI walks up that chain until it finds a goroutine serving a request.

OBI carries trace and span IDs in BPF maps keyed by goroutine. In the Go probes I read (v0.10.0) it uses no request-context values, so the deadline, cancellation and values stay where they were. For a tool that changes nothing that’s a fair deal, and it leaves that data behind. Which brings us to the one place neither route can reach: the runtime.

What a runtime would need

What follows is my opinion, start to finish. Part 5 looks at the same gap from another side: my tracing fork rides on context.Context, so the context still has to be threaded.

The runtime already copies one slot when a goroutine starts: g.labels, which runtime/pprof documents as “A new goroutine inherits the labels of the goroutine that created it”. It holds string pairs for the profiler, and the OTel profiling OTEP says “we foresee Go readers will directly read pprof labels”. That is precedent for an inherited slot, but the slot belongs to the profiler.

What I’d want is goroutine-associated storage that a new goroutine inherits and that does not lose the request’s context data: the values, the deadline and the cancellation, not only a span. That raises questions I can’t answer: copy or share on go, who clears it when the goroutine exits, what happens when two tools want the one slot, and what it costs on every go statement. I have no design, and anything touching newproc1 needs the Go team’s agreement. Until someone builds that, propagation stays unsolved, and we keep choosing which of the three jobs to lose. Enough opinion from me, though. Time to run something.

Versions, links and commands checked on 2 October 2026.

Try it

Save the program above as main.go, then run it:

1
2

go mod init demo
go run .

To read what otelc copies, print the rule yourself:

1

gh api 'repos/open-telemetry/opentelemetry-go-compile-instrumentation/contents/instrumentation/runtime/otelc.yaml?ref=v1.1.0' --jq .content | base64 -d

Then go hunting in your own services for the function three frames down that never got a ctx. It’s there. 🔍

This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 4 and Part 5, and the hands-on side of my GopherCon UK talk. It all ran on one machine: a Colima VM (Ubuntu 24.04, kernel 6.8.0, linux/arm64, 2 CPUs, 2 GB) on an M4 Max laptop, not Docker Desktop. It’s one run, a demo and not a benchmark.

A server with nothing in it

Here is the whole service. Its handler hashes the order ID 200,000 times so the profiler has CPU to sample, and //go:noinline keeps lookupOrder on the stack, as checkout does in Part 4.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

package main

import (
	"crypto/sha256"
	"fmt"
	"log"
	"net/http"
)

//go:noinline
func lookupOrder(id string) string {
	sum := sha256.Sum256([]byte(id))
	for range 200_000 {
		sum = sha256.Sum256(sum[:])
	}
	return fmt.Sprintf("order %s, checksum %x", id, sum[:4])
}

func main() {
	http.HandleFunc("GET /orders/{id}", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, lookupOrder(r.PathValue("id")))
	})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

No OpenTelemetry, no pprof. The GET /orders/{id} pattern is Go 1.22 routing. The Dockerfile builds with Go 1.27.1 and strips the binary, as in Part 4, into an empty image:

1
2
3
4
5
6
7
8

FROM golang:1.27.1 AS build
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=0 go build -ldflags='-s -w' -o /orders .

FROM scratch
COPY --from=build /orders /orders
ENTRYPOINT ["/orders"]

Did the stripping work? I copied the binary out with docker cp:

1
2
3
4
5
6
7

$ go tool nm orders-linux-arm64
reading orders-linux-arm64: no symbol section
reading orders-linux-arm64: no symbols
$ strings orders-linux-arm64 | grep -E '^main\.'
main.lookupOrder
main.main
main.main.func1

No symbols, yet the function names are still in the file. That is .gopclntab again, and the profiler will read it later. (Yes, foreshadowing. 👀) First, the cast.

The compose file

Four more services join the server: Jaeger 2.21.0 for traces (as in Part 6), a curl loop sending GET /orders/42 a few times a second, OBI v0.10.0 and the profiler. Here is the whole file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

name: see-it-run

services:
  orders:
    build: ./app

  jaeger:
    image: jaegertracing/jaeger:2.21.0
    ports:
      - "16686:16686"

  load:
    image: curlimages/curl:8.11.1
    entrypoint: ["/bin/sh", "-c"]
    command:
      - 'while true; do curl -s http://orders:8080/orders/42 > /dev/null; sleep 0.2; done'
    depends_on:
      - orders

  obi:
    image: otel/ebpf-instrument:v0.10.0
    privileged: true
    pid: "service:orders"
    environment:
      OTEL_EBPF_OPEN_PORT: "8080"
      OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4318
    depends_on:
      - orders
      - jaeger

  profiler:
    image: otel/opentelemetry-collector-ebpf-profiler:0.160.0
    privileged: true
    pid: host
    command:
      - --config=/etc/profiler.yaml
      - --feature-gates=+service.profilesSupport
    volumes:
      - /sys/kernel/debug:/sys/kernel/debug:ro # tracepoints are looked up here
      - profiles:/out
    configs:
      - source: profiler
        target: /etc/profiler.yaml
    depends_on:
      - orders

configs:
  profiler:
    content: |
      receivers:
        profiling:
      exporters:
        debug:
          verbosity: detailed
        file:
          path: /out/profiles.json
      service:
        pipelines:
          profiles:
            receivers: [profiling]
            exporters: [debug, file]

volumes:
  profiles:

OBI’s pid: "service:orders" puts it in the server’s PID namespace, so the server is PID 1 to it. OTEL_EBPF_OPEN_PORT tells it to instrument whichever process has port 8080 open. The OBI Docker docs ask for the host PID namespace; sharing only the server’s was enough here.

The profiler gets pid: host to sample the whole node, and the feature-gate flag its README uses. Collector 0.160.0 is the newest that bundles profiler v0.0.202633, the tag Part 4 cites. Its config is inline because bind-mounting a file from /tmp into my VM gave me an empty directory.

First, what OBI makes of the server.

OBI finds the server

Compose builds and starts everything. Then we read OBI’s log:

1
2
3

$ docker-compose up -d --build
$ docker-compose logs obi | grep "instrumenting process"
obi-1  | time=2026-10-04T07:50:01.412Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/orders pid=1 ino=1071739 type=go service="" logenricher=false

OBI found /orders, recognised a Go binary (type=go) and attached, with no rebuild. The server never noticed: I restarted OBI 90 seconds later, it attached again, and the server’s start time stayed at 07:49:59. Now Jaeger’s query API, condensed to the fields that matter:

1
2
3
4
5
6
7
8

$ curl -s 'localhost:16686/api/v3/traces?query.service_name=orders&query.num_traces=1&query.start_time_min=...&query.start_time_max=...'
span 'in queue' kind=1 parent='88d93c2068b37f04' duration=0.02ms
span 'processing' kind=1 parent='88d93c2068b37f04' duration=11.61ms
span 'GET /orders/{id}' kind=2 parent='-' duration=11.63ms
    http.request.method = GET
    http.response.status_code = 200
    http.route = /orders/{id}
    url.path = /orders/42

One server span (kind 2) with method, status and path, plus two internal spans, in queue and processing. http.route is the registered pattern; url.path is the raw path. Give traces 15 to 20 seconds to show up. On my first attempt nothing appeared in 45 seconds, though OBI’s log looked fine. Recreating the container (with debug logging on) fixed it, and I never found the cause.

A caller’s trace is continued too. I sent a traceparent header and asked Jaeger for the trace ID I had chosen: the server span carried it, and its parent was the span ID I sent. Passing context on to outgoing calls needs the extra capabilities Part 2 lists. I didn’t turn that on, so it’s untested here.

That’s the trace. The same request also burned CPU, and a trace can’t tell us where.

The profiler reads a stripped binary

Why not net/http/pprof? It needs an import and a redeploy. The eBPF profiler already watches every process on the node. Its first start on my VM failed:

1

Error: cannot start pipelines: failed to start "profiling" receiver: failed to attach scheduler monitor: failed to configure tracepoint on tracer.hookPoint{group:"sched", name:"sched_process_free"}: neither debugfs nor tracefs are mounted

The VM has debugfs, but a privileged container can’t see it, hence the /sys/kernel/debug mount in the compose file. With it the profiler reports “Everything is ready” and prints a profile every five seconds. Here are two lines of the output, our server and the profiler’s tag:

1
2

     -> process.executable.path: Str(/orders)
InstrumentationScope go.opentelemetry.io/ebpf-profiler v0.0.202633

Our server is in there, next to dockerd, containerd, OBI and the profiler itself. The debug output refers to frames by index, so we read the file exporter’s JSON. This script (lookup-share.py) prints the first orders stack containing main.lookupOrder, then counts how many samples do:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import json, sys

hit = total = 0
first = None
for line in open(sys.argv[1]):
    doc = json.loads(line)
    d = doc["dictionary"]
    s = d["stringTable"]

    def frames(stack):
        locs = d["stackTable"][stack]["locationIndices"]
        return [s[d["functionTable"][ln["functionIndex"]]["nameStrindex"]]
                for i in locs for ln in d["locationTable"][i].get("lines", [])]

    for rp in doc["resourceProfiles"]:
        res = {s[a["keyStrindex"]]: a["value"] for a in rp["resource"]["attributes"]}
        exe = res.get("process.executable.name", {}).get("stringValueStrindex")
        if exe is None or s[exe] != "orders":
            continue
        for sp in rp["scopeProfiles"]:
            for prof in sp["profiles"]:
                for sample in prof.get("samples", []):
                    total += 1
                    names = frames(sample["stackIndex"])
                    if "main.lookupOrder" in names:
                        hit += 1
                        first = first or names
print("\n".join(first))
print(f"{hit} of {total} samples from orders have main.lookupOrder on the stack")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ python3 lookup-share.py profiles.json
crypto/internal/fips140/sha256.blockSHA2
crypto/internal/fips140/sha256.(*Digest).Write
crypto/internal/fips140/sha256.(*Digest).checkSum
crypto/internal/fips140/sha256.(*Digest).Sum
crypto/sha256.Sum256
main.lookupOrder
main.main.func1
net/http.HandlerFunc.ServeHTTP
net/http.(*ServeMux).ServeHTTP
net/http.serverHandler.ServeHTTP
net/http.(*conn).serve
net/http.(*Server).Serve.gowrap3
runtime.goexit
100 of 106 samples from orders have main.lookupOrder on the stack

There’s the thought we held on to. The binary has no symbol table, yet the stack reads main.lookupOrder, net/http.(*conn).serve and runtime.goexit, because .gopclntab is still in the file. Read from the top, that’s a text flame graph: 100 of 106 samples were in our hashing loop, called from our handler, called from net/http. Drawing it needs an OTLP profiles backend; the README offers devfiler (a development tool, it says) and Pyroscope.

What this doesn’t show

A Compose file is a Kubernetes DaemonSet in costume, and Part 6 has the dry-run route for a cluster. The signal is Alpha, and this was one request shape on an idle VM. It shows the plumbing works. Your fleet’s verdict is a separate experiment.

Versions, links and commands checked on 4 October 2026.

Try it

I used the standalone docker-compose 5.6.0 binary because my docker compose plugin link was broken; either spelling works. Put compose.yaml next to an app/ directory with main.go, the Dockerfile, and a go.mod (module example.com/orders, go 1.27). Then:

1
2
3
4

docker compose up -d --build
docker compose logs obi | grep "instrumenting process"
docker compose logs profiler | grep "Everything is ready"
docker compose down -v

Jaeger’s UI is on localhost:16686. Try one of your own stripped binaries and see what names come back.

One request, two signals, zero edits to the server. It’s still the same 24 lines, and it doesn’t know anyone is watching. Go see what yours looks like from the outside. 🔭

This is a companion to the Why Your Go Benchmarks Are Lying series, between Part 3 (the laptop) and Part 4 (CI). Those parts use microbenchmarks. This story is about a macrobenchmark, a whole build timed end to end, and it didn’t fit my GopherCon UK talk. Disclosure: I work at Datadog and I’m one of otelc’s maintainers.

A caveat on sourcing. The CI numbers and the PR discussion are public. The two laptop numbers are first-hand: they aren’t recorded in the PR, so you have my word for them and nothing else. As of 2 October the CI job logs have also expired (GitHub answers HTTP 410), so everything from CI below comes from comments on the PR.

What the check measures

otelc instruments Go programs at compile time, through go build -toolexec. Every package in a build spawns an otelc process, even when no rule matches it. The benchmarking doc at v1.1.0 calls that overhead “the primary UX metric to track”, and the check times three scenarios:

The overhead is (otelc − plain) / plain, so 230% means the instrumented build took about 3.3 times as long. The ceiling is 150% for every scenario except baseline, which gets 500%. Here is the heart of the test as it stood when #643 merged:

1
2
3

plain := minBuildTime(t, dir, plainArgs, name)
otelcTime := minBuildTime(t, dir, otelcArgs, name)
pct := (otelcTime - plain) / plain * 100

minBuildTime runs a full rebuild (go build -a) three times and keeps the fastest, a common way to cut noise. Look at the order, though. All three plain builds run first, then all three otelc builds. If the machine changes its mood between the two halves, the percentage absorbs the whole difference. That detail will matter in a minute.

Two numbers for one change

The pull request, #643, consolidated a pile of dependency bumps, among them go.opentelemetry.io/otel core to v1.44.0. On 3 July I ran the overhead check on my own machine. I’ll keep calling it “the laptop”, since the title is already committed. I’d been running parallel builds and integration tests on it all session. CI ran the same check that day.

That’s one run on each side, from different machines, and I can’t promise both ran the identical commit. The laptop column is my recollection of the terminal, hence the tildes.

Read cold, the laptop column says the bump broke two scenarios. The CI column says only baseline fails. Which one do we believe?

The scope tell

Ask what the change could have touched. The PR edited multi’s own go.mod and go.sum, and nothing in largeidle’s module. Yet largeidle came out at 212%, about as far over its ceiling as multi. When a scenario the change didn’t touch moves by the same amount as one it did, suspect the machine before the diff.

I should be fair to the scenario, though. It wasn’t a perfectly clean control. The description of the later fix, #650, says otelc always injects the OpenTelemetry SDK initialization package, so an SDK bump can reach every scenario, largeidle included. CI settles it. On CI largeidle sat at +105%, a hundred points under the laptop’s reading, with the SDK bump already in. Whatever the bump cost, it doesn’t explain 212%.

I waited for CI instead of hunting for a regression in code that hadn’t changed. I never reran the check on a quiet machine and I didn’t log CPU pressure, so “load did it” is my attribution. It fits every number above. I can’t prove it.

Now the detail from a few paragraphs ago. In the rerun on the PR, baseline went from 2.146s plain and 13.197s with otelc to 3.432s and 21.344s. Both times grew by about 60%, and the overhead moved from +514.9% to +521.9%. A machine that is uniformly slower cancels out of a ratio, which is why CI’s failure repeated so steadily. What doesn’t cancel is load that arrives between the plain half and the otelc half, or that hits one build harder than the other. A full -a rebuild fans out across every core, so parallel builds on the same machine are a likely source. That part is my guess too.

CI’s failure was real, though. Only baseline crossed its ceiling, and my comment on the PR and the #650 description both put it down to the SDK bump. My reading of why baseline feels it most: it compiles a program with one fmt call, so a fixed cost for compiling the SDK dominates its percentage. #650 merged on 6 July and raised that ceiling from 500% to 550%. The maintainers loosened the limit rather than hold back the dependency bump. Judging a real measurement is a different job from catching a lying one.

Next, a confession about CI.

CI is a second opinion, not the truth

We could end with “trust CI”. The pinned workflow doesn’t back that up. At the commit where #643 merged, the overhead job declares runs-on: ubuntu-latest. That label names an image. It doesn’t tell us how busy the host was. The workflow’s other job, the CodSpeed walltime run, names a bare-metal machine instead. I don’t know what else was running next to the overhead check, and I won’t pretend to.

What CI gave us was a second measurement with different noise, and the two disagreed in a pattern we could read. That is Part 4’s territory: shared runners add their own kind of weather. Two readings and a scope argument beat one reading and a hunch.

Versions, links and commands checked on 2 October 2026.

Before you trust a macrobenchmark

Next time a number looks alarming, ask the cheap question first: what scenario in this run could not have changed? If it moved too, the diff is innocent until proven otherwise. Then check how the test orders its halves, since a ratio only protects you from noise that lands on both sides.

The laptop said 230%. CI said 81%. The 230% told me more about my machine than about the pull request, and the question that showed it cost nothing to ask. 🌡️

Let’s start with a binary rather than a profiler. Here is the whole program, with //go:noinline so the compiler can’t fold checkout into main:

1
2
3
4
5
6
7
8

package main

import "fmt"

//go:noinline
func checkout(n int) int { return n * 2 }

func main() { fmt.Println(checkout(21)) }

We’ll build it for Linux, strip it at link time, and ask what’s left:

1
2
3
4
5
6
7
8
9

$ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -ldflags='-s -w' -o stripped .
$ go tool nm stripped
reading stripped: no symbol section
reading stripped: no symbols
$ llvm-readelf -S stripped | grep -E 'symtab|debug|gopclntab'
  [ 5] .gopclntab        PROGBITS        00000000000bbf28 0abf28 097cf1 00   A  0   0  8
$ strings stripped | grep -E '^main\.'
main.checkout
main.main

One build, one machine: Go 1.27.1 on macOS, cross-compiling for linux/arm64. The symbol table and every .debug_* section are gone (plain readelf -S works too if llvm-readelf isn’t on your PATH), yet .gopclntab is still there and strings can read our function names out of the file. We’ll keep pulling on that: a stripped binary that still knows its function names, and a profiler that knows whom to ask.

This is part 4 of 6 in the How to Instrument Go Without Changing a Single Line of Code series, the written companion to my GopherCon UK 2026 talk. Part 3 moved the intervention to build time. Now we go back to the kernel for profiles, the fourth OpenTelemetry signal after traces, metrics and logs. Disclosure: I work at Datadog and I’m one of otelc’s maintainers, so read my comparisons with that in mind.

.gopclntab: the debug info Go can’t remove

Why did .gopclntab survive? Because the Go runtime can’t live without it. The runtime needs to walk the call stack at any time, for panics, for the garbage collector, for goroutine dumps, and it does that with this program-counter-to-line-number table. It maps raw instruction addresses to function names, file names and line numbers, and since the runtime always needs it, strip leaves it alone.

The profiler’s docs make the same point: the data stays present even in static, stripped executables, which is what lets it symbolize and unwind production executables.

A stripped Go binary can be read from the outside, then. Who does the reading?

What the profiler actually is

The reader is the opentelemetry-ebpf-profiler. It started as the Elastic Universal Profiling Agent. Elastic proposed the donation in January 2024, and OpenTelemetry approved it that June. Run the official Collector distribution, otelcol-ebpf-profiler (started with --feature-gates=+service.profilesSupport), from opentelemetry-collector-releases, not the same-named test binary in the repo’s own cmd/ directory, which is for internal use only. The latest tag as of the talk (13 August 2026) is v0.0.202633 (which reads as ISO week 33 of 2026). The project publishes tags, not GitHub Releases.

An eBPF program fires on CPU sample events in the kernel and reads process-internal data structures from outside the target process. No LD_PRELOAD, no ptrace attach, and no injector (part 1 covers why that last one is hard for Go). The README states it plainly: “100% non-intrusive: there’s no need to load agents or libraries into the processes that are being profiled.” Our stripped binary would never know anyone had read it. You typically run it as a DaemonSet with hostPID: true.

Why not net/http/pprof? It needs a blank import and a redeploy to switch on, which is the change this series avoids. In my view it still has the edge on profile types and detail, and it needs no kernel privileges.

Nothing gets loaded into the application. The fine print is on the node.

What “zero code changes” actually means

The profiled applications need nothing: no rebuild, no restart, no SDK import. The node pays instead.

The example config says to run the Collector as root, or to grant specific capabilities (it names CAP_SYS_ADMIN, CAP_PERFMON, and/or CAP_BPF, plus access to /proc). The README also says current code may require Linux 5.10 or greater.

The bill arrives per node, not per service, which is what makes it useful at scale. You still need a backend that takes OTLP profiles, and the README says mature production-ready ones have yet to emerge (Pyroscope and devfiler are its open-source options, and devfiler is Elastic’s desktop viewer for experiments, not a production backend).

A privileged agent on every node buys us CPU profiles. How far can we trust the signal?

How mature is it?

Young. These are the statuses as of October 2026, from the profiles concepts page and the specification (spec v1.61.0, OTLP 1.11.0):

The last row is the only one with a comfortable word in it.

The README is explicit: “Implements the Alpha OTel Profiles signal.” It calls its own implementation “functional but work-in-progress / evolving”, and Alpha means evolving, not backward-compatibility-guaranteed.

What works today, reliably, is on-CPU stacks sampled per OS thread, with correct Go function names via .gopclntab, the trick we started with. It also reads Go pprof labels by following the thread’s current g to its goroutine, so samples carry whatever labels the application set. The README also lists Java, Python and other runtimes it can unwind, though I only checked the Go path. Off-CPU profiling is in development and allocation profiling is a design proposal.

Alpha, then, with a working core: always-on CPU flame graphs for production Go services. Where does that sit next to the tools from the last two parts?

Where the fourth signal fits

OBI gives us traces and RED metrics for HTTP and gRPC from the kernel, otelc gives us function-level spans for the price of a rebuild, and the profiler adds always-on CPU profiles for the whole node. Together they cover three of the four signals (traces, metrics, profiles) without a single source change in the application. Logs are only partly covered: OBI can add trace context to application logs (Linux 6.0 or newer), but it enriches them and does not export them.

The useful move is to jump from a slow trace to the flame graph for the same time window: traces tell you a request was slow, profiles tell you where the CPU went. Lining them up by request is where zero-touch has a gap. The data model lets samples carry trace and span IDs, but for Go the profiler can only read request context from pprof labels that the application sets, the path OTEP 4947 describes. A compile-time tool could set span-ID labels for the profiler to read, but that is proposed work, and I haven’t verified that otelc or dd-trace-go do it. Zero-touch profiling alone correlates by resource and time window, not by request, so treat trace-to-profile pivoting as where this is heading.

Back to our stripped binary. We never rebuilt it, restarted it or added an import, and it still carries main.checkout in .gopclntab for strings or a privileged profiler to read. The signal is Alpha, but Elastic ran the same agent in customer production before donating it, so the capability is real: continuous CPU profiling with correct symbolization of stripped Go binaries. The commands at the top of this post check that any Go binary you ship still has its names; running the profiler itself takes a Linux host and the Collector command in its README, and See it run: OBI and the eBPF profiler without Kubernetes does that in a VM. Try the check on yours, then go see where your CPU time goes. 🔍

Versions, links and commands checked on 2 October 2026.

Up next

Part 5, Go runtime futures: flight recording, USDT, and the instrumentation hook problem, looks at what Go itself could add: flight recording, which shipped, and the hooks that are still missing. Three tools have worked around Go so far. Time to ask Go for help.

Note

Why this milestone matters to me

I have been involved with the Go Compile-Time Instrumentation SIG since it kicked off at the start of 2025, when Alibaba and Datadog decided to pool their efforts into a single vendor-neutral project. Watching it reach v1 last week felt genuinely satisfying — not because “v1” is a magic number, but because it marks the point where we stopped calling it experimental and started calling it something people should actually run in their builds.

The Go situation has always been a little awkward for OpenTelemetry. Every other language has a story: attach a Java agent, install a Python auto-instrumentation package, load a Node.js module. Go compiles to a static binary with no runtime hook points. The options were either manual instrumentation (correct but tedious) or an eBPF agent running outside the process (zero-code but with its own caveats). Compile-time instrumentation is the third option: use Go’s own toolchain machinery (specifically the -toolexec flag) to inject instrumentation at build time, not at startup.

The reason I find this bet interesting is that it sits exactly where Go’s constraints become an asset. Because we operate during compilation, we have the full type information and package graph available. We can instrument code you didn’t write and wouldn’t want to modify: the net/http standard library, database/sql, gRPC, Redis drivers. The binary you ship has the telemetry baked in, so there is no separate process to manage, no LD_PRELOAD tricks, no eBPF programs to load. It is just a Go binary, compiled slightly differently.

I am proud of what the SIG shipped and grateful to the maintainers and contributors who drove it across the finish line. The acknowledgments section below names them properly. v1 is a foundation — coverage will grow, the registry integration will land, and the performance story will get tighter. If you run Go services and want traces and metrics without touching application code, now is a good time to try it.

The announcement

What follows is the original post as published on the OpenTelemetry blog.

If you write Java, Python, Node.js, or .NET, you have been able to add OpenTelemetry to an application without editing its code for years: attach an agent at startup and telemetry starts flowing. Go has been the exception. A Go program compiles to a single static binary with no runtime to hook into at startup, so Go developers have had to instrument by hand or reach for an out-of-process eBPF agent.

That gap is closing. The OpenTelemetry community is announcing the first stable release of OpenTelemetry Go Compile-Time Instrumentation. When we announced this SIG at the start of 2025, Alibaba and Datadog joined forces to build one unified, vendor-neutral way to instrument Go at build time. v1 is that project’s first stable release.

If you build and run Go services, you can change a single line in how you build your binary or container image and get OpenTelemetry traces and metrics for your application and its dependencies, with no code changes. For a platform engineer or an SRE, that means you can add observability to services across your fleet without waiting for every team to instrument their own code.

What is Go Compile-time Instrumentation?

Go compiles to a single static binary, which has long made automatic instrumentation harder than in interpreted languages. This project hooks into the standard Go toolchain during the build (through its -toolexec mechanism) and injects OpenTelemetry instrumentation into your code, its dependencies, and the standard library as they are compiled. There is no separate agent and nothing to attach at runtime.

For you, that means telemetry with no source-code changes: the instrumentation is compiled directly into your binary. Your application code stays free of instrumentation concerns, and you get coverage for third-party libraries you don’t own.

Key capabilities in v1

• Zero-code instrumentation: instrument an application and its dependencies without manual code changes.
• Compile-time injection, no added runtime overhead: instrumentation is built into the binary instead of attached at runtime.
• Third-party and standard-library coverage: instrument dependencies and standard-library packages you don’t own.
• Supported instrumentations in v1: common libraries and frameworks including net/http, database/sql, gRPC, Redis, and Go runtime metrics, with more added regularly. See the supported libraries for the full, current list.
• Rule-based and extensible: add support for new libraries through the SIG’s instrumentation-rule format. See the instrumentation guide and the rules reference.
• Semantic-convention compliance: emitted telemetry follows current OpenTelemetry semantic conventions.
• CI/CD friendly: run the tool at development time or drop it into your build pipeline.

Getting started

The project ships a command-line tool called otelc that wraps the standard Go toolchain. The change to your build is a single line: run otelc go build where you used to run go build. Everything after go is forwarded to the toolchain, so the rest of your build stays the same.

Install it with go install:

1

go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest

Then build your application through it:

1

otelc go build -o myapp .

If you’d rather not change your build command, run otelc setup once to prepare the module, then point the Go toolchain at otelc through GOFLAGS and keep running go build as usual:

1
2
3

otelc setup
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'"
go build -o myapp .

By default, otelc discovers the supported libraries in your module and instruments them automatically, with no configuration and no code changes. The same swap works in a container build: install otelc in your build stage and replace the go build line in your Dockerfile with otelc go build. For the full walkthrough, see the compile-time instrumentation documentation.

When should you use it?

If you write or operate Go services, you now have three complementary ways to get OpenTelemetry telemetry, and compile-time instrumentation is the third option promised in the founding post:

• Compile-time instrumentation (this project): best when you can rebuild the application and want no code changes, no added runtime overhead, and coverage of dependencies and the standard library.
• eBPF instrumentation (OpenTelemetry eBPF Instrumentation, or OBI): best when you can’t rebuild the binary, or want zero-code, multi-language instrumentation from outside the process.
• Manual instrumentation with the OpenTelemetry Go API: best for custom spans and domain-specific telemetry, and it composes with the other two.

v1 ships a focused set of instrumentations rather than the full breadth of the Go ecosystem, and coverage will grow with each release. If a library you depend on isn’t covered yet, you can add a rule for it or combine compile-time instrumentation with manual spans. The three approaches above solve overlapping problems in different ways, so we’re working with the OBI and Go SIGs on follow-up posts that compare them in more depth.

What’s next

v1 covers the core of the compile-time approach. Our priorities from here:

• More instrumentations: broaden coverage across popular Go libraries and frameworks so more applications work the moment you swap in otelc.
• Registry-based discovery: use the existing OpenTelemetry Registry to discover and distribute instrumentations, so you can pick up support for new libraries without waiting for a new otelc release.
• Performance: keep driving down both build-time and runtime cost.
• Adoption and awareness: invest in docs, examples, and outreach so teams know the tool exists and how to fit it into their build.

Get involved

Compile-time instrumentation is v1, and the best way to shape where it goes next is to use it and get involved.

• Try it and tell us how it went. Add otelc to a build and share what worked and what didn’t in the project’s GitHub discussions and issues. Real-world feedback drives the roadmap.
• Instrument a library you use. Coverage grows through the SIG’s rule format; the instrumentation guide walks through adding one.
• Join the SIG. Find us in #otel-go-compile-instrumentation on the CNCF Slack and at our SIG meetings.

Acknowledgments

Reaching v1 is a milestone for the whole Go Compile-Time Instrumentation SIG. Thank you to the maintainers who drove the release through to stable, including Xabier Martinez (Cabify), Yi Yang (Alibaba), Haibin Zhang (Alibaba), and Dario Castañé (Datadog), along with everyone who contributed code, rules, documentation, and feedback.

A special thank-you to Azhar Momin, who joined the project through the LFX Mentorship program and has become one of its most active contributors and an approver.

References

• Original post on the OpenTelemetry blog (2026-07-16)
• opentelemetry-go-compile-instrumentation — the project on GitHub
• otelc module — go.opentelemetry.io/otelc
• Supported libraries
• OpenTelemetry Registry
• Original SIG announcement post (2025)
• PR #10760 — the community blog post PR

Today we’ll interrogate one laptop with one benchmark. BenchmarkMakeBuffer_Correct costs 11.46 ns/op on an idle Apple M4 Max. Start sixteen busy loops next to it and the same code costs 34.97 ns/op. Pin it to a single CPU inside a container and it settles at 16.28 ns/op, with its variance back near the idle level. Hold on to those three conditions; we’ll keep coming back to them.

This is part 3 of 5 in the Why Your Go Benchmarks Are Lying series, the written companion to the GopherCon UK 2026 talk. The series asks three questions of a benchmark: does the compiler let it do real work (part 1), is the sample stable enough to mean anything (part 2, which also defined CV), and is the difference large relative to the noise. This part takes the last question down to the machine in front of us, the noise floor under any difference: we’ll see what a pinned container on a Mac buys and where it stops, then borrow the Linux controls that go further. The FOSDEM 2026 post is the prerequisite for the series, and every number below comes from the demo in gopherconuk-26 at fdce88c.

Can we trust the laptop? Sometimes. Let’s find out when.

One benchmark, three conditions

We ran the benchmark with -count=20 -benchtime=1s on an Apple M4 Max (darwin/arm64, 16 logical CPUs) under three conditions. The “noisy” one starts 16 background spinners, pure busy-loops with no I/O and no allocation that saturate every logical core. The third reruns the benchmark on that same noisy host inside a golang:1.26 container started with --cpuset-cpus=0 --cpus=1 --memory=512m. The spinners never stop.

That’s one run of 20 samples per condition on one machine. The host’s Go version wasn’t recorded, which is a little embarrassing for a post about reproducibility. We’ll come back to that.

Idle to noisy: the benchmark slowed by 3× and CV jumped from 4.75% to 18.88%. Both effects most likely share a cause: the spinners and the benchmark compete for the same 16 cores, so the scheduler preempts the benchmark and moves it between cores, including the slower efficiency cores on this chip. This experiment does not separate those effects.

Noisy to pinned is the interesting one. CV drops to 5.25%, nearly back to the idle noise floor. The absolute runtime stays elevated (16.28 ns/op against 11.46 idle), which is unsurprising for a single pinned vCPU (GOMAXPROCS=1, judging by the missing -16 suffix in the pinned run’s output) inside a VM running a different OS, though this experiment does not isolate which of those accounts for it. The variance is the point, and it collapses, though at one run per condition the gap between 4.75% and 5.25% is within what sampling error alone could produce.

Pinning works, then. Sort of. 5.25% is not a triumph, and why it isn’t tells us what a container can’t do.

What a container on a Mac can and can’t pin

On a bare-metal Linux runner with SMT disabled and CPU frequency pinned, the FOSDEM 2026 experiments measured around 0.05% CV on a CPU-bound task. That is roughly a hundred times tighter, across different workloads, than Docker on a Mac gave us.

Docker Desktop on macOS runs containers inside a Linux VM (here, Apple’s Virtualization.framework). When we pass --cpuset-cpus=0, we pin to vCPU 0 inside that VM, not to a physical core on the host. The host scheduler can migrate that vCPU between physical cores at will, and nothing inside the container can pin the host CPU clock frequency. It’s a quiet room in a building whose thermostat someone else controls.

What the pinning does buy is a stable CPU for the benchmark’s threads, which stop migrating across the VM’s virtual CPUs, and a VM with nothing else running in it. The table shows that matters. Frequency scaling, thermal variation and the host’s choice of physical core for the VM’s threads all remain in play.

The Mac numbers are good for catching obvious regressions during development, not for publication. Results that need to hold up to scrutiny belong on bare-metal Linux with SMT and frequency scaling disabled, with a note in the writeup saying so. Let’s see what that environment is made of.

The Linux toolbox

On Linux we can reach the knobs the Mac’s VM hides. We’ll go from free to fussy.

CPU affinity

The OS scheduler migrates processes between cores to balance load, and each migration evicts warm cache lines. taskset stops that at zero setup cost:

1

taskset -c 0 go test -bench=. -count=10 -benchtime=2s ./...

The -c 0 pins the whole run to core 0. It’s worth it on Linux whenever we run -count of 10 or more. One side effect: the Go runtime reads the affinity mask, so this also sets GOMAXPROCS=1 for the benchmark, which changes GC scheduling and any parallel benchmarks. Free isn’t the same as harmless. When a pin isn’t enough, we can take cores away from the scheduler entirely.

Core isolation

With isolcpus on the kernel command line, specific cores leave the scheduler’s load balancing, so only tasks we pin to them run there:

1
2

# Add to kernel command line
isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

After a reboot we pin benchmarks to the isolated cores with taskset -c 2. That suits a dedicated CI runner, since it needs a reboot and root. (nohz_full stops the scheduler tick on those cores while only one task runs there, and rcu_nocbs moves RCU callbacks off them; see the kernel NO_HZ docs.)

For a softer version that needs no reboot, cset shield achieves similar isolation at runtime, but only on hosts that still use cgroup v1:

1
2
3

sudo cset shield --cpu=2,3 --kthread=on
sudo cset shield --exec -- go test -bench=. -count=10 ./...
sudo cset shield --reset

Create the shield, run inside it, tear it down. The --kthread=on flag is critical: without it, kernel threads still run on the shielded cores.

Process priority

nice raises the scheduling priority of the benchmark relative to other processes:

1

sudo nice -n -5 go test -bench=. -count=10 ./...

The effect is modest on a quiet machine but meaningful under moderate load. On dedicated machines, chrt -f (SCHED_FIFO) goes real-time:

1

sudo chrt -f 50 go test -bench=. -count=10 ./...

SCHED_FIFO can starve normal processes, the display server included, if the benchmark misbehaves (the kernel’s default real-time throttling leaves them about 5%). That’s a fine way to lose your terminal, so keep it to dedicated machines or a resource-capped container. Isolation and priority decide who gets the CPU. Frequency decides how fast it runs.

CPU frequency control

Dynamic frequency scaling (DFS) is a large source of hidden variance on Linux: the CPU boosts when thermals allow and throttles when they do not.

perflock, written by Austin Clements of the Go team, locks the CPU to a stable frequency for the duration of the run. It talks to a small daemon that holds the lock, so the setup has two steps:

1
2
3
4
5

GOBIN=$PWD go install github.com/aclements/perflock/cmd/perflock@b67f3f23152f
sudo install ./perflock /usr/bin/perflock
sudo -b perflock -daemon

perflock go test -bench=. -count=10 -benchtime=2s ./...

The first three lines build it at a pinned commit, install it and start the daemon. The last runs the benchmark under the lock. On Linux it pins the frequency by writing scaling_min_freq and scaling_max_freq through the cpufreq sysfs interface, which makes it one of the higher-value local tools after benchstat, daemon and all.

If we’d rather apply the sysfs controls by hand, here they are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Performance governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable Intel Turbo Boost (intel_pstate driver)
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

# Disable frequency boost (acpi-cpufreq or amd-pstate drivers)
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost

# Disable SMT (caution: halves logical CPU count)
echo off | sudo tee /sys/devices/system/cpu/smt/control
# Re-enable when done
echo on | sudo tee /sys/devices/system/cpu/smt/control

The bare-metal SMT and DFS numbers behind these controls are in part 4. Even a locked frequency leaves one thing we can’t lock: how hot the chip is when the run starts.

Thermal steady state

CPUs boost when cool and throttle when hot. A benchmark that starts on a cold chip and ends on a warm one shows a performance decrease across the run, which has nothing to do with the code. In raw go test output, the first few runs of a -count=20 sequence are typically faster than the last few (the expected pattern, not one we measured here).

A throwaway run before capturing helps, as does waiting 30 to 60 seconds between major runs (a rule of thumb, not a measured number). Holding the frequency steady with perflock or by disabling DFS helps regardless of thermals. Laptops are worse than desktops here: ambient temperature, prior CPU workload and fan behavior all affect how long steady state takes. Skip the warmup and the early samples are biased warm. With the environment as quiet as it’s going to get, we can finally compare two versions of the code.

The Go-native A/B loop

Controls reduce noise, but we also need a structured way to compare two versions. The manual approach is to capture a baseline, apply the change, capture again and let benchstat compare:

1
2
3
4
5
6
7
8

# Capture baseline
go test -bench=BenchmarkMyFunc -benchmem -count=10 -benchtime=2s . > old.txt

# Apply change, then capture
go test -bench=BenchmarkMyFunc -benchmem -count=10 -benchtime=2s . > new.txt

# Compare
benchstat old.txt new.txt

The flags and machine conditions have to match on both sides, or we end up benchmarking the room instead of the code. Captured minutes apart, the two files can also see different machines. The benchstat docs recommend interleaving the before and after runs, and go test -c makes that cheap: build both test binaries once, then alternate them.

1
2
3
4
5
6
7

GOOS=linux go test -c -o old.test   # on the base commit
GOOS=linux go test -c -o new.test   # with the change
for i in $(seq 10); do
  ./old.test -test.run='^$' -test.bench=. -test.count=1 >> old.txt
  ./new.test -test.run='^$' -test.bench=. -test.count=1 >> new.txt
done
benchstat old.txt new.txt

On a Mac, the GOOS=linux builds run inside the pinned container (docker run --cpuset-cpus=0 -v "$PWD":/w -w /w ..., with the loop as the command) so both sides share its conditions. For the inner development loop, benchdiff automates the cycle, with the base ref running in a temporary git worktree:

1
2

go install github.com/willabides/benchdiff/cmd/benchdiff@v0.9.1
benchdiff --base-ref=main --benchmem --count=10 --benchtime=2s

It runs both sides under the same conditions and compares them with a bundled, older copy of benchstat.

The toolchain is a condition too, and the one we forgot to record in our own table. Since Go 1.21 the go directive in go.mod is a minimum, not a pin: a newer local toolchain runs the module anyway. Different Go versions produce different numbers, so pin with GOTOOLCHAIN=go1.26.5 and mention the version in any result you share. The last round of fixes needs no tools at all.

Cheap wins that take five minutes

Close the browser, IDE and Slack, the most expensive background processes on most laptops. On macOS, turn off Spotlight indexing with sudo mdutil -a -i off so the indexer doesn’t spike mid-run, and undo it afterwards with sudo mdutil -a -i on. Switch off Wi-Fi and Bluetooth (airplane mode, if your OS has one) to cut NIC interrupts and background traffic. On a noisy machine, closing the browser alone can noticeably lower CV before you touch anything else. Nobody has ever regretted closing Slack.

Back to the three rows

Back to the rows from the top: 4.75% CV on an idle Mac, 18.88% with the spinners running, 5.25% in the pinned container. On a Mac, we run -count=20 -benchtime=2s, watch the ± column in benchstat (a 95% confidence interval, not CV), compute CV as in part 2, and treat a CV above 5% as a reason to investigate the environment and above 10% as a reason not to trust the comparison at all (the bands from part 2).

Everywhere, that means -count of 10 or more, benchstat, benchdiff and closed apps. On Linux we add taskset, perflock and nice. On a dedicated machine or runner we add SMT and frequency control, isolcpus and cset shield. On a Mac, a pinned container helps with variance, not absolute numbers. Time to check our own numbers.

Try it

To reproduce the three conditions, we clone the demo at the pinned commit. This needs Docker, and the container step mounts the current directory, so run it from a path your Docker VM can see:

1
2
3
4
5
6

git clone https://github.com/kakkoyun/gopherconuk-26
cd gopherconuk-26
git checkout fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb
cd talks/go-benchmarks-lying/demo
make bench-docker
make cv

make bench-docker re-runs all three conditions (expect a minute or two) and overwrites results/, so your numbers will differ from ours. make cv recomputes the summary from whatever is in results/.

Versions, links and commands checked on 2 October 2026.

Up next

Part 4, Benchmark CI That Doesn’t Lie, takes the same discipline to CI: why shared runners lie, a two-tier gate, and the tools that back it. If sixteen spinners felt rude, wait until we meet other teams’ Docker builds. Between the two sits a companion, The laptop said 230%, where a loaded laptop and CI disagreed about one change.

1
2

go build -o myapp .
otelc go build -o myapp .

Same flags, same package, one extra word in front. The binary that comes out has OpenTelemetry instrumentation compiled in for the libraries otelc knows about, net/http and gRPC among them, and nobody opened a handler file.

That extra word is what we’ll follow through this post. This is part 3 of 6 in the How to Instrument Go Without Changing a Single Line of Code series, the written companion to my GopherCon UK 2026 talk. Part 2 covered the kernel route. Here we move the intervention to build time and follow that one command: who built it and what it does to go build, then what it sees, what it costs and where it fits, and whether “without a single line” survives the trip.

Disclosure: I work at Datadog, which maintains Orchestrion and dd-trace-go, and I’m one of otelc’s maintainers. Read my comparisons with that in mind.

Why there are two tools

Before we open the hood, a mix-up to clear: otelc has a lookalike. Naming is hard, and naming two tools that play the same trick is harder.

Orchestrion is Datadog’s tool, CLI binary orchestrion. The release pinned for the talk (13 August 2026) is v1.12.0 (2026-07-30). Its README says other vendors, OpenTelemetry among them, may provide alternate integrations, but the ones it defaults to (dd-trace-go/v2) are Datadog’s.

otelc is the OpenTelemetry Go compile-time instrumentation tool, CLI binary otelc. Why a second tool? The SIG announcement (January 2025) says Alibaba and Datadog had each proposed donating a tool, Alibaba’s opentelemetry-go-auto-instrumentation and Datadog’s Orchestrion. Then the two organizations “decided to join forces” and set up a new SIG, with further contributions from Quesma. The goal was a unified, vendor-neutral approach that “picks the best aspects of each solution”, and it “won’t be Alibaba’s or Datadog’s solution that ‘wins’”. The SIG built otelc from scratch. Orchestrion was never donated and stays at github.com/DataDog/orchestrion as Datadog’s own product.

v1.0.0 and v1.0.1 both went out on 2026-07-14, four weeks before the talk. The v1.0.0 release notes retract it because otelc pin wrote incorrect module paths, so v1.0.1 is the first usable stable release and the one pinned for the talk. v1.1.0 followed on 2026-08-24. The OpenTelemetry blog’s v1 announcement tells the milestone story (I cross-posted it).

The two tools share a mechanism and differ in codebase and default tracer, so choosing is mostly about which tracer SDK you’re committing to. The trick underneath is the same either way. Let’s see what that extra word does to go build.

The mechanism

Go’s build system has a -toolexec flag. Normally go build invokes the compiler directly. With -toolexec, the go command runs each toolchain program (compile, asm, link, vet) through a wrapper binary first. go help build calls it “a program to use to invoke toolchain programs like vet and asm.” That generic hook turns out to be exactly the right one for compile-time AOP: Orchestrion’s maintainers describe their tool as, in some ways, a “compile-time-woven Aspect-oriented Programming (AoP) framework” (golang/go#69887). My guest post builds this -toolexec mechanism up step by step, cache included.

Our extra word plugs into that hook. otelc go build adds the flag for us and registers otelc as the wrapper. When the wrapper intercepts a compile for a package one of its rules matches, it parses that package’s .go files into an AST, applies the rules (which functions to wrap, which spans to inject, how to propagate context), and hands the rewritten source to the real compiler. Everything else passes through untouched. The compiler never sees the original, so as far as it knows you wrote the instrumentation yourself, and it’s too polite to ask.

How far does the rewriting reach? The OpenTelemetry blog says:

“hooks into the standard Go toolchain during the build (through its -toolexec mechanism) and injects OpenTelemetry instrumentation into your code, its dependencies, and the standard library as they are compiled.”

If net/http needs instrumentation, otelc can inject it as net/http gets compiled into your binary. The result has OTel spans baked in: no runtime agent, no sidecar, no dynamic injection. The rewrite injects a small trampoline (a generated stub in the target function) that calls a hook function, linked in with //go:linkname. The hooks are ordinary Go in otelc’s instrumentation packages, and they call the OTel API and SDK, so at runtime the spans look like ones you’d have written by hand.

That’s the whole trick, and it comes with a price tag. Let’s read it before we buy.

The constraint you need to know

otelc needs a Go 1.25 or newer toolchain (its go.mod says go 1.25.0). The toolchain is what counts: a service whose go.mod says go 1.24 still builds with a newer toolchain, and otelc raises the directive with a warning. Builds pinned to a Go 1.23 or 1.24 toolchain can’t use otelc, nor Orchestrion v1.12.0 (go.mod). For those, OBI (part 2) needs no rebuild.

When I prepared the talk, Go 1.26 (February 2026) was current, and Go 1.27 arrived on 2026-08-19, which ends upstream support for 1.25. Toolchain pinning is a fine habit until a tool wants a newer one. In slow-upgrade environments that’s the first question to answer. If the toolchain clears, the next question is what the rebuild buys us.

What it instruments

The default rules cover a list of libraries; for your own functions you write a custom rule. A rule is a YAML entry naming a target package, a selector and an action such as inject_hooks (schema); its hooks are Go in a package of their own. At v1.0.1 the getting-started guide listed net/http (client and server), gRPC, database/sql, gin, go-redis v9, mongo-driver, Kubernetes client-go, the OpenAI Go SDK (v1 to v3) and segmentio/kafka-go. v1.1.0 added Anthropic, AWS SDK v2, linodego and mongo-driver v2; the docs’ supported-libraries page has the current list.

Orchestrion with dd-trace-go has its own list. Its supported-integrations table in dd-trace-go v2.9.1 (the version Orchestrion v1.12.0 pins) marks which integrations work with Orchestrion. They include HTTP frameworks (net/http, gin, gorilla/mux, chi, echo, fiber), gRPC, database/sql and the layers above it (pgx, gorm, mongo-driver), Redis (go-redis v6 to v9, redigo, rueidis), Kafka, AWS SDK v1 and v2, and Kubernetes client-go. Compare both lists against your own dependencies before choosing.

Coverage is one axis. The other is whether to rebuild at all, so let’s put otelc next to OBI.

Where it fits relative to OBI

OBI and otelc attack the same problem, zero source code changes, from opposite ends. Which end do we pick?

OBI attaches from outside the process at runtime. It needs no rebuild, works on deployed services, and covers other languages too. But it’s bounded by what eBPF can observe at library boundaries: RED metrics and library-level spans for the Go libraries it supports (13 in the OBI v0.10.0 support matrix, the release pinned for the talk). It cannot generate custom spans or instrument business logic.

otelc works at build time. It requires a rebuild and Go 1.25 or newer. In exchange, it can reach stdlib internals, your own functions, dependencies and business logic, with a custom rule for anything the default set doesn’t cover. The injected spans have the fidelity of hand-written OTel calls, because the hooks call the same OTel API and SDK you would.

My decision rule: deployed and can’t rebuild, need baseline visibility across services? OBI. Building or rebuilding, want granular spans, on Go 1.25 or newer? otelc. Running Datadog APM with the widest framework coverage? Orchestrion. Part 6 turns this into a runbook. For now, time to run that command.

Try it

Install otelc, pinned to a version. I ran v1.1.0; its runtime rules match v1.0.1, the version pinned for the talk:

1

go install go.opentelemetry.io/otelc/tool/cmd/otelc@v1.1.0

Then build the service with otelc wrapping the build command, exactly as in the opening:

1

otelc go build -o myapp .

That’s the complete usage change. To see traces, run the binary with OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 and OTEL_SERVICE_NAME=my-go-service. otelc exports OTLP over HTTP by default, so use 4318, not the gRPC port 4317 (part 6 explains). In CI, put otelc in $PATH and prefix your existing build command. If you can’t prefix it, the getting-started guide describes a second route: run otelc setup once, then set GOFLAGS="'-toolexec=otelc toolexec'" and use plain go build.

“Without a single line” sounds absolute, so here’s the fine print. You don’t edit go.mod yourself, but otelc does touch it. During the build it adds its hook modules as replace directives, runs go mod tidy, and may raise the go directive (it prints Bumped go version when it does), then restores go.mod and go.sum when the build finishes. A .otelc-build/ directory stays behind. If a build is interrupted, the troubleshooting guide says to run otelc cleanup and git restore go.mod go.sum.

Already using otelhttp and your own spans? I built a small server with otelhttp.NewHandler and one manual span using otelc go build (v1.1.0, Go 1.27.1, macOS arm64) and sent three requests to Jaeger. Each request was one trace: otelc’s server span at the root, the otelhttp server span under it, and my manual span under that. Nothing is orphaned, but you get two server spans per request. I didn’t test dd-trace-go.

Back to our one command, then. “Without a single line” really means one build command substitution and Go 1.25 on your toolchain. I’ll take that trade most days. 🔧

Versions, links and commands checked on 2 October 2026.

Up next

Part 4, The fourth signal: continuous profiling without code changes, goes back to the kernel to add profiles, and asks what a stripped binary still remembers. Next time, nobody rebuilds anything.

The benchmark is BenchmarkMakeBuffer_Correct, run on a machine with sixteen CPU-bound background processes competing for the same cores. Here are two of its twenty runs (-16 is the GOMAXPROCS suffix), with the B/op and allocs/op columns dropped:

1
2
3

BenchmarkMakeBuffer_Correct-16    41877204    39.39 ns/op
...
BenchmarkMakeBuffer_Correct-16    52521198    27.54 ns/op

Those are runs 1 and 9 of one capture on an Apple M4 Max (darwin/arm64) from the companion repository. Same code, same binary, eight runs apart, a 43% swing. Run it once, file the PR, and you could have reported either number. (I would love to say I’ve never done that.) Scheduler preemption, cache evictions and stolen CPU time are the likely culprits. The benchmark measured what the machine was doing; it just wasn’t what we thought we were measuring. Any single result is one draw from a distribution whose shape, center and spread we haven’t seen.

This is part 2 of 5 in the Why Your Go Benchmarks Are Lying series, the written companion to the GopherCon UK 2026 talk. In part 1 we asked whether the compiler lets a benchmark do real work. Here we ask the second of the three questions: is the sample stable enough to mean anything? We also pick up the statistics for the third (p-value and effect size), whether the difference is large relative to the noise, and part 3 and part 4 then deal with the machine. The prerequisite is the FOSDEM 2026 post on measuring software performance, which covers the statistics in language-agnostic terms. Here we go one level deeper, into the Go tooling.

If one run can say 39 or 27, the first move is to stop trusting one run.

From runs to a distribution with benchstat

The cure for a noisy measurement is more samples, not better luck. Let’s collect twenty runs and let benchstat summarize them:

1
2
3
4
5

go test -bench=BenchmarkMakeBuffer_Correct -benchmem -count=20 -benchtime=1s . \
  | tee results.txt

go install golang.org/x/perf/cmd/benchstat@v0.0.0-20260929162123-406019bb8b68
benchstat results.txt

We’ll start with the idle machine as our control. This is the benchstat output for the idle capture (same -count=20 -benchtime=1s, same M4 Max), trimmed to the sec/op table because the tool also prints B/op and allocs/op:

1
2
3
4
5
6
7

goos: darwin
goarch: arm64
pkg: github.com/kakkoyun/gopherconuk-26/demo
cpu: Apple M4 Max
                      │ results/idle.txt │
                      │      sec/op      │
MakeBuffer_Correct-16        11.32n ± 5%

Two numbers here. 11.32n is the median of the 20 runs, and ± 5% is how far the 95% confidence interval around that median reaches, as a percentage of it: a range built to contain the true median 95% of the time.

Now let’s put the loaded machine next to it.

Comparing two runs

Comparison is where benchstat earns its keep. We collect one file before a change and one after, then hand it both:

1
2
3
4

go test -bench=. -benchmem -count=20 -benchtime=1s . > old.txt
# make your change
go test -bench=. -benchmem -count=20 -benchtime=1s . > new.txt
benchstat old.txt new.txt

Here the “change” is the load, so we compare the idle capture against the loaded one (sec/op only; allocations were identical):

1
2
3

                      │ results/idle.txt │           results/noisy.txt           │
                      │      sec/op      │    sec/op     vs base                 │
MakeBuffer_Correct-16        11.32n ± 5%   37.40n ± 25%  +230.34% (p=0.000 n=20)

The delta is +230.34%, so the loaded machine took about 3.3 times as long, and a p-value of effectively zero, the chance that noise alone would produce a difference this large, says the difference is real. When the p-value exceeds 0.05, benchstat prints ~ instead of a delta, which is the right answer: “no measurable difference with this sample size.” Report it as such. (The test underneath is Mann-Whitney U on the samples, which compares ranks and needs no bell curve, not the Welch’s t-test from the FOSDEM post.)

benchstat says the two machines differ. It doesn’t say whether the loaded one is a place where any comparison means anything, and that is the gap we fill next.

What benchstat doesn’t tell you

Look at the ± 25% on the loaded side. The ± figures are 95% confidence intervals on the median, and they hint at spread, but they don’t say whether the environment is stable enough to trust a comparison run in it. The interval narrows as runs are added; CV does not. For that we need a different number: the coefficient of variation.

CV = σ / μ, as a percentage. Where benchstat answers “is A different from B?”, CV answers “is this machine a reliable place to ask?” benchstat compares two distributions; it doesn’t characterize the environment producing them, much as a referee doesn’t inspect the pitch. A short awk script does that pass:

1
2
3
4

make cv
# or directly:
awk -f cv.awk results/idle.txt
awk -f cv.awk results/noisy.txt

cv.awk reads any go test -bench output and computes the mean, standard deviation and CV per benchmark. Over the idle and loaded captures (same -count=20 -benchtime=1s on the same M4 Max) it gives this table. Part 3 adds a third condition.

The loaded machine is three times slower and four times noisier. A CV of 18.88% puts the standard deviation at nearly one-fifth of the mean, so we are measuring scheduling interference at least as much as the code. Any A/B comparison collected there is uninterpretable.

My rule of thumb for reading it, not benchstat’s:

At 18.88% the loaded condition misses the 10% line by nearly a factor of two. More samples would not help, because the environment is biased and averaging does not remove a bias (noise shrinks with more runs; bias does not). What kind of bias? For that we go back to the raw runs.

Back to the run that lied

The mean of 34.97 ns/op looks unremarkable until we read the twenty runs. The first seven land between 38 and 44 ns/op. Run eight is 37. Runs nine through sixteen drop to 25 to 29, roughly a third lower. Then the last four climb back: 34, then 38 to 39. The mean sits in the gap between the two clusters, where only a couple of runs land. It describes a machine that was almost never in that state.

Now our two runs make sense. Run 1 (39.39) sits in the first cluster, run 9 (27.54) in the second. The machine shifted between two regimes during the capture, and we happened to sample one of each. Mann-Whitney assumes one distribution per sample, which these runs are not. One dot per measurement, the strip plot from the FOSDEM post, shows this at once. A boxplot would obscure it, and a mean buries it entirely.

I did not measure why the regimes differ. Scheduler interference is the likely explanation, though P/E-core placement or frequency scaling would look similar on this machine. More statistics cannot recover that; a pinned container is the start of the answer, and part 3 measures what isolation buys. First, a question we’ve been dodging: even on a good machine, how many runs do we collect?

How many runs are enough?

Ten is the practical floor: benchstat’s own documentation says to run “at least 10, ideally 20”. Below six samples it can’t compute a confidence interval at all and prints ± ∞, so five is a sanity check and nothing more. Ten gives a reliable local A/B comparison, and 20 is for a CI nightly suite or anything we intend to report. Kalibera and Jones (ISMM 2013) argue for repetition counts based on measured variance, and 20 is a reasonable engineering compromise between rigor and machine time. Fifty or more is for environments we can’t quiet down. It tightens the interval and keeps the same bias, so for the run that lied it would only give a sharper picture of two moods.

As a rule of thumb, resolving a change of Δ% at a CV of c% takes about 16·(c/Δ)² runs per side. At 5% CV, a 5% change needs 16 to 20 runs; at the loaded machine’s 18.88%, more than 200. (This is Lehr’s rule, 16σ²/Δ² per group for about 80% power at α=0.05. It assumes normal noise, and two regimes break it.)

By default -benchtime=1s lets Go calibrate b.N to fill one second per run. For CI baselines I prefer a fixed count such as -benchtime=1000000x (a million iterations per run, no calibration) with -count=20, because a calibration loop that reacts to momentary system load is variance nobody asked for. Pick N so each run lasts milliseconds, not microseconds.

We now know how many runs to collect. One habit can still waste every one of them.

The p-hacking trap

The most common way to get a benchmark result you like is to keep running until you see one. I have done this, and I suspect you have too.

That habit is p-hacking, and it invalidates the p-values benchstat reports, which are calibrated for a pre-specified number of experiments, not an open-ended search. benchstat’s documentation calls rerunning until it reports a change a common statistical error: at the default α of 0.05, about one comparison in twenty shows a difference when nothing changed, whatever your -count. Across 40 benchmarks that is about two false flags per run: re-measure a flag with a pre-committed N, and keep a ledger of the ones that do not reproduce. Select for the runs that clear the threshold and you have selected flukes.

The discipline is to decide N before running, run once, and report what benchstat says, ~ results included. If it matters enough to look again, a larger run is a new experiment: fix its N up front and label it a second look in the report. What we can’t do is rerun after a disappointing p-value and keep whichever result we like. “It only improved on the third attempt” is p-hacking. “It improved when I closed my editor” is environmental confounding, which is equally problematic.

Suppose we resisted all that and the p-value is low. Are we done?

Effect size versus statistical significance

Not yet. A low p-value says the difference is unlikely to be zero. It doesn’t say the difference matters.

With 100 samples on a quiet machine (CV around 1% or less), a 0.3% change can clear p=0.05. On an HTTP handler running at 200 µs that saves 600 nanoseconds per request: statistically real, practically irrelevant. The inverse also holds. With five samples a 15% regression might not reach significance, but 15% on a critical path deserves a look anyway, and a larger pre-committed N can settle it as a new experiment.

We report both numbers. Here is my rule of thumb, not benchstat’s:

Regressions that creep in at 1 to 2% per commit never trip these thresholds; the FOSDEM post’s change-point detection section covers them.

That is the whole kit for the sample: a distribution, a stability number, a run count and a discipline. Time to check our arithmetic.

Try it

To recompute the CV table from the captured results in the companion repository:

1
2
3
4
5

git clone https://github.com/kakkoyun/gopherconuk-26
cd gopherconuk-26/talks/go-benchmarks-lying/demo
git checkout fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb

make cv            # recompute the CV table from results/

make cv prints all three conditions, the two above and the pinned container from part 3. Re-running them is in part 3.

Versions, links and commands checked on 2 October 2026.

Up next

Part 3, Before CI: Can You Trust a Benchmark on Your Own Laptop? measures what container pinning and the Linux controls buy you locally, and where it stops. The run that lied is about to get a third condition. Let’s see if it behaves.

This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 2 and Part 3. The series accompanies my GopherCon UK talk. Part 2 showed what OBI sees. Here we look at how it gets its probes onto Go code, and why USDT keeps coming up. Our function is below. The //go:noinline keeps the compiler from folding it into main, so there is something left to probe.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package main

import "fmt"

//go:noinline
func handle(n int) int {
	if n < 0 {
		return -1
	}
	if n == 0 {
		return 0
	}
	return len(fmt.Sprint(n))
}

func main() {
	fmt.Println(handle(42))
}

One function, three exits

Let’s build it for linux/amd64 with Go 1.27.1 and ask go tool objdump for main.handle. The listing is trimmed to the lines we need (… marks what I cut), and the rest is byte for byte what the tool printed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

TEXT main.handle(SB) ./main.go
  main.go:6		0x49a080		493b6610		CMPQ SP, 0x10(R14)
  main.go:6		0x49a084		765b			JBE 0x49a0e1
  …
  main.go:11		0x49a09c		c3			RET
  …
  main.go:13		0x49a0d2		c3			RET
  …
  main.go:8		0x49a0e0		c3			RET
  …
  main.go:6		0x49a0e6		e8b52afeff		CALL runtime.morestack_noctxt.abi0(SB)
  main.go:6		0x49a0f0		eb8e			JMP main.handle(SB)

The entry is CMPQ SP, 0x10(R14). R14 holds the current goroutine, so this compares the stack pointer with the goroutine’s stack guard. If the stack is too small, JBE jumps to runtime.morestack, which grows the stack (a copy, so it moves) and restarts the function with JMP main.handle. The body then leaves through three RET instructions, one per return in the source.

Watching handle start is easy, since we know where it starts. Watching it finish is where Go makes life hard.

Way one: an entry uprobe and a uprobe on every RET

The textbook way to catch a return is a uretprobe. The kernel rewrites the return address on the stack so the function returns into a trampoline. After poking through the uretprobe implementation in 2018, Austin Clements wrote: “It’s not clear to me how the runtime (or anything that unwinds the stack) could account for this.” Ian Lance Taylor had already explained the other half: Go’s stacks “grow, and therefore move”. A return address the kernel wrote into a stack that then moves points nowhere good.

OBI skips uretprobes. Its code comment says why: “since go linkage is non-standard we can’t use uretprobe”. Instead it decodes the function’s machine code with x86asm and records every RET, then attaches a plain uprobe at each one. That is v0.10.0, the release pinned for the talk and for part 2. For handle it means one entry probe and three return probes, found by reading disassembly. In v0.14.0 the same idea is still there, now batched into one attachment.

The entry probe has its own wart. The v0.14.0 eBPF code (the talk pins v0.10.0) says that when Go grows the stack it restarts the function, so the entry probe fires twice. That is our JMP main.handle, and OBI’s eBPF code has to allow for it.

Then there are the arguments. Reading a struct field from a probe needs its offset, and some offsets move between Go releases. OBI keeps a table of them and refreshes it with a daily workflow. By my count it covers 59 types and 96 fields in offsets.json at v0.10.0 (1,510 lines), and 91 types and 183 fields at v0.14.0 (2,746 lines). One field, net/http.http2ClientConn.fr, sits at offset 208 in Go 1.17.0 and 368 from Go 1.26.0.

That is a lot of work to do before the first trap. What does a trap cost?

What one hit costs

The kernel’s own BPF selftests can tell us. When Jiri Olsa added USDT rows to the uprobe benchmark, his commit message recorded one run on an x86_64 kernel. The benchmark hits a probe from a single producer thread and runs a trivial BPF counter program. The CPU is not stated. In millions of hits per second, with my 1/x in brackets:

A USDT site is a NOP with the same trap put on it, and it costs what a uprobe on a NOP costs. The table has no row for a Go entry, which starts with CMPQ, not a NOP. The ret row is the slowest one, and I don’t know whether a Go RET lands in that regime, because I haven’t measured it.

The disabled case is not free either. PostgreSQL’s docs say that with --enable-dtrace, probe arguments “will be evaluated whenever control passes through the macro, even if no tracing is being done”. libbpf’s own documentation says a USDT “normally has zero overhead, unless it is being traced by some external entity”.

USDT is not zero overhead. If the price is the same trap, the case for it has to rest on something else.

Way two: a named site in the body

A USDT site puts a NOP in the function body and a note in the .note.stapsdt ELF section. Per libbpf’s description, the note carries a provider, a name, the site’s address and, per argument, a size and a location such as -4@%edi. Three things follow.

A tool attaches with less work. It reads the address and the argument locations from the note, and it does not disassemble anything. For handle that would be one site instead of an entry probe plus three RET probes. If the site’s author computes the arguments, the tool also has no struct offsets to track. That last part is my opinion, based on what OBI carries today.

Go’s moving stacks stop mattering. A site in the body is a plain uprobe, so there is no return hijack for a moving stack to break.

It can be a stable contract. The name outlives a refactor that moves the code around, and a tool asking for net_http:server_request_start doesn’t care where the compiler put it. The caveat is that the project has to keep its probes. CPython’s docs are blunt: “No guarantees are made about probe compatibility between versions of CPython”.

Go has none of this natively today. Part 5 covers my poc_usdt fork, which is an experiment. The idea I’d argue for is generic tracepoints that the standard library exposes on purpose. USDT is one way to build them, and it needs more work from me and acceptance from the Go runtime team.

Back to handle, then.

Two probes, one function

With way one, handle gets an entry uprobe and three RET uprobes, found by decoding the function, plus a per-release table of struct offsets kept current by a daily job. With way two, it gets a named site, with the argument locations stated in the binary. Both cost a trap per hit. The difference is who does the work: a tool’s maintainers, again for every Go release, or a runtime, once, if it accepts the idea. Neither helps with carrying trace context from an incoming request to an outgoing call. That is still unsolved.

Versions, links and commands checked on 2 October 2026.

Try it

You can repeat the listing. Save the function above as main.go, then build and count the exits:

1
2

GOOS=linux GOARCH=amd64 go build -trimpath -o handle main.go
go tool objdump -s '^main.handle$' handle | grep -c RET

With Go 1.27.1 I get 3. Addresses may differ on another toolchain.

One function, two ways to watch it. One asks the tool to learn every habit Go has. The other asks Go to introduce itself. 🔭

This is part 2 of 6 in the How to Instrument Go Without Changing a Single Line of Code series, the written companion to my GopherCon UK 2026 talk. Part 1 named three places to intervene: build time, process start and the kernel. We take the kernel route now, with OBI (OpenTelemetry eBPF Instrumentation): what it sees, what it costs, where it fits. The FOSDEM 2026 post has a wider first look at OBI next to the other eBPF options.

Disclosure: I work at Datadog and I’m one of otelc’s maintainers; I don’t contribute to OBI, so read my comparisons with that in mind.

What OBI is

OBI is the direct successor to Grafana Beyla. Grafana Labs donated Beyla to the CNCF OpenTelemetry project in 2025, renamed it OBI, and moved its core development to the opentelemetry-ebpf-instrumentation repository. Beyla lives on as Grafana Labs’ distribution of OBI.

The release pinned for the talk (13 August 2026) is v0.10.0 (2026-06-30), still in Development status. The project itself notes that breaking changes between minor releases are expected while it stays at v0, which is semver for “mind the gap”.

How does OBI get a look at our request without being invited?

How it works

OBI places eBPF uprobes (probes on user-space binaries) in the target binaries and kprobes (the same for kernel code) in the kernel, and the kernel JIT-compiles those eBPF programs to the host architecture (x86-64 or ARM64). For Go, OBI hooks specific library functions, not generic network traffic, which is what gives it library-level span context rather than just raw packets. A probe already sits on the net/http code that will serve GET /orders/42. A uprobe on runtime.newproc1 records which goroutine started which, and OBI walks up to six parents to find the request.

For Go HTTP/gRPC RED metrics (rate, errors, duration), OBI attaches without source changes, recompilation, restarts, or an in-process agent. You deploy it as a DaemonSet (or a sidecar, or a host process), and spans and metrics start flowing to your OTel collector. Our service never gets a say, which is rather the point.

Being watched is one thing. What does the watcher write down?

What it actually instruments in Go

For Go, OBI documents 13 library-level baselines as of v0.10.0. The list includes net/http, golang.org/x/net/http2, gorilla/mux, gin-gonic/gin, google.golang.org/grpc, net/rpc/jsonrpc, database/sql (with the go-sql-driver/mysql and lib/pq drivers), redis/go-redis/v9, Kafka (segmentio/kafka-go and IBM/sarama), and go.mongodb.org/mongo-driver v1 and v2. Full version constraints are in the SUPPORT_MATRIX.md file in the v0.10.0 tag.

Say our service answers with net/http, queries MySQL through database/sql, and calls the next hop over gRPC. All three are on that list. Here is what “zero code changes” buys at each step:

The last row is the library-bound visibility limit of kernel-level instrumentation. The kernel sees function arguments and return values at library boundaries, and can’t synthesize business context that isn’t there. The Limitations section of the official docs says so plainly:

“Use language agents or manual instrumentation when you need custom spans, application-specific attributes, business events, or other in-process telemetry that eBPF-based instrumentation cannot derive automatically.”

For business context you still need manual OTel SDK usage or a compile-time tool like otelc. The view is good. What does it charge for the ticket?

What it requires

The “zero changes to your application” claim moves the bill from the app team to the node. (The docs pages I link are unversioned; I read them on 2026-10-02.)

The kernel comes first: the docs ask for Linux 5.8+, with a RHEL exception of 4.18+ for RHEL 8-family distros with the required eBPF backports. It must also expose BTF (BPF Type Format) information. Most current mainstream distros ship with it; check that /sys/kernel/btf/vmlinux exists. On macOS, eBPF is a Linux party, so bring a VM; See it run: OBI and the eBPF profiler without Kubernetes does exactly that.

Then the capabilities. OBI has no single capability set, because the required access grows with the features you turn on. For plain application observability it needs six Linux capabilities when it runs unprivileged: CAP_BPF, CAP_SYS_PTRACE, CAP_NET_RAW, CAP_CHECKPOINT_RESTORE, CAP_DAC_READ_SEARCH and CAP_PERFMON (plus CAP_SYS_RESOURCE on kernels before 5.11, which the 5.8 floor still allows). The security page explains why each one is needed and has the per-feature breakdown.

More features need more capabilities. CAP_NET_ADMIN is added when context propagation is enabled, which is off by default: a socket-level eBPF program then adds a traceparent header to outgoing HTTP. CAP_SYS_ADMIN is required for Go library-level propagation, where a uprobe writes the header into Go’s own request buffer with bpf_probe_write_user, and it stands in for CAP_PERFMON when kernel.perf_event_paranoid is set high (the AKS and EKS defaults need it). Running with privileged: true in Kubernetes works, but it’s the sledgehammer option. Security teams will scrutinize this list, and that review is the operational cost to factor in.

On Kubernetes, the preferred model for covering many services is a DaemonSet: one OBI pod per node, with hostPID: true so it can see every process and no changes to application pods. The sidecar model (one OBI container per pod, with shareProcessNamespace: true and privileged: true) gives finer control at the cost of efficiency at scale. I would start with the DaemonSet: deploy once, and every service on every node, ours included, gets baseline observability.

What does it cost while running? I have no measurements of OBI v0.10.0 itself; the FOSDEM post’s benchmark is one demo workload. Every hit on an attached uprobe traps into the kernel, so a hot function pays per call (What a uprobe costs, and what USDT buys takes that cost apart). The kernel’s own BPF selftests recorded about 313 ns per hit on an uprobe over a NOP (3.190 M/s, one x86_64 run, CPU not stated, commit 0c4fc6bd6105); a 2023 bpftime paper measured 3,224 ns on older, unstated hardware. The two sources differ tenfold, and I haven’t measured OBI’s. I’m measuring it, with a reproduction kit, for a follow-up post. The rest is upkeep. OBI looks up Go struct-field offsets by Go and library version (offsets.json), and Go 1.26 removing pcHeader.textStart broke its symbol resolution in PIE and cgo binaries until PR #1851.

Is that enough, or just a start?

Where it fits

Back to GET /orders/42. With nobody touching the service, we got RED metrics for the request, library-level spans for its database and gRPC calls, and, with the right opt-ins and privileges, SQL text and trace context across services. We did not get what the on-call engineer asks first: which customer, which cart, which business step.

That split shows where OBI fits: baseline observability across services you don’t own or can’t rebuild right now. Ship the DaemonSet, then decide which services deserve more.

OBI and compile-time tools are complementary. OBI gives breadth without touching CI pipelines. Compile-time tools like otelc give depth (custom spans, business logic, stdlib instrumentation) at the price of a rebuild and, for otelc, Go 1.25+. Part 6 turns the choice into a decision rule.

The caveat worth repeating: the feature set is real, since OBI is the same codebase as Beyla, which reached 1.0 in November 2023. The API stability guarantees, though, are explicitly not there yet, and the v0.10.0 release notes carry a long security-hardening section. Budget for minor release changes, and canary before a fleet-wide rollout.

“Zero code changes”, then? For our request: yes, inside a defined scope, with a defined price list.

Versions, links and commands checked on 2 October 2026.

Up next

Part 3, otelc: zero-touch Go traces at compile time, moves the intervention to build time. The service gets rebuilt; in exchange, we get depth. Same promise, new vantage point. 👀

Where did the allocations go? They were never there. The compiler looked at the benchmark loop, noticed that the return value was unused, and, once the call was inlined, could prove that the body had no observable effect. It removed the body. The loop still ran b.N times. It just ran empty. The fastest code is the code that never runs, and the least useful to benchmark. This is not a compiler bug: the optimizer did its job, and nobody told it we wanted to watch. The transformations that make production code fast make microbenchmarks adversarial.

This is part 1 of 5 in the Why Your Go Benchmarks Are Lying series, the written companion to the GopherCon UK 2026 talk of the same name. The series asks three questions of every benchmark number: did the compiler let it measure real work, is the sample stable, and is the difference large relative to the noise? This part answers the first. We’ll catch the compiler emptying a loop in two ways, find the one column of go test output it cannot fool, trip over the benchmark timer ourselves, and finish with testing.B.Loop, the standard library’s fix for most of it. Part 2 takes on the sample, parts 3 and 4 the machine, and part 5 turns the three questions into a checklist. New to testing.B? The primer Benchmarking Go, quickly covers writing and reading one first. A benchmark bot once flagged a pull request of mine as 6 to 9% slower; hold that thought for part 5.

The prerequisite is Measuring Software Performance: Why Your Benchmarks Are Probably Lying, which covers why hardware noise and statistical method matter in any language. Here we deal with what the Go compiler does before the benchmark ever reaches a CPU. All the code comes from the demo repository (talks/go-benchmarks-lying/demo/, at commit fdce88c).

The compiler is not a neutral observer

Our suspect is dead-code elimination, with inlining as its accomplice. Its cousin, constant folding, does the same damage by another route. We start with the suspect.

Dead-code elimination

Dead-code elimination (DCE) follows a simple rule: if a computation produces a value that nothing ever reads, the compiler can remove it. In a benchmark we typically call a function and throw away the return value. Inlining is the accomplice: the compiler replaces a call to a small function with a copy of its body, nearly always a win in production. In a benchmark, once the body sits inside the loop, the compiler can see the result is unused and delete the lot (a call it cannot inline is kept).

Here is the pair behind our suspect, excerpted from dce_bench_test.go (long comments left out):

1
2
3
4
5
6
7
8
9

func makeBuffer(n int) []byte {
	return make([]byte, n) // heap-escaping allocation
}

func BenchmarkMakeBuffer_DCE(b *testing.B) {
	for range b.N {
		makeBuffer(64) // result discarded → compiler removes the call
	}
}

makeBuffer really does put its buffer on the heap when called on its own. The benchmark discards the result, and because makeBuffer is small enough to inline, the compiler sees a make nobody reads and deletes it, allocation included.

This reproduces on Go 1.25 and later (I tested 1.25.0, 1.26.0 and 1.27.1). On Go 1.24 the inlined make still escapes to the heap, so it allocates even with the result discarded. And the trick needs inlining: put //go:noinline on makeBuffer and the same benchmark reports 64 B/op, 1 allocs/op.

The fix is the sink pattern: assign the result to a local variable inside the loop, then write that local to a package-level variable after the loop. The compiler treats a store to a package-level variable as observable, so it must keep the computation behind it. Same file again:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// sink is the package-level variable that defeats DCE.
var sink []byte

func BenchmarkMakeBuffer_Correct(b *testing.B) {
	var s []byte
	for range b.N {
		s = makeBuffer(64)
	}
	sink = s
}

The two-variable idiom matters: writing to sink inside the loop would add a global write per iteration, which is measurable, while one write after the loop costs almost nothing and still keeps the whole chain alive. Every iteration still runs: with the last buffer stored in sink, the make escapes, so each one allocates. runtime.KeepAlive is callable too, but forces no escape.

A sink keeps a result alive, but it cannot help if the compiler already knows the answer before the program runs.

Constant folding

If every input to an expression is a compile-time constant, the compiler evaluates the expression at compile time and replaces it with a literal. The benchmark then loads a constant, and the compiler already did the work. The bits.OnesCount pair in the same file shows it: the first function feeds in a constant, the second a package-level variable, which the compiler cannot constant-propagate because its value can change at run time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

var sinkInt int

func BenchmarkOnesCount_ConstantFolded(b *testing.B) {
	var s int
	for range b.N {
		s = bits.OnesCount(0b10110) // constant → evaluated at compile time
	}
	sinkInt = s
}

// onesInput breaks the constant chain — compiler cannot prove this is 0b10110.
var onesInput uint = 0b10110

func BenchmarkOnesCount_Correct(b *testing.B) {
	var s int
	for range b.N {
		s = bits.OnesCount(onesInput)
	}
	sinkInt = s
}

On Apple Silicon both versions report similar timings, because bits.OnesCount compiles to a four-instruction NEON sequence that runs near the timer floor regardless. The timing can’t tell them apart, so we ask the assembly instead (make asm-dce runs this). XXX matches nothing, so no benchmark runs, and plain go build would print nothing because it never compiles _test.go files:

1

go test -gcflags='-S' -run XXX -bench XXX . 2>&1 | grep -A14 'OnesCount_ConstantFolded(SB)'

The constant-folded version shows MOVD $3, Rxx: the compiler substituted the literal 3 for the entire bits.OnesCount call. The correct version shows VCNT and VUADDLV, an actual popcount sequence (arm64 names; on amd64, widen the grep to -A30 and look for MOVL $3 against POPCNTQ). Similar timing, entirely different code.

Both tricks leave ns/op looking perfectly believable. If the time column can’t be trusted, what can?

The honest signal

Back to our suspect. 0.34 ns/op sits right next to the measurement floor on Apple Silicon, around 0.25 ns, about one loop iteration per cycle. An empty loop and a blazing-fast function look identical down there. allocs/op has no such floor. An allocation is a discrete event: the testing framework reads runtime.ReadMemStats at the start and end of the run, takes the delta, and divides by b.N. Either a heap allocation happened or it did not.

Let’s run the DCE pair side by side (make bench-dce does it, after a one-time make tools to install benchstat):

1

go test -run XXX -bench=BenchmarkMakeBuffer -benchmem -count=10 .

benchstat summarizes the ten runs as medians. This is one run on one machine: an Apple M4 Max (darwin/arm64) under background load (load average 8 to 11), Go 1.27.1. The ns/op column is noisy and allocs/op is exact, which is the point.

I’m recapturing these on a quiet machine with a reproduction kit, and I’ll update the table when the new numbers are in.

BenchmarkMakeBuffer_DCE reports zero bytes and zero allocations. Not “almost none”: zero. The make([]byte, 64) in the function body never executed, and no timer resolution or clock speed can change that column. BenchmarkMakeBuffer_Correct reports 64 B/op, 1 allocs/op: the allocation happened, so that measurement is trustworthy. Mystery solved: the 0.34 ns/op was an empty loop, and the honest cost is about 15 ns and one allocation. Remember that zero, though. It has one more trick left.

This is why -benchmem belongs in every Go benchmark invocation (go test -bench=. -benchmem ./...). 0 allocs/op for a function you know calls make is a red flag, and a result under 1 ns/op for anything non-trivial is the strongest hint that DCE or constant folding has struck. Two caveats apply. allocs/op is an integer average, so a loop that hits the heap on every fourth iteration rounds down to 0, and B/op gives it away. And a zero can also mean the buffer stayed on the stack, so check -gcflags=-m: inlining call to X means it was inlined; can inline X only means it could be. And for a function that never allocates, allocs/op stays silent; check the assembly instead.

The compiler is not the only thing that can make a benchmark measure the wrong thing. We can do it to ourselves, with the timer.

Timer traps

First, how testing.B keeps time. It runs the benchmark with a growing iteration count until one run fills -benchtime; ns/op is that run’s duration divided by the count. Go compiles ahead of time, so there is no JIT warmup, and each -count line is one such mean, not a distribution.

The testing.B timer starts when the benchmark function is called, so everything that runs before the loop is measured unless we reset it.

b.ResetTimer() zeros the elapsed time and the allocation counters, so we call it after one-time setup, just before the loop. It does not stop the timer: a running timer keeps running, from zero.

Per-iteration setup: StopTimer and StartTimer

When each iteration needs its own setup, we bracket the setup with b.StopTimer() and b.StartTimer(), and the order matters. This one is from timer_bench_test.go, where buildFixture, fixtureSize, processString and sinkStr are helpers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

func BenchmarkProcess_PerIterSetup_Correct(b *testing.B) {
	var s string
	for range b.N {
		b.StopTimer()
		input := buildFixture(fixtureSize)
		b.StartTimer() // ← timer restarts; only processString is measured
		s = processString(input)
	}
	sinkStr = s
}

The timer stops before the fixture is built and restarts before the work, so only processString is timed. Call StartTimer after the work instead, as BenchmarkProcess_TimerOrder_BUG in the same file does, and the timer measures fixture construction while processString runs with the timer off. The reported ns/op is then the cost of the wrong thing.

The benchmark that reports nonsense

Worse: StopTimer with no StartTimer at all. The testing framework decides how long to run a benchmark by accumulating timed duration until it reaches the target time (default 1 second). If the timer is stopped and never restarted, the accumulated duration holds only the time up to the first StopTimer. The framework keeps growing b.N until it hits its cap of 1e9. For a trivial body with a single StopTimer, it reaches that cap in about 20 seconds and prints something like 0.0000007 ns/op. Another number that looks suspiciously good.

The demo repository describes this case rather than demonstrating it. The comment in timer_bench_test.go reads “Don’t run that live.” We found out the direct way. If one of your own benchmarks seems to hang, look for a missing b.StartTimer(), or a stop/start pair left at the default -benchtime. Ctrl-C, fix it, run again. (Each StopTimer/StartTimer call runs runtime.ReadMemStats, which stops the world, which is why the demo pins these benchmarks to -benchtime=50000x, a fixed count with no ramp.)

Empty loops, forgotten resets, timers left off: all mistakes the testing package could make impossible. Since Go 1.24, it covers most of them.

testing.B.Loop in Go 1.24 removes most of this

Austin Clements proposed testing.B.Loop in Go issue #61515 because the b.N pattern has failure modes that are easy to hit and hard to detect statically. It shipped in Go 1.24. Here is the demo’s hashing benchmark in the new form, from bloop_bench_test.go (payload is a package-level []byte fixture):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

func BenchmarkHash_BLoop(b *testing.B) {
	// Setup: excluded from timing automatically.
	data := make([]byte, 1024)
	copy(data, payload)

	var s [32]byte
	for b.Loop() { // ← each call to Loop() is one measured iteration
		s = sha256.Sum256(data)
	}
	_ = s
}

Notice what is missing: no b.ResetTimer(). The first call to Loop resets the timer and the call that returns false stops it, so setup and cleanup are not measured. The benchmark function is called exactly once per -count value, so expensive setup does not re-execute while the framework ramps b.N up. And the compiler recognizes loops whose condition is syntactically b.Loop() and keeps the arguments, results and assigned variables of calls in the loop body alive (it wraps them in a runtime.KeepAlive), so inlining can no longer leave a dead body behind. Go 1.24 and 1.25 did this by refusing to inline into the body, which could add heap allocations that production code would not have. Go 1.26 keeps inlining and keeps the values alive instead.

The DCE protection has limits. It had a bug in Go 1.26.0 to 1.26.2, where assigning a call’s result to _ inside the loop still let the compiler drop the body (#77654, fixed in Go 1.26.3 and 1.27). It applies only when the loop condition is written exactly as b.Loop(): assigning the method to a variable first (loop := b.Loop; for loop()) does not trigger the compiler transformation.

b.Loop protects the call, not its inputs: bits.OnesCount(0b10110) in a for b.Loop() body still folds to MOVL $3 on amd64 (Go 1.27.1), so keep a non-constant input. _ = s after the loop only silences “declared and not used”; -benchmem remains the check.

A StartTimer that never comes is a fatal error with b.Loop (“B.Loop called with timer stopped”), not a hang.

For new benchmarks, prefer b.Loop. Migrating an old one is mechanical: swap for range b.N for for b.Loop() and drop any b.ResetTimer() that only excluded setup. Code that reads b.N inside the loop needs a second look: b.N is 0 until the loop ends.

Now back to our number one last time. 0.34 ns/op started as an empty loop, and the sink turned it into 15 ns and one allocation. On Go 1.26 and later, for b.Loop() { makeBuffer(64) } reports 0 B/op, 0 allocs/op for the same buffer, and this time the zero is honest: the body runs, but the inlined buffer no longer escapes and lives on the stack, as it would in production code. Same digit, opposite meaning. Which makeBuffer number do we report? The b.Loop one. The sink version measures a heap escape that production code may not have, and b.Loop on Go 1.26 or later measures the function as the compiler would really treat it. If an allocation count surprises you, -gcflags=-m tells you which case you are in.

Enough reading. Let’s run it.

Try it

Everything above runs from the demo repository. The module declares go 1.26.5, so Go fetches that toolchain if yours is older. The assembly step shows arm64 instructions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

git clone https://github.com/kakkoyun/gopherconuk-26
cd gopherconuk-26
git checkout fdce88cc0ce129b7d2edfb1a20d02fe647c17eeb
cd talks/go-benchmarks-lying/demo

# DCE and the sink pattern: compare allocs/op
go test -run XXX -bench='BenchmarkMakeBuffer' -benchmem -count=10 .

# Constant folding: MOVD $3 against VCNT and VUADDLV
go test -gcflags='-S' -run XXX -bench XXX . 2>&1 | grep -A14 'OnesCount_ConstantFolded(SB)'

# Timer ordering (fixed iterations, as make bench-timer does)
go test -run XXX -bench='BenchmarkProcess' -benchmem -count=3 -benchtime=50000x .

BenchmarkMakeBuffer_DCE should report 0 allocs/op, and BenchmarkMakeBuffer_Correct should report 1. Expect the ns/op column to wobble with whatever else your machine is doing. That wobble is the next part of the story.

Versions, links and commands checked on 2 October 2026.

Up next

Part 2, A Single Benchmark Number Is a Lie, assumes the benchmark does real work and asks what one ns/op number is worth: how to read the distribution behind it, how to compare commits with benchstat, and when a difference is real. The compiler is out of the way. The statistics are not. 📊

1

java -javaagent:agent.jar -jar app.jar

The JVM offers each class’s bytes to the agent’s ClassFileTransformer before it defines the class, so the agent can rewrite a method on its way in. Python lets you swap a method at runtime, which is the monkey-patching of the title. That flag is Java’s agent slot, and Python doesn’t even need one. Go has no such slot, and that follows from how Go was designed.

This is part 1 of 6 in the How to Instrument Go Without Changing a Single Line of Code series, the written companion to my GopherCon UK 2026 talk. We’ll hunt for that missing slot the whole way. First we’ll see why it isn’t there, then visit the three places people stand instead (build time, process start and the kernel) and watch how far each bends to fake it. Parts 2 to 4 take the tools one at a time, part 5 asks what Go itself might add, and part 6 ends with a runbook an agent can follow.

Disclosure: I work at Datadog, which maintains Orchestrion and dd-trace-go, and I’m one of otelc’s maintainers, so read my comparisons with that in mind.

The structural problem

Where would an agent hook in? Go compiles to native machine code, and go build gives you a self-contained binary: no bytecode, no classloader, nothing between your code and the CPU. With no bytecode there is nothing to rewrite at load time, and with no classloader there is no moment between “file on disk” and “code executing” to grab. LD_PRELOAD, the classic way to push a shared library into any process, needs the dynamic linker, and Go links pure-Go programs statically by default (more below). And Go has no general runtime hook API. The internal exithook covers only program termination.

What about go:linkname, which lets a package reach unexported runtime symbols? Tracers use it, but listen to how the runtime talks about it. Above gopark in proc.go:

“gopark should be an internal detail, but widely used packages access it using linkname. Notable members of the hall of shame include…”

The Go team added those annotations reluctantly. The comment says “Do not remove or change the type signature” and points at golang/go#67401, which locked down future linkname uses. That is an accommodation, not a hook. A hall of shame is an odd place to look for a feature.

No slot, then. If nobody will hand us one, where do people stand instead?

Three places to intervene

At build time, we intercept the build instead of the program. Go’s -toolexec flag puts a wrapper in front of every go tool compile call. The wrapper parses each source file into a syntax tree, injects instrumentation, and hands the modified source to the real compiler, which thinks you wrote it yourself. The wrapper itself adds no runtime overhead and needs no kernel privilege, but you have to rebuild. I wrote a guest post that builds this mechanism up from a stopwatch.

At process start, we load a library before main, with no rebuild. That’s the closest cousin of -javaagent, and for Go the hardest.

In the kernel, Linux’s eBPF subsystem can attach probes to function entry and exit points in a running binary without modifying it. A privileged agent on the node attaches and the Go binary never knows. The price is visibility: you see function boundaries (arguments, return values, call counts), and each tool brings its own kernel and privilege requirements. Part 2 has OBI’s.

None of them wins outright, so the series treats them as layers. The middle one is the agent slot in its purest form, so we try it first.

Process start: the injector

An injector is a shared library that the dynamic loader maps in before main, so it acts before any of your code does. On a host, Datadog’s Single Step Instrumentation arms itself through /etc/ld.so.preload, and so does the OpenTelemetry injector. The OTel one is a smaller trick than it sounds: it sets variables like JAVA_TOOL_OPTIONS and NODE_OPTIONS, and the runtime loads its own agent. It types our -javaagent flag for us. It covers Java, Node.js, .NET, Ruby, and Python (disabled by default), and its design notes say statically linked executables “will not be affected by LD_PRELOAD at all.” Go has no agent slot for such a variable to point at.

Why can’t Go take LD_PRELOAD? A pure-Go build is a static ELF with no interpreter (the dynamic loader a binary names in its headers), so the loader never runs. With cgo disabled there is no escape hatch, because -linkmode=external, which hands the final link to the system C linker, needs cgo. (cgo binaries, including those using the default net resolver on Linux, are dynamically linked and take LD_PRELOAD on glibc.) golang/go#28909 is still open as of October 2026, and Ian Lance Taylor was “perfectly comfortable in saying that anybody who wants to use LD_PRELOAD must force the use of external linking.” External linking is supported, but forcing it just for LD_PRELOAD is a workaround with sharp edges.

To check yours, run file ./myapp. For a small net/http program on Go 1.27.1 (linux/arm64), CGO_ENABLED=0 printed statically linked and CGO_ENABLED=1 printed dynamically linked, interpreter /lib/ld-linux-aarch64.so.1. Even the dynamic one offers no slot: a preloaded library interposes on dynamic symbols such as libc’s read, and readelf --dyn-syms shows no net/http function to wrap.

Datadog has a Go injector hook in development, which I’ll show on a slide at GopherCon UK. It isn’t released, so I’m not going into how it works yet.

For a pure-Go binary the slot stays empty. That leaves build time, and even that route has to fake something Java ships with: a place to keep the current request’s context.

The goroutine problem

Distributed tracing is where it bites. A request often hops goroutines (Go’s lightweight threads), and its trace context has to hop with it. Go has no goroutine-local storage, so after go func() { ... }() there is no built-in way to inherit arbitrary values from the parent (pprof labels aside, which are string pairs for profiles). The usual answer is to pass a context.Context down by hand, which is the code change this series is trying to avoid. Code injected into a library function often has no ctx in scope anyway, so the tracer needs somewhere else to keep the context.

Orchestrion, Datadog’s -toolexec tool from part 3, answers with something striking: give the tracer somewhere to keep it. dd-trace-go v2.0.0 ships an aspect that injects a synthetic field into the runtime’s internal g struct, the one that represents each goroutine. The start of it, abridged:

1
2
3
4
5
6
7
8

aspects:
  - id: __dd_gls_v2
    join-point:
      struct-definition: runtime.g
    advice:
      - add-struct-field:
          name: __dd_gls_v2
          type: any

The join point says where to cut and the advice says what to add: a field __dd_gls_v2 any on runtime.g. Orchestrion rewrites the runtime source before the compiler sees it, so every goroutine gets one. The same aspect injects a getter and a setter as go:linknamed function variables. Here is that template, re-indented the way gofmt would print it:

1
2
3
4
5
6
7
8
9

//go:linkname __dd_orchestrion_gls_get __dd_orchestrion_gls_get.V2
var __dd_orchestrion_gls_get = func() any {
	return getg().m.curg.__dd_gls_v2
}

//go:linkname __dd_orchestrion_gls_set __dd_orchestrion_gls_set.V2
var __dd_orchestrion_gls_set = func(val any) {
	getg().m.curg.__dd_gls_v2 = val
}

getg() asks the runtime for the current goroutine, and .m.curg goes through the m (the OS thread running it) to make sure it’s yours and not a system one, per the runtime’s HACKING.md. A second aspect sets the field to nil at the top of goexit1, so finished goroutines don’t leak their context.

The field doesn’t follow a go statement by itself: dd-trace-go still carries a TODO: handle cross-goroutine context values in its context stack. otelc goes further. It adds its own fields to g (otel_trace_context and otel_baggage_container) and copies them in newproc1 when a goroutine starts, so trace context and baggage follow go statements. That is still not the request’s context.Context: its values, deadline and cancellation stay behind, and carrying trace context from an incoming connection to an outgoing one is unsolved; Context across goroutines and connections shows where it breaks in a small program. Part 5 comes back to it.

Back to the agent slot. Other runtimes have a designed hook point for this: thread-local storage in Java, contextvars in Python, AsyncLocalStorage in Node.js. Go’s answer is to patch the runtime’s struct layout at build time and hope the runtime keeps allowing it. Impressive engineering, and a fair measure of how far you bend to fake what Java ships with.

Who is standing on those three spots?

The tools in this series

Four tools. OBI (OpenTelemetry eBPF Instrumentation, which started life as Grafana Beyla) is the kernel path with no rebuild: part 2. otelc (OpenTelemetry Go compile-time instrumentation) takes the -toolexec path and needs Go 1.25 or newer, and Orchestrion uses the same mechanism with the goroutine hack above: part 3. The OpenTelemetry eBPF profiler is the kernel path for profiles: part 4.

None of them is clean. They’re workarounds for a runtime built for simplicity, not observability, but they work. The kernel route needs no rebuild, so that’s where we go next.

Versions, links and commands checked on 2 October 2026.

Up next

Part 2, OBI: eBPF auto-instrumentation for Go in production, takes the kernel route: what OBI covers, what it needs, and what it costs. Java gets a flag. Go gets a kernel probe and some paperwork.

1

BenchmarkMakeBuffer-16	99632604	        18.53 ns/op	      64 B/op	       1 allocs/op

This is a primer before part 1 of Why Your Go Benchmarks Are Lying, the written companion to my GopherCon UK talk, Why Your Go Benchmarks Are Lying. The series argues about numbers like the one above, so it helps to know how they are made first. If you have run go test -bench before, skim.

Write it, run it

Our benchmark allocates a 64-byte buffer and keeps it. It’s the one the series uses as its running suspect, and it lives in a scratch module (go mod init tiny, one file, tiny_test.go):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

package tiny

import "testing"

func makeBuffer(n int) []byte {
	return make([]byte, n)
}

var sink []byte

func BenchmarkMakeBuffer(b *testing.B) {
	var s []byte
	for range b.N {
		s = makeBuffer(64)
	}
	sink = s
}

A benchmark is a function named BenchmarkXxx that takes a *testing.B and does its work b.N times. The package-level sink is there so the compiler can’t throw the work away; part 1 is about what happens without it. Let’s run it:

1
2
3
4
5
6
7
8

$ go test -run XXX -bench . -benchmem
goos: darwin
goarch: arm64
pkg: tiny
cpu: Apple M4 Max
BenchmarkMakeBuffer-16	99632604	        18.53 ns/op	      64 B/op	       1 allocs/op
PASS
ok  	tiny	2.105s

Every number in this post is from one run on one machine: an Apple M4 Max, Go 1.27.1, with a load average near 28 when I checked. A busy laptop is a fine teacher here, as you’ll see. I’m recapturing these outputs on a quiet machine with a reproduction kit, and I’ll update them when the new numbers are in.

Now the flags. -bench . selects benchmarks by regular expression, and by default none run. -run XXX does the same for tests; nothing is named XXX, so no test runs and the timing isn’t muddied by them. -benchmem adds the last two columns: bytes and heap allocations per iteration. The first number is how many times the loop ran, and the second, ns/op, is the time per iteration. We’ll start with the first.

Where does 99,632,604 come from?

Nobody chose it. The testing package runs the function once with b.N = 1, then predicts how many iterations would fill the benchmark time, adds 20% headroom, grows by at most 100× per step and stops at a billion. To watch it happen, I temporarily added a fmt.Fprintln(os.Stderr, "b.N =", b.N) to a copy of the benchmark:

1
2
3
4
5
6
7
8

$ go test -run XXX -bench Ramp 2>&1 | grep -E 'b\.N|ns/op'
b.N = 1
BenchmarkRamp-16	b.N = 100
b.N = 10000
b.N = 1000000
b.N = 28893382
b.N = 47900115
47900115	        22.76 ns/op

The function ran six times, and the printed line comes from the last run alone. The result is its total time divided by b.N, which is why ns/op is a mean. The ramp isn’t a fixed ladder: on the earlier run the last b.N was 99,632,604. The target is time, not a count, and -benchtime sets it. The default is 1s; a value like 100x means a fixed number of iterations.

Every -count repetition is its own ramp and its own mean:

1
2
3
4

$ go test -run XXX -bench . -benchmem -count=3
BenchmarkMakeBuffer-16	96041296	        23.08 ns/op	      64 B/op	       1 allocs/op
BenchmarkMakeBuffer-16	61289114	        23.37 ns/op	      64 B/op	       1 allocs/op
BenchmarkMakeBuffer-16	100000000	        17.26 ns/op	      64 B/op	       1 allocs/op

Same binary, same code, and the slowest sample is about 35% slower than the fastest. Each line is one mean, so each is one sample. Part 2 is about what to do with a handful of them.

Go is compiled ahead of time, so there is no JIT warmup for the ramp to wait out. One iteration still isn’t representative. With -benchtime=1x the same benchmark reported 2500, 1667 and 2000 ns/op. I haven’t chased down why (cold caches, the first allocation and timer granularity are my suspects).

That leaves the suffix. The -16 is GOMAXPROCS, and the suffix is dropped when the value is 1. With -cpu 1,4 we get both forms:

1
2
3

$ go test -run XXX -bench 'MakeBuffer$' -cpu 1,4
BenchmarkMakeBuffer     	83583531	        18.06 ns/op
BenchmarkMakeBuffer-4   	100000000	        20.72 ns/op

A missing suffix in someone else’s output means they ran with GOMAXPROCS=1. We can read the whole line now. What we can’t tell yet is whether the loop did what we think it did.

The newer loop, and a suspicious zero

Go 1.24 added testing.B.Loop as a replacement for the b.N loop. We add a second benchmark that uses it, with no sink:

1
2
3
4
5

func BenchmarkMakeBufferLoop(b *testing.B) {
	for b.Loop() {
		makeBuffer(64)
	}
}

Loop resets the timer on its first call and stops it when it returns false, so setup before it isn’t measured. The benchmark function runs once per -count, and the compiler keeps the arguments and results of calls in the body alive, so it can’t optimize away the whole loop body. (It has had bugs; part 1 lists them.) Go 1.26 changed how it keeps them alive: inlining in the body is no longer blocked. Here are both benchmarks together:

1
2
3
4
5
6
7

$ go test -run XXX -bench . -benchmem -count=3
BenchmarkMakeBuffer-16    	92055165	        21.90 ns/op	      64 B/op	       1 allocs/op
BenchmarkMakeBuffer-16    	88833778	        19.98 ns/op	      64 B/op	       1 allocs/op
BenchmarkMakeBuffer-16    	100000000	        23.30 ns/op	      64 B/op	       1 allocs/op
BenchmarkMakeBufferLoop-16	494076908	         3.145 ns/op	       0 B/op	       0 allocs/op
BenchmarkMakeBufferLoop-16	548045512	         2.975 ns/op	       0 B/op	       0 allocs/op
BenchmarkMakeBufferLoop-16	562811305	         2.071 ns/op	       0 B/op	       0 allocs/op

Same function, yet the loop version reports about 3 ns against about 20, and zero allocations. The series spends a post on numbers like that. Let’s ask the compiler directly.

Ask the compiler

-gcflags=-m prints the compiler’s inlining and escape-analysis decisions. The code lives in a _test.go file, so it has to be go test, and -run XXX -bench XXX compiles everything without running anything. I kept the two lines about our buffer:

1
2
3

$ go test -gcflags=-m -run XXX -bench XXX . 2>&1 | grep 'make(\[\]byte, 64)'
./tiny_test.go:14:17: make([]byte, 64) escapes to heap
./tiny_test.go:21:13: make([]byte, 64) does not escape

Line 14 is the b.N benchmark: the buffer goes to the heap, hence 1 allocs/op. Line 21 is the b.Loop one: the buffer stays on the stack, so the zero is honest. -S goes one level lower and prints the assembly. The b.N benchmark calls the allocator (I trimmed the file path; the same grep on BenchmarkMakeBufferLoop STEXT prints nothing):

1
2

$ go test -gcflags=-S -run XXX -bench XXX . 2>&1 | grep -A40 'tiny.BenchmarkMakeBuffer STEXT' | grep 'CALL.*makeslice'
0x0040 00064 (tiny_test.go:6)	CALL	runtime.makeslice(SB)

That is enough compiler for now; part 1 reads these outputs in anger.

Where the profiler fits

A benchmark tells you how long; a profile tells you where. Our buffer is a poor subject, since in my run its profile was dominated by runtime frames, so let’s use the series demo’s hashing benchmark, BenchmarkHash_BLoop, from the companion repository. We add -cpuprofile to the run and read it with go tool pprof -top, keeping three rows:

1
2
3
4
5
6

$ go test -run XXX -bench BenchmarkHash_BLoop -benchmem -benchtime=2s -cpuprofile hash.prof .
$ go tool pprof -top hash.prof
      flat  flat%   sum%        cum   cum%
     1.95s 92.86% 92.86%      1.95s 92.86%  crypto/internal/fips140/sha256.blockSHA2
     0.09s  4.29% 97.14%      0.09s  4.29%  runtime.pthread_cond_signal
     0.02s  0.95% 98.10%      1.97s 93.81%  crypto/internal/fips140/sha256.block

The benchmark spent 92.86% of its CPU in blockSHA2, the SHA-256 block function, which is what a hash benchmark should be doing. If the profile of your own benchmark is dominated by runtime and testing frames instead, you may be timing the test framework.

Versions, links and commands checked on 2 October 2026.

Try it

b.Loop needs Go 1.24, but the zero allocs/op result needs Go 1.26. On 1.24 and 1.25 the loop version shows 1 alloc/op too, which is the inlining change from earlier. Make a directory, run go mod init tiny, put the two benchmarks in tiny_test.go and run go test -run XXX -bench . -benchmem -count=3. Your numbers will differ from mine; on Go 1.26 or later your allocs/op column should not. Then add -benchtime=1x, -cpu 1,4 and -gcflags=-m one at a time and watch which column moves.

We wrote a dozen lines of Go, got a line of output, and know where every field came from: the ramp, the mean, the sample, the suffix and the allocator. What we can’t yet say is whether a given ns/op deserves trust. That’s the question the series asks, and part 1 starts by catching the compiler emptying a loop. 🔎

The frame I’m borrowing comes from a piece my colleagues at Bloomberg published with the CNCF in March: Sustaining OpenTelemetry: Moving from Dependency Management to Stewardship (also on Bloomberg’s company stories). The phrase has stuck with me. It names something I’ve been trying to articulate for years, and gives me a concrete vocabulary for talking about what mentorship is for inside a project the size of OpenTelemetry.

From dependency management to stewardship

Most companies use open source the same way they use electricity. ¹

Stewardship is the opposite stance. You don’t just consume the project; you take responsibility for the version of it that exists ten years from now. You pay for that version not just with code but with time spent growing the people who will maintain it. ²

The Bloomberg piece is making this case at the level of an organization. I want to make it at the level of a single SIG meeting.

A glimpse from the SIG room

In my field notes from OTel Unplugged EU 2026, the maintainer crisis came up in at least three rooms:

“Not enough maintainers, too many PRs, codeowners disappearing. The JavaScript SIG Special Interest Group — OpenTelemetry’s organizational unit, roughly one per language or component (e.g. SIG Go, SIG JavaScript, SIG Collector). has an automated script to move inactive maintainers to emeritus after three months. […] Some SIGs have tried a buddy/mentor system for onboarding new contributors — it helps, but it doesn’t scale across all SIGs when the existing maintainers barely have time to review PRs.”

And the line that stuck with me from those rooms:

“Maintainership is privilege AND responsibility.”