Engineering

Note This post was originally published on the OpenTelemetry blog on 2026-07-16. I wrote it for the OpenTelemetry community; this is a mirror on my personal blog with some added commentary. Why this milestone matters to me I have been involved with the Go Compile-Time Instrumentation SIG since it kicked off at the start of 2025, when Alibaba and Datadog decided to pool their efforts into a single vendor-neutral project. Watching it reach v1 last week felt genuinely satisfying — not because “v1” is a magic number, but because it marks the point where we stopped calling it experimental and started calling it something people should actually run in their builds. ...

Every benchmark number starts life on somebody’s laptop. We write the function, run it, like what we see and open the pull request, long before CI or a pinned runner gets a say. That makes it the least controlled measurement in the pipeline, and the one we act on. If it lies, everything downstream inherits the lie. Sending a noisy benchmark to CI doesn’t fix the noise. It industrialises it. ...

We’re going to instrument a Go program without touching its source, and the whole change fits in one build command. Here’s the before and after: 1 2 go build -o myapp . otelc go build -o myapp . Same flags, same package, one extra word in front. The binary that comes out has OpenTelemetry instrumentation compiled in for the libraries otelc knows about, net/http and gRPC among them, and nobody opened a handler file. ...

Here is one benchmark that gave us two different answers for the same code. We’re going to take that run apart: see what benchstat can and can’t say about it, add the one number it doesn’t report, and decide how many runs it takes before a result earns any trust. Let’s call it the run that lied. The benchmark is BenchmarkMakeBuffer_Correct, run on a machine with sixteen CPU-bound background processes competing for the same cores. Here are two of its twenty runs (-16 is the GOMAXPROCS suffix), with the B/op and allocs/op columns dropped: ...

We’re going to put a probe on one small Go function, twice. First the way OBI (OpenTelemetry eBPF Instrumentation) does it today, then the way a USDT site would. On the way we’ll price a single probe hit, and find that USDT’s pitch is not a discount. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 2 and Part 3. The series accompanies my GopherCon UK talk. Part 2 showed what OBI sees. Here we look at how it gets its probes onto Go code, and why USDT keeps coming up. Our function is below. The //go:noinline keeps the compiler from folding it into main, so there is something left to probe. ...