Engineering · · 8 min

otelc: zero-touch Go traces at compile time

We’re going to instrument a Go program without touching its source, and the whole change fits in one build command. Here’s the before and after: 1 2 go build -o myapp . otelc go build -o myapp . Same flags, same package, one extra word in front. The binary that comes out has OpenTelemetry instrumentation compiled in for the libraries otelc knows about, net/http and gRPC among them, and nobody opened a handler file. ...

July 16, 2026 · 8 min · 1632 words · Kemal Akkoyun
Engineering · · 11 min

A Single Benchmark Number Is a Lie

Here is one benchmark that gave us two different answers for the same code. We’re going to take that run apart: see what benchstat can and can’t say about it, add the one number it doesn’t report, and decide how many runs it takes before a result earns any trust. Let’s call it the run that lied. The benchmark is BenchmarkMakeBuffer_Correct, run on a machine with sixteen CPU-bound background processes competing for the same cores. Here are two of its twenty runs (-16 is the GOMAXPROCS suffix), with the B/op and allocs/op columns dropped: ...

July 14, 2026 · 11 min · 2140 words · Kemal Akkoyun
Engineering · · 7 min

What a uprobe costs, and what USDT buys

We’re going to put a probe on one small Go function, twice. First the way OBI (OpenTelemetry eBPF Instrumentation) does it today, then the way a USDT site would. On the way we’ll price a single probe hit, and find that USDT’s pitch is not a discount. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 2 and Part 3. The series accompanies my GopherCon UK talk. Part 2 showed what OBI sees. Here we look at how it gets its probes onto Go code, and why USDT keeps coming up. Our function is below. The //go:noinline keeps the compiler from folding it into main, so there is something left to probe. ...

July 11, 2026 · 7 min · 1340 words · Kemal Akkoyun
Engineering · · 7 min

OBI: eBPF auto-instrumentation for Go in production

Take one HTTP request, GET /orders/42, headed for a Go service nobody is going to rebuild this quarter. We want a span for it, another for its database call, and the trace context handed on to the next service, all without touching the service. “Zero code changes” travels far in conference talks and vendor docs. Today we watch that one request from the kernel and find out where the phrase holds. ...

July 9, 2026 · 7 min · 1418 words · Kemal Akkoyun
Engineering · · 13 min

The Go Benchmark That Measured Nothing: Compiler Honesty in testing.B

We’re going to chase down one benchmark result that looks suspiciously good. The function under test calls make([]byte, 64) every time it runs, and the benchmark reports 0.34 ns/op, 0 B/op, 0 allocs/op. That is almost three billion heap allocations per second on a laptop, and the allocator counted none of them. (It comes from a real run, more on that below.) Keep it in your pocket; we’ll follow it the whole way. ...

July 7, 2026 · 13 min · 2713 words · Kemal Akkoyun