Engineering · · 11 min

A Single Benchmark Number Is a Lie

Here is one benchmark that gave us two different answers for the same code. We’re going to take that run apart: see what benchstat can and can’t say about it, add the one number it doesn’t report, and decide how many runs it takes before a result earns any trust. Let’s call it the run that lied. The benchmark is BenchmarkMakeBuffer_Correct, run on a machine with sixteen CPU-bound background processes competing for the same cores. Here are two of its twenty runs (-16 is the GOMAXPROCS suffix), with the B/op and allocs/op columns dropped: ...

July 14, 2026 · 11 min · 2140 words · Kemal Akkoyun
Engineering · · 7 min

What a uprobe costs, and what USDT buys

We’re going to put a probe on one small Go function, twice. First the way OBI (OpenTelemetry eBPF Instrumentation) does it today, then the way a USDT site would. On the way we’ll price a single probe hit, and find that USDT’s pitch is not a discount. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 2 and Part 3. The series accompanies my GopherCon UK talk. Part 2 showed what OBI sees. Here we look at how it gets its probes onto Go code, and why USDT keeps coming up. Our function is below. The //go:noinline keeps the compiler from folding it into main, so there is something left to probe. ...

July 11, 2026 · 7 min · 1340 words · Kemal Akkoyun
Engineering · · 7 min

OBI: eBPF auto-instrumentation for Go in production

Take one HTTP request, GET /orders/42, headed for a Go service nobody is going to rebuild this quarter. We want a span for it, another for its database call, and the trace context handed on to the next service, all without touching the service. “Zero code changes” travels far in conference talks and vendor docs. Today we watch that one request from the kernel and find out where the phrase holds. ...

July 9, 2026 · 7 min · 1418 words · Kemal Akkoyun
Engineering · · 13 min

The Go Benchmark That Measured Nothing: Compiler Honesty in testing.B

We’re going to chase down one benchmark result that looks suspiciously good. The function under test calls make([]byte, 64) every time it runs, and the benchmark reports 0.34 ns/op, 0 B/op, 0 allocs/op. That is almost three billion heap allocations per second on a laptop, and the allocator counted none of them. (It comes from a real run, more on that below.) Keep it in your pocket; we’ll follow it the whole way. ...

July 7, 2026 · 13 min · 2713 words · Kemal Akkoyun
Engineering · · 8 min

Why Go can't be monkey-patched (and what people do about it)

Start a Java service with one extra flag and it gets traced, with no source change and no rebuild: 1 java -javaagent:agent.jar -jar app.jar The JVM offers each class’s bytes to the agent’s ClassFileTransformer before it defines the class, so the agent can rewrite a method on its way in. Python lets you swap a method at runtime, which is the monkey-patching of the title. That flag is Java’s agent slot, and Python doesn’t even need one. Go has no such slot, and that follows from how Go was designed. ...

July 2, 2026 · 8 min · 1595 words · Kemal Akkoyun