Engineering · · 11 min

A PR gate that actually fails

Part 4 ended with a promise: the PR gate blocks only when benchstat says the delta is real and bigger than a floor we chose. A fair reader question is how. benchstat prints a table, and I checked that it exits 0 even when the table says +199.00% (two made-up result files, the pinned version, Go 1.27.1). Who, then, turns that table into a red check? We’ll build that job together. We’ll write one deliberately slow commit, let the job judge it, revert the commit, and let the job judge the revert. Two runs, two verdicts, both on a GitHub-hosted runner: ...

July 31, 2026 · 11 min · 2277 words · Kemal Akkoyun
Engineering · · 9 min

Go runtime futures: flight recording, USDT, and the instrumentation hook problem

Every route in this series goes around the Go runtime: rewrite the source at build time, inject something at process start, or watch from the kernel, because Go gives us no agent API to plug into. This time we ask the runtime what it could hand us. The honest summary for 2026: one long-awaited feature shipped, two open proposals document gaps that have caused real production problems for years, and two experiments of mine are aspirations, not proposals. ...

July 30, 2026 · 9 min · 1826 words · Kemal Akkoyun
Engineering · · 14 min

Benchmark CI That Doesn't Lie

Say we put a benchmark gate on every pull request: if performance regresses, the check fails. After a week, something strange shows up. A PR that does nothing performance-sensitive fails the gate. Another PR, one that rewrites a hot path, sails through with a green check. Nobody is imagining it. Both outcomes are correct given the data, and the data is just wrong. A shared CI runner is a multi-tenant machine, so our benchmark lands on a host that also runs other teams’ compile jobs, Docker builds and test suites. A real 10% regression can vanish into that noise, and a phantom one can appear from nowhere. The gate isn’t measuring our code. It’s measuring the lottery of what happens to be running next door. 🎰 ...

July 28, 2026 · 14 min · 2874 words · Kemal Akkoyun
Engineering · · 7 min

Context across goroutines and connections

We’re going to follow one request, GET /orders/42, and ask what “propagation” means at each hop. Three different jobs hide behind that word, the tools in this series do different ones, and the request’s own data falls into the gaps between them. The cheapest way to see a gap is a short Go program. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 4 and Part 5. The series accompanies my GopherCon UK talk. Part 1 showed tools hacking a field into g. Here we look at what that field holds, what it can’t, and who does the rest. ...

July 27, 2026 · 7 min · 1409 words · Kemal Akkoyun
Engineering · · 8 min

See it run: OBI and the eBPF profiler without Kubernetes

Parts 2 and 4 of this series describe what OBI (OpenTelemetry eBPF Instrumentation) and the OpenTelemetry eBPF profiler can do. Here we make them do it with Docker Compose and no Kubernetes: one Go server nobody touches, OBI watching its requests, the profiler watching its CPU. We’ll follow one request, GET /orders/42, and see it twice, first as a trace and then as CPU samples. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 4 and Part 5, and the hands-on side of my GopherCon UK talk. It all ran on one machine: a Colima VM (Ubuntu 24.04, kernel 6.8.0, linux/arm64, 2 CPUs, 2 GB) on an M4 Max laptop, not Docker Desktop. It’s one run, a demo and not a benchmark. ...

July 25, 2026 · 8 min · 1548 words · Kemal Akkoyun