Engineering · · 6 min

The laptop said 230%

Here is a pull request with two numbers for the same change. My machine said the multi scenario cost 230% over a plain build. CI said 81%. Both numbers were honest readings of a stopwatch. I think only one of them was measuring the pull request. We’ll work out which, and find the one question that would have told me before CI did. This is a companion to the Why Your Go Benchmarks Are Lying series, between Part 3 (the laptop) and Part 4 (CI). Those parts use microbenchmarks. This story is about a macrobenchmark, a whole build timed end to end, and it didn’t fit my GopherCon UK talk. Disclosure: I work at Datadog and I’m one of otelc’s maintainers. ...

July 24, 2026 · 6 min · 1134 words · Kemal Akkoyun
Engineering · · 7 min

The fourth signal: continuous profiling without code changes

Many tools that profile Go binaries resolve function names from the ELF symbol table or DWARF debug info, and production builds strip both. Point one of those profilers at a stripped binary and you get addresses, not names. The opentelemetry-ebpf-profiler doesn’t have this problem, and we’re going to find out why. Let’s start with a binary rather than a profiler. Here is the whole program, with //go:noinline so the compiler can’t fold checkout into main: ...

July 23, 2026 · 7 min · 1385 words · Kemal Akkoyun
Engineering · · 7 min

OpenTelemetry Go Compile-time Instrumentation v1

Note This post was originally published on the OpenTelemetry blog on 2026-07-16. I wrote it for the OpenTelemetry community; this is a mirror on my personal blog with some added commentary. Why this milestone matters to me I have been involved with the Go Compile-Time Instrumentation SIG since it kicked off at the start of 2025, when Alibaba and Datadog decided to pool their efforts into a single vendor-neutral project. Watching it reach v1 last week felt genuinely satisfying — not because “v1” is a magic number, but because it marks the point where we stopped calling it experimental and started calling it something people should actually run in their builds. ...

July 22, 2026 · 7 min · 1434 words · Kemal Akkoyun
Engineering · · 12 min

Before CI: Can You Trust a Benchmark on Your Own Laptop?

Every benchmark number starts life on somebody’s laptop. We write the function, run it, like what we see and open the pull request, long before CI or a pinned runner gets a say. That makes it the least controlled measurement in the pipeline, and the one we act on. If it lies, everything downstream inherits the lie. Sending a noisy benchmark to CI doesn’t fix the noise. It industrialises it. ...

July 21, 2026 · 12 min · 2373 words · Kemal Akkoyun
Engineering · · 8 min

otelc: zero-touch Go traces at compile time

We’re going to instrument a Go program without touching its source, and the whole change fits in one build command. Here’s the before and after: 1 2 go build -o myapp . otelc go build -o myapp . Same flags, same package, one extra word in front. The binary that comes out has OpenTelemetry instrumentation compiled in for the libraries otelc knows about, net/http and gRPC among them, and nobody opened a handler file. ...

July 16, 2026 · 8 min · 1632 words · Kemal Akkoyun