Blog

Parts 2 and 4 of this series describe what OBI (OpenTelemetry eBPF Instrumentation) and the OpenTelemetry eBPF profiler can do. Here we make them do it with Docker Compose and no Kubernetes: one Go server nobody touches, OBI watching its requests, the profiler watching its CPU. We’ll follow one request, GET /orders/42, and see it twice, first as a trace and then as CPU samples. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 4 and Part 5, and the hands-on side of my GopherCon UK talk. It all ran on one machine: a Colima VM (Ubuntu 24.04, kernel 6.8.0, linux/arm64, 2 CPUs, 2 GB) on an M4 Max laptop, not Docker Desktop. It’s one run, a demo and not a benchmark. ...

Here is a pull request with two numbers for the same change. My machine said the multi scenario cost 230% over a plain build. CI said 81%. Both numbers were honest readings of a stopwatch. I think only one of them was measuring the pull request. We’ll work out which, and find the one question that would have told me before CI did. This is a companion to the Why Your Go Benchmarks Are Lying series, between Part 3 (the laptop) and Part 4 (CI). Those parts use microbenchmarks. This story is about a macrobenchmark, a whole build timed end to end, and it didn’t fit my GopherCon UK talk. Disclosure: I work at Datadog and I’m one of otelc’s maintainers. ...

Many tools that profile Go binaries resolve function names from the ELF symbol table or DWARF debug info, and production builds strip both. Point one of those profilers at a stripped binary and you get addresses, not names. The opentelemetry-ebpf-profiler doesn’t have this problem, and we’re going to find out why. Let’s start with a binary rather than a profiler. Here is the whole program, with //go:noinline so the compiler can’t fold checkout into main: ...

Note This post was originally published on the OpenTelemetry blog on 2026-07-16. I wrote it for the OpenTelemetry community; this is a mirror on my personal blog with some added commentary. Why this milestone matters to me I have been involved with the Go Compile-Time Instrumentation SIG since it kicked off at the start of 2025, when Alibaba and Datadog decided to pool their efforts into a single vendor-neutral project. Watching it reach v1 last week felt genuinely satisfying — not because “v1” is a magic number, but because it marks the point where we stopped calling it experimental and started calling it something people should actually run in their builds. ...

Every benchmark number starts life on somebody’s laptop. We write the function, run it, like what we see and open the pull request, long before CI or a pinned runner gets a say. That makes it the least controlled measurement in the pipeline, and the one we act on. If it lies, everything downstream inherits the lie. Sending a noisy benchmark to CI doesn’t fix the noise. It industrialises it. ...