Engineering · · 9 min

Making zero-touch Go observability agent-actionable

You can’t reproduce the bug locally, so you add a log line and redeploy. Wrong place. You add another one and redeploy again. That is the debugging loop from my GopherCon UK talk, and in this last part we’re going to shorten it: a decision rule for two zero-touch tools, then a runbook an AI coding agent (not a Java-style one) can execute. The runbook starts with one boring question: is this Go service already running, or can I rebuild it? If the service has to stay untouched, OBI (OpenTelemetry eBPF Instrumentation) is the default. If I can rebuild it and need function-level spans, otelc (OpenTelemetry Go compile-time instrumentation) is the default. The rest of the decision tree is checking privileges, Go version, collector endpoint, and whether RED metrics are enough. ...

August 6, 2026 · 9 min · 1898 words · Kemal Akkoyun
Engineering · · 9 min

Go runtime futures: flight recording, USDT, and the instrumentation hook problem

Every route in this series goes around the Go runtime: rewrite the source at build time, inject something at process start, or watch from the kernel, because Go gives us no agent API to plug into. This time we ask the runtime what it could hand us. The honest summary for 2026: one long-awaited feature shipped, two open proposals document gaps that have caused real production problems for years, and two experiments of mine are aspirations, not proposals. ...

July 30, 2026 · 9 min · 1826 words · Kemal Akkoyun
Engineering · · 7 min

The fourth signal: continuous profiling without code changes

Many tools that profile Go binaries resolve function names from the ELF symbol table or DWARF debug info, and production builds strip both. Point one of those profilers at a stripped binary and you get addresses, not names. The opentelemetry-ebpf-profiler doesn’t have this problem, and we’re going to find out why. Let’s start with a binary rather than a profiler. Here is the whole program, with //go:noinline so the compiler can’t fold checkout into main: ...

July 23, 2026 · 7 min · 1385 words · Kemal Akkoyun
Engineering · · 8 min

otelc: zero-touch Go traces at compile time

We’re going to instrument a Go program without touching its source, and the whole change fits in one build command. Here’s the before and after: 1 2 go build -o myapp . otelc go build -o myapp . Same flags, same package, one extra word in front. The binary that comes out has OpenTelemetry instrumentation compiled in for the libraries otelc knows about, net/http and gRPC among them, and nobody opened a handler file. ...

July 16, 2026 · 8 min · 1632 words · Kemal Akkoyun
Engineering · · 7 min

OBI: eBPF auto-instrumentation for Go in production

Take one HTTP request, GET /orders/42, headed for a Go service nobody is going to rebuild this quarter. We want a span for it, another for its database call, and the trace context handed on to the next service, all without touching the service. “Zero code changes” travels far in conference talks and vendor docs. Today we watch that one request from the kernel and find out where the phrase holds. ...

July 9, 2026 · 7 min · 1418 words · Kemal Akkoyun