Auto-Instrumentation

A production bug, a service you cannot touch, and a debugging loop measured in deploys. This talk follows one Go process through its lifecycle and finds an instrumentation point at each station: at build time, at process start, and in the kernel. At build time, otelc rewrites the code during compilation, so the binary ships already instrumented. At process start, the OpenTelemetry injector loads an agent into the process without a rebuild. In the kernel, OBI attaches eBPF probes to a static Go binary and collects traces and metrics from outside the process. Along the way we pick up CPU profiles, see where request correlation across goroutines and connections breaks down, and when the baseline still cannot answer the question, a live debugging probe captures the missing logs and variables within explicit limits. ...

The debugging loop is slow. You cannot reproduce the bug locally, so you add a log line and redeploy. Wrong place. You do it again. An agent alone does not fix this: it still has to pick a mechanism, and it has to know what that mechanism costs. This talk is about three approaches that take the source edit out of that loop. Each acts at a different point in the software lifecycle, and each inherits the strengths and constraints of its intervention point. ...

This post expands on the FOSDEM 2026 Go Devroom talk I co-presented with Hannah S. Kim. The talk, demo code, and all benchmark scenarios are available in the fosdem-2026 repository. The Problem Go is one of the best languages for building production backend services. It compiles to native binaries, has excellent concurrency primitives, and produces predictable performance characteristics. But when it comes to auto-instrumentation — adding observability without modifying source code — Go is uniquely difficult. ...

Brussels Again, But Make It Unplugged The day after FOSDEM, about a hundred of us gathered at Sparks Meeting on Rue Ravenstein in Brussels for OTel Unplugged EU 2026 — an unconference dedicated entirely to OpenTelemetry. Purple stage lights, a mid-century auditorium with wood paneling, and the familiar buzz of people who spend their days thinking about telemetry pipelines. If you know, you know. ...

Zero-touch observability for Go is finally becoming real. In this talk, we walk through the different strategies you can use to instrument Go applications without changing a single line of code, and what they cost you in terms of overhead, stability, and security. We compare several concrete approaches and projects: eBPF-based auto-instrumentation using OpenTelemetry’s Go auto-instrumentation agent and OBI (OpenTelemetry eBPF Instrumentation), compile-time manipulation using tools like Orchestrion and the OpenTelemetry Compile-Time Instrumentation SIG, runtime injection via Frida/ptrace, and USDT (User Statically-Defined Tracing) probes — both via libstapsdt and a custom Go runtime fork. ...