The debugging loop is slow. You cannot reproduce the bug locally, so you add a log line and redeploy. Wrong place. You do it again. An agent alone does not fix this: it still has to pick a mechanism, and it has to know what that mechanism costs.

This talk is about three open-source approaches that take the rebuild out of that loop. Each acts at a different point in the software lifecycle, and each inherits the strengths and constraints of its intervention point.

Build time. otelc rewrites Go code during the build via -toolexec, preserving Go-level semantics across platforms. The rebuild is the cost.

Process start. An injector loads a shared library at startup via LD_PRELOAD. No source change, no rebuild. The binary is the constraint.

Kernel. OBI (OpenTelemetry eBPF Instrumentation) observes running processes from the Linux kernel without rebuilding them. Linux, privileges, and kernel contracts are the cost.

We connect the signals. Tracing follows requests, profiling finds CPU cost, and the two become much stronger when they share request context. Go cannot use the native thread-local mechanism directly, so pprof labels become the bridge. The OpenTelemetry eBPF Profiler samples whole-node CPU activity without changing applications, and .gopclntab survives even fully stripped static binaries.

The talk closes with a decision framework: start with the constraint you cannot change, then add the next layer only when its signal pays for its operational cost.

Links

Events

Related