Go Instrumentation - kakkoyun's Brain Overflow

# Instrument Go without changing code: otelc, OBI, eBPF profiler > [!summary] TL;DR > Go builds a self-contained native binary: no bytecode, no classloader, no general runtime hook, so there is no `-javaagent` slot. Zero-touch instrumentation moves to three places: the build (`otelc`, a `-toolexec` wrapper that injects OpenTelemetry spans at compile time), the kernel (OBI, eBPF uprobes and kprobes on a running binary), and whole-node profiling (the `.gopclntab` section read from outside the process). The series covers what each route sees, what it costs, the open gaps, and a Docker Compose demo with no Kubernetes. ## Start here 1. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|Why Go cannot be monkey patched]]: Why is there no `-javaagent` for Go, and where can you intervene instead? 2. [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc compile-time instrumentation]]: How does one word in front of `go build` inject OpenTelemetry spans? 3. [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI eBPF Auto-Instrumentation|OBI eBPF auto-instrumentation]]: How does OBI trace a running Go binary with no source change, rebuild or restart? 4. [[resources/documentation/Go/Instrumentation/Go Instrumentation - uprobe vs USDT Cost|uprobe vs USDT cost]]: What does an eBPF probe cost per hit, and why does OBI skip uretprobes on Go? 5. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Flight Recording and USDT|Flight recording and USDT]]: What shipped in Go 1.25's `runtime/trace`, and which httptrace gap breaks HTTP/2 client spans? 6. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Continuous Profiling Without Code Changes|Continuous profiling without code changes]]: How do you profile a stripped production Go binary from outside the process? 7. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Context Propagation Across Goroutines|Context propagation across goroutines]]: What crosses a `go` statement, and what does a goroutine slot repair? 8. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Agent-Actionable Observability|Agent-actionable observability]]: OBI or otelc: which one, when? 9. [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI and eBPF Profiler Demo Without Kubernetes|OBI and eBPF profiler demo without Kubernetes]]: What does one untouched Go server look like as a trace and as CPU samples? ## Key points - Go compiles to a self-contained static binary: no bytecode to rewrite at load time, no classloader moment, no general runtime hook API. `LD_PRELOAD` needs the dynamic linker, and pure-Go programs link statically by default. - Three places to intervene: build time (a `-toolexec` wrapper in front of every `go tool compile` call), process start (an injector library, with limits), and the kernel (eBPF). - otelc: `otelc go build -o myapp .` is the whole usage change. It needs Go 1.25 or newer and a rebuild; v1.0.1 is the first usable stable release (v1.0.0 was retracted). Default rules at v1.0.1 cover `net/http`, gRPC, `database/sql`, gin, go-redis v9, mongo-driver and Kubernetes client-go, among others. - OBI, the direct successor to Grafana Beyla, attaches eBPF uprobes and kprobes to a running binary: no source changes, no recompile, no restart, no in-process agent. RED metrics and library-level spans for 13 Go libraries at v0.10.0. It needs Linux 5.8+ (RHEL 8: 4.18+), BTF, and a set of Linux capabilities. - Kernel uprobe benchmark (x86_64): a uprobe on a NOP costs about 313 ns per hit, a USDT site 309 ns, a probe on a bare RET 913 ns. OBI skips uretprobes because Go stacks move; it decodes each function and attaches a uprobe at every RET instruction instead. - Go 1.25 shipped a flight recorder in `runtime/trace`: a circular buffer that keeps the last N seconds of execution trace in memory and snapshots on demand (`trace.NewFlightRecorder` with `MinAge` and `MaxBytes`). - An httptrace gap (no `GotResponseEnd` hook) means OTel Go's HTTP/2 client receive spans are never ended; golang/go#75654 proposes the hook. - Production builds strip the symbol table and DWARF debug info, but `.gopclntab` survives because the runtime needs it. The opentelemetry-ebpf-profiler reads it from outside the process: no SDK import, no rebuild, no restart. The price is one privileged agent per node, and the OTel profiles signal is Alpha. - Context propagation across goroutines is unsolved at the language level: Go has no goroutine-local storage, and a goroutine handed `context.Background()` loses the trace ID, the deadline and the cancellation. otelc's goroutine slot repairs the span tree but not the request context; OBI can propagate trace headers from outside (off by default, needs extra privileges) via uprobes on `runtime.newproc1` and BPF maps keyed by goroutine. - Decision rule: the service is already running and cannot be rebuilt, OBI is the default; you control the build and need function-level spans on Go 1.25+, otelc is the default. - Disclosure: the author works at Datadog, which maintains Orchestrion and dd-trace-go, and is one of otelc's maintainers; the author does not contribute to OBI; read the comparisons with that in mind. - Demo without Kubernetes: one Docker Compose stack, five services, one untouched Go server. OBI produced one server span (`GET /orders/{id}`, 11.63 ms) in Jaeger with no rebuild, and the profiler read `main.lookupOrder` off a stripped binary (100 of 106 samples landed there). ## FAQ ### How do I instrument Go without changing code? Two routes: `otelc go build` injects spans at compile time (Go 1.25+, needs a rebuild), or OBI's eBPF probes attach to the already-running binary. The routing question in [[resources/documentation/Go/Instrumentation/Go Instrumentation - Agent-Actionable Observability|Agent-Actionable Observability]] ### Why can Go not be monkey patched like Java? Go compiles to static native code: no bytecode, no classloader, no runtime hook API to intercept. Intervene at build time, process start or in the kernel instead, per [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|Why Go Cannot Be Monkey Patched]] ### What is otelc? The OpenTelemetry Go compile-time instrumentation tool. Prefix `go build` with `otelc` and get a binary with OTel spans for the libraries it knows, `net/http` and gRPC among them. Go 1.25+ and a rebuild required: [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc Compile-Time Instrumentation]] ### What is OBI? OpenTelemetry eBPF Instrumentation, the direct successor to Grafana Beyla. eBPF uprobes and kprobes attach to a running Go binary with no source change, recompile or restart; 13 Go library baselines at v0.10.0: [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI eBPF Auto-Instrumentation|OBI eBPF Auto-Instrumentation]] ### How much does an eBPF uprobe cost? About 313 ns per hit on a NOP (kernel benchmark, x86_64); a USDT site costs 309 ns and a probe on a bare RET 913 ns. OBI skips uretprobes on Go because Go stacks move: [[resources/documentation/Go/Instrumentation/Go Instrumentation - uprobe vs USDT Cost|uprobe vs USDT Cost]] ### How do I profile a stripped Go binary in production? `.gopclntab` survives stripping because the runtime needs it to walk stacks. The opentelemetry-ebpf-profiler reads it from outside the process; one privileged agent per node, and the OTel profiles signal is still Alpha: [[resources/documentation/Go/Instrumentation/Go Instrumentation - Continuous Profiling Without Code Changes|Continuous Profiling Without Code Changes]] ### How do I propagate trace context across goroutines? Not automatically. Go has no goroutine-local storage, and a goroutine handed `context.Background()` loses the trace ID, the deadline and the cancellation. otelc's slot repairs the span tree; OBI can propagate headers via uprobes on `runtime.newproc1` (off by default, needs extra privileges): [[resources/documentation/Go/Instrumentation/Go Instrumentation - Context Propagation Across Goroutines|Context Propagation Across Goroutines]] ### OBI or otelc: which should I use? Service already running and cannot be rebuilt: OBI. You control the build and want function-level spans on Go 1.25+: otelc. The decision rule in [[resources/documentation/Go/Instrumentation/Go Instrumentation - Agent-Actionable Observability|Agent-Actionable Observability]] ### Can I see OBI run without Kubernetes? Yes. One Docker Compose stack, one untouched Go server: OBI produced a trace in Jaeger and the profiler read function names from a stripped binary. The walkthrough in [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI and eBPF Profiler Demo Without Kubernetes|OBI and eBPF Profiler Demo Without Kubernetes]] ### Did Go 1.25 add a flight recorder? Yes, in `runtime/trace`: a circular buffer of the last N seconds of execution trace, written on demand. The API and its gaps in [[resources/documentation/Go/Instrumentation/Go Instrumentation - Flight Recording and USDT|Flight Recording and USDT]] ## Related - [[resources/documentation/Go/Go - uretprobes]] - [[resources/documentation/Go/Go - Internals]] ## Sources - [How to Instrument Go Without Changing a Single Line of Code (GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/)