Go Instrumentation - Flight Recording and USDT

# Go flight recorder, USDT probes, and missing runtime hooks > [!summary] TL;DR > Go 1.25 shipped a flight recorder in runtime/trace: a circular buffer that keeps the last N seconds of execution trace in memory and snapshots on demand. Two open proposals document gaps that have caused real production problems for years: an httptrace hook to end HTTP/2 client spans, and better -toolexec support for compile-time tools. USDT probes for Go exist only in an experimental fork, and trace context propagation without threading context.Context stays unsolved. ## Key points - Go 1.25 includes a circular buffer tracer in runtime/trace (golang/go#63185, closed and shipped), in the spirit of Java Flight Recorder. It records goroutine, syscall and GC events read with go tool trace, not OpenTelemetry spans in Jaeger. - The API is a config struct, not setter methods: trace.NewFlightRecorder with MinAge and MaxBytes, then Start and WriteTo. MinAge is a lower bound; MaxBytes wins if they conflict. - The recorder traces continuously. The Go blog puts a busy service at around 10 MB/s of trace data and suggests a MinAge of about 2x the event window. - Only one goroutine may call WriteTo at a time; a concurrent call returns an error. - The shipped API wants a few lines in the service: wire WriteTo to an error handler or a debug HTTP endpoint (net/http/pprof-style). - The poc_flight_recorder fork, one of the author's own experiments, adds an EventFilter bitmask (FilterHTTP, FilterSQL, FilterNet), SetEventFilter, SetSampleRate, W3C TraceContext parsing, and HTTPServerRequest and HTTPClientRequest spans; an aspiration, and the Go team is not expected to extend the flight recorder this way. - golang/go#75654 proposes a GotResponseEnd hook on httptrace.ClientTrace, fired exactly once per request when the response body is fully consumed; no acceptance decision yet. - OTel Go's workaround, PutIdleConn, is documented as not used for HTTP/2, so OTel Go's http.receive client span is never ended on HTTP/2 (open-telemetry-go-contrib#4876, still open in October 2026). - otelc's net/http client hook (v1.1.0) doesn't reference httptrace, and OBI watches net/http with uprobes; HTTP/2 spans from either were not tested by the author. - golang/go#69887: a -toolexec tool can't influence the build cache per package, only via the -V=full version probe, which works at the level of the complete build. otelc changes every tool's version line, so instrumented builds never share cache entries with plain go build; its own cache keeps warm rebuilds fast. - The toolchain doesn't tell -toolexec tools the full build arguments, so Orchestrion crawls its own process tree for a parent go build and parses its arguments. - Michael Matloob: dedicated support for source-rewriting tools would add a lot of complexity to the go command and the costs are not worth it. Austin Clements: a useful problem report, line-information handling a known weakness, suggested go list plus -overlay. Still Incoming on the proposals project as of October 2026. golang/go#73798, a goroutine-start hook, was closed as not planned on the day of filing, with a first reply pointing at toolexec rewrites. - USDT probes are named sites compiled in as NOPs; attaching with bpftrace, perf, or a custom eBPF program replaces the NOP with an INT3 interrupt, the same kernel path as uprobes. A USDT site is a uprobe on a NOP, so it still costs a trap per hit. - The poc_usdt fork (commit c739f4e), one of the author's own experiments, adds USDT probes to net/http, database/sql, crypto/tls, and net via a go tool usdt subcommand. On Linux, a binary built with the fork and the internal linker gets standard library instrumentation with no SDK import and no application changes; on darwin the probes compile to nothing, and external linking drops them. - Trace context propagation is unsolved: without threading context.Context by hand it needs goroutine-associated storage that keeps deadlines, cancellation and values; the fork's WithSpan and SpanFromContext still ride on context.Context. Disclosure: the author works at Datadog (Orchestrion) and is one of otelc's maintainers. ## Flight recording shipped in Go 1.25 OBI tells you something is slow; flight recording tells you what the runtime was doing at the time. Asking for it uses a config struct, imports left out: ```go fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{ MinAge: 2 * time.Second, MaxBytes: 64 * 1024 * 1024, }) if err := fr.Start(); err != nil { log.Fatal(err) } // Later, on a slow request or error: var buf bytes.Buffer if _, err := fr.WriteTo(&buf); err != nil { log.Println("flight recorder:", err) } // buf now holds at least the last 2 seconds of trace data // (MinAge is a lower bound; MaxBytes wins if they conflict) ``` Start begins recording and WriteTo copies the buffer out when something goes wrong. Only writing and keeping the data is on demand: the recorder traces continuously, so you pay for tracing and for the in-memory buffer all the time. The Go blog post puts a busy service at around 10 MB/s of trace data and suggests a MinAge of about 2x the event window; only one goroutine may call WriteTo at a time, and a concurrent call returns an error. The poc_flight_recorder fork pushes further: an EventFilter bitmask (FilterHTTP, FilterSQL, FilterNet and friends) with SetEventFilter, SetSampleRate, W3C TraceContext parsing, and HTTPServerRequest and HTTPClientRequest spans to runtime/trace, aiming at always-on distributed tracing. That's an aspiration. The [FOSDEM 2026 post](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/) has the longer write-up. ## The httptrace gap that breaks HTTP/2 spans No hook reliably tells the client, on every protocol, that a response body has been fully consumed. This is the proposal's pseudo-code, not something httptrace has today: ```go // Add to httptrace.ClientTrace: GotResponseEnd func(err error) // Fires exactly once per request when resp.Body.Read returns io.EOF, // a non-nil error, or the body is closed early. ``` One callback, fired once. The shape is still being argued over, with commenters asking about responses with no body and bodies the application never reads, and there's no acceptance decision yet. PutIdleConn is the workaround OTel Go uses, but the standard library documents that hook as not used for HTTP/2, so on HTTP/2 OTel Go's http.receive client span is never ended. That's tracked in open-telemetry-go-contrib#4876, still open in October 2026. In the sources read, otelc's net/http client hook (v1.1.0) doesn't reference httptrace, and OBI watches net/http with uprobes, so neither should hit this, though HTTP/2 spans from either were not tested. ## Why compile-time instrumentation has rough edges Orchestrion and otelc (OpenTelemetry Go compile-time instrumentation) intercept the compilation pipeline with -toolexec, as covered in [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc Compile-Time Instrumentation]]. golang/go#69887 is Romain Marcadier's proposal to improve it. Two of its gaps explain the sharp edges: - A -toolexec tool can't influence the build cache per package. The only hook point is intercepting the -V=full version probe, which works at the level of the complete build, so the issue describes excessively frequent cache invalidation for untransformed packages. otelc changes every tool's version line, so instrumented builds never share cache entries with plain go build; its own cache keeps warm rebuilds fast. - The toolchain doesn't tell -toolexec tools the full build arguments, so Orchestrion crawls its own process tree looking for a parent go build invocation and parses its arguments. That's the workaround you write when the API won't give you what you need. Michael Matloob said dedicated support for source-rewriting tools would add a lot of complexity to the go command and that he doesn't think the costs are worth it. Austin Clements called it a useful problem report and line-information handling a known weakness, and suggested go list plus -overlay as a way around the build-graph changes. As of October 2026 it's still at Incoming on the proposals project. A neighbouring ask, a hook on goroutine start (golang/go#73798), was closed as not planned on the day of filing, with a first reply that pointed at toolexec rewrites. ## USDT probes in the poc_usdt fork USDT (User Statically-Defined Tracing) probes are named sites compiled into a binary as NOP instructions, which do nothing. When a consumer such as bpftrace, perf, or a custom eBPF program attaches, the NOP is replaced with an INT3 interrupt and the kernel delivers the event, the same kernel path as uprobes. A USDT site is a uprobe on a NOP, so it still costs a trap per hit, and the fork computes its arguments even with nothing attached, which hasn't been benchmarked. The win is less work for the tool, which reads site addresses and argument locations from the note instead of hunting for every RET, and a probe name that can outlive refactors. The fork (pinned at commit c739f4e) adds USDT probes to net/http, database/sql, crypto/tls, and net via a go tool usdt subcommand. This listing is abridged from the fork's README: ```bash $ go tool usdt list ./myserver PROVIDER NAME ADDRESS ARGUMENTS net_http server_request_start 0x63296c 8@%rsi -8@%r8 8@%rdx -8@%r9 net_http server_request_end 0x631c5c -4@%ecx $ go tool usdt bpftrace ./myserver > trace.bt $ sudo bpftrace trace.bt ``` - On Linux, a binary built with the fork and the internal linker gets standard library instrumentation automatically, with no SDK import and no application code changes. - On darwin the probes compile to nothing, and external linking currently drops them. - The probes are standard .note.stapsdt ELF notes, so readelf -n can read them. - bpftrace's usdt: attach doesn't find the probes in a static Go binary, so the script go tool usdt bpftrace generates uses uprobe: probes at the probe addresses instead. - ARM64 argument parsing in bpftrace also has issues with the probe argument notation the fork emits. - Nothing here is proposed upstream. The closest thing to a Go-team discussion is Felix Geisendörfer asking whether the Go team had considered USDT support, recorded in the notes for the 2024-12-19 performance and diagnostics sync (golang/go#57175). That's a brainstorm, not a plan, and the author is not aware of a proposal filed since. - The bigger idea is generic tracepoints: safe, named instrumentation points that the standard library and runtime expose on purpose, with stable arguments external tools can consume. USDT is one way to build them, and it needs more work and acceptance from the Go runtime team. ## Observing isn't propagating Tracepoints let a tool see a request. Distributed tracing also has to carry trace headers from the incoming connection to the outgoing ones. Without threading context.Context by hand, that needs some goroutine-associated storage, and it must not lose what the request's context already carries: deadlines, cancellation, and values the application attached. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|Why Go Cannot Be Monkey Patched]] showed tools hacking a field into g for this. The fork's WithSpan and SpanFromContext show the gap from the other side: they ride on context.Context, so the context still has to be threaded. [[resources/documentation/Go/Instrumentation/Go Instrumentation - Context Propagation Across Goroutines|Context Propagation Across Goroutines]] walks that gap with a small program. There is no solution. This is the unsolved part. Versions, links and commands checked on 2 October 2026. ## FAQ ### Did Go ship a flight recorder? Yes. Go 1.25 includes a circular buffer tracer in runtime/trace (golang/go#63185, closed and shipped). It keeps the last N seconds of execution trace in memory and snapshots on demand. ### How does the Go flight recorder API work? trace.NewFlightRecorder(trace.FlightRecorderConfig{MinAge: 2 *time.Second, MaxBytes: 64* 1024 * 1024}), then Start and WriteTo. The API is a config struct, not setter methods. MinAge is a lower bound; MaxBytes wins if they conflict. ### How much trace data does flight recording produce? The Go blog puts a busy service at around 10 MB/s of trace data and suggests a MinAge of about 2x the event window. The recorder traces continuously; only writing and keeping the data is on demand. ### Why are OTel Go's HTTP/2 client spans never ended? OTel Go uses PutIdleConn, which the standard library documents as not used for HTTP/2. The proposed fix is GotResponseEnd (golang/go#75654), fired once per request; there is no acceptance decision yet. ### What does the poc_usdt fork add? A go tool usdt subcommand and USDT probes in net/http, database/sql, crypto/tls, and net, pinned at commit c739f4e. On Linux, a binary built with the fork and the internal linker gets standard library instrumentation automatically. ### Is a USDT site faster than a uprobe? No. A USDT site is a uprobe on a NOP and still costs a trap per hit. The win is less work for the tool and a stable probe name; speed is unchanged. ### Does the Go team plan USDT support? The author is not aware of a proposal filed. Felix Geisendörfer asked about USDT support in the notes for the 2024-12-19 performance and diagnostics sync (golang/go#57175); that's a brainstorm, not a plan. ### Why is trace context propagation unsolved? Without threading context.Context by hand, tools need goroutine-associated storage that keeps deadlines, cancellation and values. The fork's WithSpan and SpanFromContext still ride on context.Context. ## Related - [[resources/documentation/Go/Instrumentation/Go Instrumentation|Go Instrumentation]] (series hub) - [[resources/documentation/Go/Instrumentation/Go Instrumentation - Continuous Profiling Without Code Changes|Continuous Profiling Without Code Changes]] - [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc Compile-Time Instrumentation]] - [[resources/documentation/Go/Instrumentation/Go Instrumentation - uprobe vs USDT Cost|uprobe vs USDT Cost]] - [[resources/documentation/Go/Instrumentation/Go Instrumentation - Context Propagation Across Goroutines|Context Propagation Across Goroutines]] - [[resources/documentation/Go/Go - uretprobes|uretprobes]] - [[resources/documentation/Go/Go - Internals|Go Internals]] ## Field notes from agent sessions (2026-11-10) Later sessions with the USDT fork and the eBPF toolchain added toolchain-level detail. They cover what it takes to add a compiler intrinsic, how USDT notes must be laid out for tracers, and why uretprobes do not work on Go. - The shipped flight recorder API differs from the accepted proposal (golang/go#63185). The only methods on `*FlightRecorder` are `Start`, `Stop`, `Enabled`, and `WriteTo`. There is no `SetMinAge` method. The shipped source is the ground truth, not the proposal or prior session notes. - `httptrace.ClientTrace` hooks run from multiple goroutines at the same time. The transport calls each hook from the goroutine that handles that connection phase. Hook implementations must be goroutine-safe. - `defer` evaluates its arguments when the defer statement runs, not at handler exit. A deferred USDT probe that passes `int32(w.StatusCode)` records the value at defer time, not the final status. Use a closure over a `status` variable that changes before exit. The defect came from the published FOSDEM snippet, not from review of the drafts. - Compiler intrinsics are keyed by `{arch, pkg, fn}` in `initIntrinsics()`. Adding one needs entries in the generic op table and in the arch-specific lowering rules. It also needs a `regInfo` struct that defines the input and output register constraints. - Variadic-argument SSA ops need `argLength: -1` and a `regInfo` with enough input entries to cover the maximum supported argument count. Register allocation then fills the slots per the ABI calling convention. - Metadata that crosses the compile/link boundary travels in `Aux` records on `LSym`. `goobj.AuxFuncInfo` is the pattern to follow when you attach per-function USDT note data. - `.note.stapsdt` needs a companion `.stapsdt.base` section. That section is a symbol with a known virtual address. Tracers use it to compute ASLR-adjusted probe addresses in PIE binaries. Go's static binaries make this easier, but `bpftrace` still expects the section. - The USDT argdesc wire format encodes signedness in the size. `-4@%rbx` means a signed 32-bit value in `rbx`. The size field is the C `sizeof` with the sign bit. Both `bpftrace` and SystemTap parse it. - `elfnote` adds `12 + sz` per size argument, but the section size function already counts the 12-byte note header per probe. The `.note.stapsdt` section size then exceeds the written bytes by 12 per probe. Parsers that trust `sh.Size` read a trailing junk note (`namesz=0`) off the end. - Non-aligned notes record `descsz` as the padded size, not the true size. This is a plausible real cause of `bpftrace` `usdt:` probe failures on such binaries. - Probe-site patchability is architecture-asymmetric. The amd64 probe is a 1-byte `0x90` NOP, so it cannot hold a 5-byte call. The only 1-byte patch is `INT3`. Go's runtime treats SIGTRAP as `_SigThrow` and does not forward a hardware-trap `si_code` to a preinstalled handler, so INT3 injection is expected to crash. The arm64 4-byte NOP is exactly one `B` instruction, so trampoline patching works there. The asymmetry argues for emitting a 5-byte NOP on amd64. - Generated files such as `opGen.go` must be regenerated from their `_gen/` source, not hand-edited. Upstream changed `regMask` from a single `uint64` to a two-word struct, and the checked-in generated file no longer compiled. Run the generator from its own module (go 1.24 is enough) so the bootstrap toolchain can execute it. - Go eBPF agents attach uprobes at every return statement instead of uretprobes. Goroutine stack growth can copy the stack to a new address. That copy invalidates the return-address trampolines that uretprobes install. - `.gopclntab` survives stripping and static linking. An eBPF profiler therefore symbolizes production Go binaries without DWARF and without build-time instrumentation. That table is what makes eBPF profiling zero-touch for Go. - A USDT site costs about the same per hit as a uprobe on a NOP (3.235M/s vs 3.190M/s, Linux commit 0c4fc6bd6105). The kernel nop5 optimization roughly doubles throughput on NOP sites only (3.160M/s to 7.286M/s). The shipped nop5 variant clobbered the user red zone and was replaced by nop10 (554ba38456da). nop10 is in v7.3-rc5 but not in v7.2. The "more stable" claim for USDT holds only partly. - `limactl start template://ubuntu` gives a kernel 5.8+ VM with BTF for eBPF work in one command. Benchmarks then run via `limactl shell ubuntu`. ### Observations - [go-api] The shipped flight recorder API is the ground truth, not the accepted proposal. #go #flight-recorder (source: agent sessions, 2026-07) - [go] httptrace ClientTrace hooks run from several goroutines. Keep them goroutine-safe. #go #httptrace (source: agent sessions, 2026-03) - [go] A deferred call captures its arguments at defer time. Use a closure for a value that changes before handler exit, such as the final status code. #go #usdt (source: agent sessions, 2026-10) - [go-compiler] A new intrinsic needs the generic op table, the arch-specific lowering rules, and a regInfo. #go-compiler (source: agent sessions, 2026-01) - [go-compiler] Variadic SSA ops need `argLength: -1` and enough regInfo input entries for the maximum argument count. #go-compiler (source: agent sessions, 2026-01) - [go-compiler] Cross-module metadata rides in Aux records on LSym. Follow the goobj.AuxFuncInfo pattern. #go-compiler #usdt (source: agent sessions, 2026-01) - [go-linker] A stapsdt note needs a companion .stapsdt.base section. Tracers use it to compute ASLR-adjusted probe addresses in PIE binaries. #usdt #go-linker (source: agent sessions, 2026-01) - [usdt] The argdesc size field is signed. It encodes both the C sizeof and the signedness of the argument. #usdt (source: agent sessions, 2026-01) - [go-linker] elfnote double-counts the 12-byte note header. `sh.Size` then runs 12 bytes long per probe. #usdt #elf (source: agent sessions, 2026-08) - [go-linker] Non-aligned stapsdt notes record descsz as the padded size. This can make bpftrace usdt: probes fail. #usdt (source: agent sessions, 2026-08) - [go-runtime] The amd64 NOP probe site cannot hold a 5-byte call, and INT3 crashes because Go treats SIGTRAP as fatal. Arm64 patching works. Emit a 5-byte NOP on amd64 instead. #usdt #go-runtime (source: agent sessions, 2026-08) - [go-compiler] Regenerate opGen.go from its `_gen/` source. Never hand-edit a stale generated file. #go-compiler (source: agent sessions, 2026-08) - [ebpf] Goroutine stack movement invalidates uretprobes. Go eBPF agents probe return sites with uprobes instead. #ebpf #go (source: agent sessions, 2026-07) - [ebpf] The .gopclntab table gives symbolization in stripped, static Go binaries without DWARF. #ebpf #profiling (source: agent sessions, 2026-07) - [ebpf] A USDT site costs the same per hit as a uprobe on a NOP. The nop5 optimization was unsafe and became nop10, which is not yet in v7.2. #ebpf #usdt (source: agent sessions, 2026-10) - [tips] limactl provides a BTF-equipped kernel 5.8+ VM in one command for eBPF benchmarks. #ebpf #lima (source: agent sessions, 2026-08) ## Sources - [How to Instrument Go Without Changing a Single Line of Code (GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/) - [Flight recorder proposal, closed and shipped (golang/go#63185)](https://github.com/golang/go/issues/63185) - [Go blog: flight recorder](https://go.dev/blog/flight-recorder) - [poc_flight_recorder fork](https://github.com/kakkoyun/go/tree/1f6836932a219381e4088d61cafdfefadd0dd03c) - [GotResponseEnd proposal on httptrace.ClientTrace (golang/go#75654)](https://github.com/golang/go/issues/75654) - [OTel Go contrib: HTTP/2 client span never ended (#4876)](https://github.com/open-telemetry/opentelemetry-go-contrib/issues/4876) - [toolexec improvement proposal (golang/go#69887)](https://github.com/golang/go/issues/69887) - [Hooking into the Go toolchain: the cache will lie to you](https://internals-for-interns.com/posts/hooking-into-the-go-toolchain/#the-cache-will-lie-to-you) - [Hooking into the Go toolchain: back to the stopwatch](https://internals-for-interns.com/posts/hooking-into-the-go-toolchain/#back-to-the-stopwatch) - [Goroutine-start hook closed as not planned (golang/go#73798)](https://github.com/golang/go/issues/73798) - [poc_usdt fork, pinned at commit c739f4e](https://github.com/kakkoyun/go/tree/c739f4e38a792c85c22ab79ec6b44c9bcd0ca691) - [poc_usdt README](https://github.com/kakkoyun/go/blob/c739f4e38a792c85c22ab79ec6b44c9bcd0ca691/src/runtime/trace/usdt/README.md) - [Performance and diagnostics sync notes, 2024-12-19 (golang/go#57175)](https://github.com/golang/go/issues/57175)