# uprobe vs USDT cost in Go: what one probe hit costs
> [!summary] TL;DR
> The kernel's uprobe benchmark (x86_64) prices a uprobe on a NOP at 313 ns per hit and a USDT site at 309 ns; a probe on a bare RET costs 913 ns. OBI skips uretprobes because Go stacks move, so it decodes each function and attaches a uprobe at every RET. USDT moves the work from the tool to the runtime; the trap costs the same per hit.
## Key points
- OBI (OpenTelemetry eBPF Instrumentation) attaches one entry uprobe plus one uprobe per RET instruction, found by decoding the machine code with x86asm.
- OBI skips uretprobes: "since go linkage is non-standard we can't use uretprobe" (v0.10.0, instructions.go).
- Go stacks "grow, and therefore move" (Ian Lance Taylor, golang/go#22008). A return address the kernel rewrote into a stack that then moves points nowhere good.
- Austin Clements on uretprobes (golang/go#27077): "It's not clear to me how the runtime (or anything that unwinds the stack) could account for this."
- Kernel uprobe benchmark, x86_64, one producer thread, trivial BPF counter program, CPU not stated: no probe 152.507 M/s (6.6 ns), uprobe-nop 3.190 M/s (313 ns), usdt-nop 3.235 M/s (309 ns), uprobe-ret 1.095 M/s (913 ns).
- Disabled probes are not free. PostgreSQL with --enable-dtrace evaluates probe arguments "whenever control passes through the macro, even if no tracing is being done". libbpf: a USDT "normally has zero overhead, unless it is being traced by some external entity".
- OBI keeps a table of struct offsets: 59 types and 96 fields at v0.10.0 (1,510 lines), 91 types and 183 fields at v0.14.0 (2,746 lines), refreshed by a daily workflow.
- One field, net/http.http2ClientConn.fr, sits at offset 208 in Go 1.17.0 and 368 from Go 1.26.0.
- The entry probe can fire twice: when Go grows the stack, the function restarts, and OBI's eBPF code has to allow for it.
- In OBI v0.14.0 the RET probes are batched into one attachment.
- A USDT site puts a NOP in the function body and a note in .note.stapsdt carrying a provider, a name, the site address, and per argument a size and a location such as -4@%edi.
- A USDT site is a plain uprobe in the body, so a moving stack cannot break it, and there are no struct offsets to track if the site's author computes the arguments. That last part is the author's opinion, based on what OBI carries today.
- A probe name can outlive a refactor. The caveat: CPython's docs say "No guarantees are made about probe compatibility between versions of CPython".
- Go has no native USDT today. The poc_usdt fork is an experiment; the larger idea is generic tracepoints the standard library exposes on purpose.
- Neither way carries trace context from an incoming request to an outgoing call. That stays unsolved.
## The probe target and its disassembly
The function under test uses //go:noinline so the compiler cannot fold it into main, leaving something left to probe:
```go
package main
import "fmt"
//go:noinline
func handle(n int) int {
if n < 0 {
return -1
}
if n == 0 {
return 0
}
return len(fmt.Sprint(n))
}
func main() {
fmt.Println(handle(42))
}
```
Built for linux/amd64 with Go 1.27.1, go tool objdump shows the entry check and every RET, one per return in the source. The listing is trimmed (… marks what was cut); the rest is byte for byte what the tool printed:
```text
TEXT main.handle(SB) ./main.go
main.go:6 0x49a080 493b6610 CMPQ SP, 0x10(R14)
main.go:6 0x49a084 765b JBE 0x49a0e1
…
main.go:11 0x49a09c c3 RET
…
main.go:13 0x49a0d2 c3 RET
…
main.go:8 0x49a0e0 c3 RET
…
main.go:6 0x49a0e6 e8b52afeff CALL runtime.morestack_noctxt.abi0(SB)
main.go:6 0x49a0f0 eb8e JMP main.handle(SB)
```
The entry is `CMPQ SP, 0x10(R14)`. R14 holds the current goroutine, so this compares the stack pointer with the goroutine's stack guard. If the stack is too small, JBE jumps to runtime.morestack, which grows the stack (a copy, so it moves) and restarts the function with JMP main.handle. Three RET instructions leave the body.
## How OBI probes Go functions today
- The textbook return trap is a uretprobe: the kernel rewrites the return address on the stack so the function returns into a trampoline. That is what moving Go stacks break.
- OBI decodes the function's machine code with x86asm, records every RET, and attaches a plain uprobe at each one (v0.10.0). For handle that means one entry probe and three return probes.
- In v0.14.0 the same idea is still there, now batched into one attachment.
- Reading a struct field from a probe needs its offset, and some offsets move between Go releases, so OBI keeps offsets.json current with a daily workflow.
- The entry probe fires twice when the stack grows and the function restarts (v0.14.0 eBPF code; the talk pins v0.10.0).
Count the exits yourself:
```bash
GOOS=linux GOARCH=amd64 go build -trimpath -o handle main.go
go tool objdump -s '^main.handle