# OBI: eBPF auto-instrumentation for Go without code changes > [!summary] TL;DR > OBI (OpenTelemetry eBPF Instrumentation), the direct successor to Grafana Beyla, attaches eBPF uprobes and kprobes to a running binary: no source changes, no recompile, no restart, no in-process agent. You get RED metrics and library-level spans for 13 Go libraries. It needs Linux 5.8+ (RHEL 8: 4.18+), BTF, and a set of Linux capabilities; custom spans and business context still need the OTel SDK or a compile-time tool. ## Key points - OBI is the direct successor to Grafana Beyla: Grafana Labs donated Beyla to the CNCF OpenTelemetry project in 2025, renamed it OBI, and moved core development to the [`opentelemetry-ebpf-instrumentation`](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.10.0) repository. Beyla lives on as Grafana Labs' distribution of OBI. - The release pinned for the talk (13 August 2026) is [v0.10.0](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.10.0) (2026-06-30), still in Development status. Breaking changes between minor releases are expected while it stays at `v0`. - How it works: eBPF uprobes (probes on user-space binaries) and kprobes (the same for kernel code), with the kernel JIT-compiling those programs to the host architecture (x86-64 or ARM64). For Go, OBI hooks specific library functions, not generic network traffic, which is what gives library-level span context. - A uprobe on `runtime.newproc1` records which goroutine started which, and OBI walks up to six parents to find the request. - For Go HTTP/gRPC RED metrics (rate, errors, duration), OBI attaches without source changes, recompilation, restarts, or an in-process agent. Deploy it as a DaemonSet (or a sidecar, or a host process) and spans and metrics flow to your OTel collector. - OBI documents 13 library-level baselines as of v0.10.0: `net/http`, `golang.org/x/net/http2`, `gorilla/mux`, `gin-gonic/gin`, `google.golang.org/grpc`, `net/rpc/jsonrpc`, `database/sql` (with the `go-sql-driver/mysql` and `lib/pq` drivers), `redis/go-redis/v9`, Kafka (`segmentio/kafka-go` and `IBM/sarama`), and `go.mongodb.org/mongo-driver` v1 and v2. Full version constraints are in [`SUPPORT_MATRIX.md`](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/v0.10.0/SUPPORT_MATRIX.md). - What "zero code changes" buys: HTTP RED metrics, yes; a library-level span for the database call, yes; the SQL statement text, opt-in via `db.query.text` (bound parameters are not captured); trace context on the gRPC call to the next service, yes once enabled (off by default, needs extra privileges); a span for "checkout" or an attribute like the customer's plan, no, that requires the SDK or a compile-time tool. - Kernel: the [docs](https://opentelemetry.io/docs/zero-code/obi/) ask for Linux 5.8+, with a RHEL exception of 4.18+ for RHEL 8-family distros with the required eBPF backports. It must also expose BTF (BPF Type Format); check that `/sys/kernel/btf/vmlinux` exists. On macOS, bring a VM. - Capabilities when running unprivileged for plain application observability: `CAP_BPF`, `CAP_SYS_PTRACE`, `CAP_NET_RAW`, `CAP_CHECKPOINT_RESTORE`, `CAP_DAC_READ_SEARCH` and `CAP_PERFMON`, plus `CAP_SYS_RESOURCE` on kernels before 5.11. - More features, more capabilities: `CAP_NET_ADMIN` is added when context propagation is enabled (a socket-level eBPF program adds a `traceparent` header to outgoing HTTP; off by default); `CAP_SYS_ADMIN` is required for Go library-level propagation (a uprobe writes the header into Go's own request buffer with `bpf_probe_write_user`), and it stands in for `CAP_PERFMON` when `kernel.perf_event_paranoid` is set high (the AKS and EKS defaults need it). - On [Kubernetes](https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes/), the preferred model for covering many services is a DaemonSet: one OBI pod per node, `hostPID: true`, no changes to application pods. The sidecar model (one OBI container per pod, `shareProcessNamespace: true` and `privileged: true`) gives finer control at the cost of efficiency at scale. - Cost: no measurements of OBI v0.10.0 itself; the [FOSDEM post](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/)'s benchmark is one demo workload. Every hit on an attached uprobe traps into the kernel, so a hot function pays per call. - Numbers from elsewhere: the kernel's own BPF selftests recorded about 313 ns per hit on an uprobe over a NOP (3.190 M/s, one x86_64 run, CPU not stated, [commit 0c4fc6bd6105](https://github.com/torvalds/linux/commit/0c4fc6bd61054a9378bce149b3758f9b6e8fb5ab)); a [2023 bpftime paper](https://arxiv.org/pdf/2311.07923) measured 3,224 ns on older, unstated hardware. The two sources differ tenfold. - Upkeep: OBI looks up Go struct-field offsets by Go and library version (`offsets.json`), and Go 1.26 removing `pcHeader.textStart` broke its symbol resolution in PIE and cgo binaries until [PR #1851](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1851). - Where it fits: baseline observability across services you don't own or can't rebuild right now. Complementary to compile-time tools like otelc, which give depth (custom spans, business logic, stdlib instrumentation) at the price of a rebuild and, for otelc, Go 1.25+. - Stability: the feature set is real, since OBI is the same codebase as Beyla, which reached 1.0 in November 2023. The API stability guarantees are explicitly not there yet, and the v0.10.0 release notes carry a long security-hardening section. Budget for minor release changes, and canary before a fleet-wide rollout. - Disclosure: the author works at Datadog and is one of otelc's maintainers; he does not contribute to OBI, so read the comparisons with that in mind. - The linked OBI docs pages are unversioned; they were read on 2026-10-02. - Versions, links and commands checked on 2 October 2026. ## How eBPF instrumentation watches a Go binary OBI places eBPF uprobes (probes on user-space binaries) in the target binaries and kprobes (the same for kernel code) in the kernel, and the kernel JIT-compiles those eBPF programs to the host architecture (x86-64 or ARM64). For Go, OBI hooks specific library functions, not generic network traffic, which is what gives it library-level span context rather than just raw packets. For a request like `GET /orders/42`, a probe already sits on the `net/http` code that will serve it. A uprobe on `runtime.newproc1` records which goroutine started which, and OBI walks up to six parents to find the request. For Go HTTP/gRPC RED metrics (rate, errors, duration), OBI attaches without source changes, recompilation, restarts, or an in-process agent. You deploy it as a DaemonSet (or a sidecar, or a host process), and spans and metrics start flowing to your OTel collector. Your service never gets a say. ## What OBI instruments in Go: the 13-library baseline For Go, OBI documents 13 library-level baselines as of v0.10.0. The list includes `net/http`, `golang.org/x/net/http2`, `gorilla/mux`, `gin-gonic/gin`, `google.golang.org/grpc`, `net/rpc/jsonrpc`, `database/sql` (with the `go-sql-driver/mysql` and `lib/pq` drivers), `redis/go-redis/v9`, Kafka (`segmentio/kafka-go` and `IBM/sarama`), and `go.mongodb.org/mongo-driver` v1 and v2. Full version constraints are in the [`SUPPORT_MATRIX.md`](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/v0.10.0/SUPPORT_MATRIX.md) file in the v0.10.0 tag. Say a service answers with `net/http`, queries MySQL through `database/sql`, and calls the next hop over gRPC. All three are on that list. Here is what "zero code changes" buys at each step of the request: | Step of the request | Zero code changes? | | --- | --- | | The request arrives: HTTP RED metrics | Yes | | The handler queries the database: a library-level span | Yes | | That span carries the SQL statement text | Yes, opt-in via `db.query.text`; bound parameters are not captured | | The gRPC call to the next service carries the trace context (for supported protocols) | Yes, once enabled (off by default, needs extra privileges) | | A span for "checkout", or an attribute like the customer's plan | No; requires SDK or compile-time tool | The last row is the library-bound visibility limit of kernel-level instrumentation. The kernel sees function arguments and return values at library boundaries, and can't synthesize business context that isn't there. The Limitations section of the [official docs](https://opentelemetry.io/docs/zero-code/obi/) says so plainly: "Use language agents or manual instrumentation when you need custom spans, application-specific attributes, business events, or other in-process telemetry that eBPF-based instrumentation cannot derive automatically." For business context you still need manual OTel SDK usage or a compile-time tool like otelc. ## What it requires: kernel, BTF, capabilities, deployment The kernel comes first: the [docs](https://opentelemetry.io/docs/zero-code/obi/) ask for Linux 5.8+, with a RHEL exception of 4.18+ for RHEL 8-family distros with the required eBPF backports. It must also expose BTF (BPF Type Format) information; most current mainstream distros ship with it, so check that `/sys/kernel/btf/vmlinux` exists. On macOS, eBPF is a Linux party, so bring a VM; the demo [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI and eBPF Profiler Demo Without Kubernetes|OBI and the eBPF profiler without Kubernetes]] does exactly that. Then the capabilities. OBI has no single capability set, because the required access grows with the features you turn on. For plain application observability it needs six Linux capabilities when it runs unprivileged: `CAP_BPF`, `CAP_SYS_PTRACE`, `CAP_NET_RAW`, `CAP_CHECKPOINT_RESTORE`, `CAP_DAC_READ_SEARCH` and `CAP_PERFMON` (plus `CAP_SYS_RESOURCE` on kernels before 5.11, which the 5.8 floor still allows). The [security page](https://opentelemetry.io/docs/zero-code/obi/security/) explains why each one is needed and has the per-feature breakdown. More features need more capabilities. `CAP_NET_ADMIN` is added when context propagation is enabled, which is off by default: a socket-level eBPF program then adds a `traceparent` header to outgoing HTTP. `CAP_SYS_ADMIN` is required for Go library-level propagation, where a uprobe writes the header into Go's own request buffer with `bpf_probe_write_user`, and it stands in for `CAP_PERFMON` when `kernel.perf_event_paranoid` is set high (the AKS and EKS defaults need it). Running with `privileged: true` in Kubernetes works, but it's the sledgehammer option. Security teams will scrutinize this list, and that review is the operational cost to factor in. On [Kubernetes](https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes/), the preferred model for covering many services is a DaemonSet: one OBI pod per node, with `hostPID: true` so it can see every process and no changes to application pods. The sidecar model (one OBI container per pod, with `shareProcessNamespace: true` and `privileged: true`) gives finer control at the cost of efficiency at scale. Start with the DaemonSet: deploy once, and every service on every node gets baseline observability. ## What it costs while running, and upkeep There are no measurements of OBI v0.10.0 itself; the [FOSDEM post](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/)'s benchmark is one demo workload. Every hit on an attached uprobe traps into the kernel, so a hot function pays per call; [[resources/documentation/Go/Instrumentation/Go Instrumentation - uprobe vs USDT Cost|What a uprobe costs, and what USDT buys]] takes that cost apart. The kernel's own BPF selftests recorded about 313 ns per hit on an uprobe over a NOP (3.190 M/s, one x86_64 run, CPU not stated, [commit 0c4fc6bd6105](https://github.com/torvalds/linux/commit/0c4fc6bd61054a9378bce149b3758f9b6e8fb5ab)); a [2023 bpftime paper](https://arxiv.org/pdf/2311.07923) measured 3,224 ns on older, unstated hardware. The two sources differ tenfold, and OBI's own number is being measured with a reproduction kit for a follow-up post. The rest is upkeep. OBI looks up Go struct-field offsets by Go and library version (`offsets.json`), and Go 1.26 removing `pcHeader.textStart` broke its symbol resolution in PIE and cgo binaries until [PR #1851](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1851). ## Where OBI fits Back to `GET /orders/42`. With nobody touching the service, OBI gives RED metrics for the request, library-level spans for its database and gRPC calls, and, with the right opt-ins and privileges, SQL text and trace context across services. It does not give what the on-call engineer asks first: which customer, which cart, which business step. That split shows where OBI fits: baseline observability across services you don't own or can't rebuild right now. Ship the DaemonSet, then decide which services deserve more. OBI and compile-time tools are complementary. OBI gives breadth without touching CI pipelines; compile-time tools like otelc give depth (custom spans, business logic, stdlib instrumentation) at the price of a rebuild and, for otelc, Go 1.25+. The decision rule is in [[resources/documentation/Go/Instrumentation/Go Instrumentation - Agent-Actionable Observability|part 6, the runbook]]. The caveat worth repeating: the feature set is real, since OBI is the same codebase as Beyla, which reached 1.0 in November 2023. The API stability guarantees, though, are explicitly not there yet, and the [v0.10.0 release notes](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.10.0) carry a long security-hardening section. Budget for minor release changes, and canary before a fleet-wide rollout. "Zero code changes" for a request like this: yes, inside a defined scope, with a defined price list. ## FAQ ### Does OBI require code changes, a rebuild, or a restart? No. For Go HTTP/gRPC RED metrics, OBI attaches without source changes, recompilation, restarts, or an in-process agent. A privileged agent on the node deploys it (DaemonSet, sidecar, or host process) and spans and metrics flow to your OTel collector. ### Which Go libraries does OBI instrument? 13 library-level baselines as of v0.10.0: `net/http`, `golang.org/x/net/http2`, `gorilla/mux`, `gin-gonic/gin`, `google.golang.org/grpc`, `net/rpc/jsonrpc`, `database/sql` (go-sql-driver/mysql, lib/pq), `redis/go-redis/v9`, Kafka (segmentio/kafka-go, IBM/sarama), and mongo-driver v1 and v2. Full version constraints are in SUPPORT_MATRIX.md. ### Can OBI capture SQL statements? Yes, opt-in via `db.query.text`: the library-level span carries the SQL statement text, but bound parameters are not captured. ### Can OBI create custom spans or business attributes? No. The kernel sees function arguments and return values at library boundaries and cannot synthesize business context that isn't there. The docs say to use language agents or manual instrumentation for custom spans, application-specific attributes, business events, or other in-process telemetry. ### What Linux kernel and capabilities does OBI need? Linux 5.8+ (RHEL 8-family: 4.18+ with the required eBPF backports), BTF exposed (check `/sys/kernel/btf/vmlinux`), and for plain application observability six capabilities when unprivileged: `CAP_BPF`, `CAP_SYS_PTRACE`, `CAP_NET_RAW`, `CAP_CHECKPOINT_RESTORE`, `CAP_DAC_READ_SEARCH`, `CAP_PERFMON`, plus `CAP_SYS_RESOURCE` on kernels before 5.11. ### DaemonSet or sidecar for OBI on Kubernetes? Start with the DaemonSet: one OBI pod per node with `hostPID: true`, no changes to application pods, every service on every node covered. The sidecar (one OBI container per pod, `shareProcessNamespace: true` and `privileged: true`) gives finer control at the cost of efficiency at scale. ### How much overhead does an eBPF uprobe add? There are no measurements of OBI v0.10.0 itself. Every uprobe hit traps into the kernel, so a hot function pays per call; the kernel's BPF selftests recorded about 313 ns per hit over a NOP (one x86_64 run), while a 2023 bpftime paper measured 3,224 ns on older, unstated hardware. The two differ tenfold. ### Is OBI stable enough for production? The feature set is real, since OBI is the same codebase as Beyla (1.0 in November 2023). But v0.10.0 is still in Development status, breaking changes between minor releases are expected at `v0`, and the release notes carry a long security-hardening section. Budget for minor release changes and canary before a fleet-wide rollout. ## Related - [[resources/documentation/Go/Instrumentation/Go Instrumentation|Go Instrumentation]] (series hub) - [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|Why Go Cannot Be Monkey Patched]] (part 1: the three places to intervene) - [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc Compile-Time Instrumentation]] (part 3: the build-time route, rebuild for depth) - [[resources/documentation/Go/Instrumentation/Go Instrumentation - uprobe vs USDT Cost|uprobe vs USDT Cost]] (what a uprobe hit costs) - [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI and eBPF Profiler Demo Without Kubernetes|OBI and eBPF Profiler Demo Without Kubernetes]] (running OBI in a VM) - [[resources/documentation/Go/Go - uretprobes]] - [[resources/documentation/Go/Go - Internals]] ## Field notes from agent sessions (2026-11-10) Agent sessions from July to November 2026 checked OBI's releases and documentation against the tagged source. The checks found documentation lagging the code, and one probe-source reading settled a claim the docs could not. - OBI's eBPF path yields traces and metrics only. Secondhand writeups that add "logs" fabricate this. - The OBI v1.0.0 tag sits on a release branch cut from rc.2. It is not on `main`, which has 130 newer commits. Commit history confirms "promoted unchanged from rc.2": only PR #3972 sits between the tags, and it changes only the schema URL and the Collector `go.mod`. Read vendor docs at the tag. - OBI docs lag code in two places. The Auto SDK support matrix says `otel` up to v1.46.0, but the code accepts more. The v1.0.0 README still says "Development", while the release notes call v1.0.0 the first stable core release. Quote the release notes, not the README. - "OBI never sees the `context.Context`" was false as written. The v0.10.0 `go_grpc.c` probe receives a `ctx_ptr` and reads `valueCtx.val`, even though it never uses the value. Reading the probe source settled a claim the docs could not. ### Observations - [otel] OBI eBPF path yields traces and metrics only. "Logs" in secondhand writeups is fabricated. #obi (source: agent sessions, 2026-07) - [otel] OBI v1.0.0 tag sits on a release branch cut from rc.2, not on `main` (130 newer commits). Only PR #3972 sits between the tags. #obi (source: agent sessions, 2026-11) - [otel] OBI support matrix caps `otel` at v1.46.0 but the code accepts more. The v1.0.0 README says "Development" while the release notes say first stable core release. #obi #docs (source: agent sessions, 2026-11) - [ebpf] The v0.10.0 `go_grpc.c` probe receives a `ctx_ptr` and reads `valueCtx.val` without using the value. #obi #ebpf (source: agent sessions, 2026-10) ## Sources - [How to Instrument Go Without Changing a Single Line of Code (talk, GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/) - [How to Instrument Go Without Changing a Single Line of Code (series)](https://kakkoyun.me/series/how-to-instrument-go-without-changing-a-single-line-of-code/) - [FOSDEM 2026: auto-instrumenting Go with eBPF](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/) - [OBI documentation](https://opentelemetry.io/docs/zero-code/obi/) - [OBI security page](https://opentelemetry.io/docs/zero-code/obi/security/) - [OBI Kubernetes setup](https://opentelemetry.io/docs/zero-code/obi/setup/kubernetes/) - [opentelemetry-ebpf-instrumentation repository (v0.10.0)](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/v0.10.0) - [OBI v0.10.0 release notes](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases/tag/v0.10.0) - [OBI v0.10.0 support matrix](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/v0.10.0/SUPPORT_MATRIX.md) - [PR #1851: Go 1.26 pcHeader.textStart fix](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/pull/1851) - [Linux commit 0c4fc6bd6105: uprobe NOP benchmark](https://github.com/torvalds/linux/commit/0c4fc6bd61054a9378bce149b3758f9b6e8fb5ab) - [bpftime paper (2023)](https://arxiv.org/pdf/2311.07923)