# OBI vs otelc: choosing zero-touch Go instrumentation
> [!summary] TL;DR
> Two zero-touch routes exist for a Go service, and one boring question routes between them: is the service already running, or can I rebuild it? If it must stay untouched, OBI (OpenTelemetry eBPF Instrumentation) is the default; if you can rebuild it and need function-level spans, otelc (compile-time instrumentation) is the default. With otelc the entire change is `otelc go build` plus an endpoint and a service name. On Kubernetes, `kubectl obi attach --dry-run` prints the plan and stops at a confirmation gate until a human approves.
## Key points
- The debugging loop this shortens: a bug that won't reproduce locally costs a log line and a redeploy, then another when the first guess lands in the wrong place.
- The boring question: if the service has to stay untouched, OBI is the default; if you can rebuild it and need function-level spans, otelc is the default. The rest of the decision tree is checking privileges, Go version, collector endpoint, and whether RED metrics are enough.
- Three intervention points exist (build time, process start, the kernel); for Go, two are real routes. Process start is out on purpose: the host injector preloads agents for Java, Node.js, .NET, Ruby, and Python (disabled by default), not Go, and a pure-Go build is static, so `LD_PRELOAD` has nothing to attach to.
- The OpenTelemetry Operator's `inject-go` annotation adds a per-pod sidecar running the Go auto-instrumentation agent (`go.opentelemetry.io/auto`, not OBI), off by default, and it needs a privileged root container.
- Reach for OBI when the service is already running (production, staging, Kubernetes, Docker Compose) and rebuilding or redeploying isn't acceptable, when the fleet is polyglot, or when HTTP/gRPC RED metrics (rate, errors, duration) and library-level spans are all you need.
- Reach for otelc when you control the build (a laptop, CI or the release pipeline), want granular spans for specific functions or one slow path, and are willing to rebuild with `otelc go build`. The tiebreaker is one question: can this service be rebuilt and redeployed (otelc), or does it have to stay untouched (OBI)?
- The profiler from part 4 is a layer beside either route, adding whole-node CPU profiles; a practical stack is otelc in the build, OBI for service boundaries and other languages, and the profiler for the node. The flight recorder lives in your code, so it stays out of the runbook. Orchestrion is not in this tree: the zeroins lookups cover only OBI and otelc, and part 3 still points to it for Datadog APM.
- Both routes need somewhere to send the output, such as an OTel Collector or Jaeger. OBI needs a BTF-enabled Linux kernel and a specific capability set; otelc needs Go 1.25 or newer.
- OBI instruments 13 Go libraries out of the box, including `net/http`, gin, gRPC and `database/sql`. It attaches from outside the process: no `LD_PRELOAD`, no rebuild, and in DaemonSet mode no restart.
- OBI skips services it sees publishing their own OpenTelemetry telemetry (`exclude_otel_instrumented_services`, on by default), so an OTel Java or Python agent shouldn't double up; the docs describe this for OpenTelemetry SDK calls, so test a pod with a vendor agent first.
- The OBI Helm chart 0.10.0 (`open-telemetry/opentelemetry-ebpf-instrumentation`) defaults are enthusiastic: every namespace, privileged, exporting to a collector on the node IP (`${HOST_IP}:4317` for traces, `:4318` for metrics). Point it elsewhere before installing it on a cluster you care about.
- The cost is span depth: custom spans and business-logic events still need code changes when they are not at the HTTP/gRPC boundary, and OBI can't see inside your own functions.
- `otelc go build` is a drop-in for `go build`: it instruments your code, its dependencies, and parts of the standard library at the AST level, and the injected spans cost the same as manually written OTel code. "Without a single line" means not one line of application source, one changed line in the build script.
- otelc's runtime exports OTLP over HTTP (`http/protobuf`) by default, so the endpoint is 4318; 4317 is the gRPC port. Point the default protocol at 4317 and you get `malformed HTTP response` errors instead of traces; to use 4317, set `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`. The tradeoff is build-time coupling (no attaching to a running binary); the getting-started guide covers CI/CD and calls it stable and ready for production use as of v1.0.0.
- The agent skill `collect-go-telemetry` (installed with `npx skills add kakkoyun/zeroins --all`) wraps a fixed workflow around the tree: preflight, coverage lookup, dry-run plan, confirmation gate, bounded attach, confirm traces, session audit. Coverage lookups (`zeroins obi lookup <library>`, `zeroins otelc lookup <import-path>`) are offline, from a snapshot embedded in the binary (OBI v0.10.0 and otelc v1.0.1 in zeroins v0.2.0).
- `kubectl obi` is experimental, not a production installer: read the dry-run first. `--endpoint` is required (an OTLP HTTP(S) base URL, no `/v1/traces` suffix) and it needs helm v4. The DaemonSet installs the pinned OBI chart (0.10.0, OBI v0.10.0, the release pinned for the talk on 13 August 2026) into `obi-system`; sidecar mode patches one deployment and restarts it, so the "no restart" promise holds only for the DaemonSet. Guardrails: `--duration` auto-detach, `zeroins sessions list` and `reap`, and a confirmation gate before any attach.
## The routing decision: two routes for a Go service
The series follows three intervention points: build time, process start and the kernel. For a Go service, two of them are real routes. Reach for OBI when the service is already running (production, staging, Kubernetes, Docker Compose) and rebuilding or redeploying isn't acceptable, when the fleet is polyglot, or when HTTP/gRPC RED metrics (rate, errors, duration) and library-level spans are all we need. Reach for otelc when we control the build (a laptop, CI or the release pipeline), want granular spans for specific functions or one slow path, and are willing to rebuild with `otelc go build`. If neither fits cleanly, the tiebreaker is one question: can this service be rebuilt and redeployed (otelc), or does it have to stay untouched (OBI)?
The profiler from [[resources/documentation/Go/Instrumentation/Go Instrumentation - Continuous Profiling Without Code Changes|part 4]] isn't a third route but a layer beside either one, adding whole-node CPU profiles. A practical stack is otelc in the build for semantic spans, OBI for service boundaries and other languages, and the profiler for the node; add a layer when its signal earns the cost. The flight recorder lives in your code, so it stays out of the runbook. Orchestrion is not in this tree, since the zeroins lookups cover only OBI and otelc (the where-it-fits-relative-to-obi section of [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|part 3]] still points to it for Datadog APM).
Both routes need somewhere to send the output, such as an OTel Collector or Jaeger. OBI needs a BTF-enabled Linux kernel and a specific capability set (the what-it-requires section of [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI eBPF Auto-Instrumentation|part 2]] has the list), and otelc needs Go 1.25 or newer.
What happened to process start? It was left out on purpose. The [OpenTelemetry host injector](https://github.com/open-telemetry/opentelemetry-injector/blob/v0.10.1/README.md) preloads agents for Java, Node.js, .NET, Ruby, and Python (disabled by default), not Go, and a pure-Go build is static, so [`LD_PRELOAD` has nothing to attach to](https://github.com/open-telemetry/opentelemetry-injector/blob/v0.10.1/DESIGN.md). The OpenTelemetry Operator's [`inject-go` annotation](https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/) doesn't change that: it adds a per-pod sidecar running the Go auto-instrumentation agent (`go.opentelemetry.io/auto`, not OBI), off by default, that needs a privileged root container. A fleet-wide injector can keep the other runtimes while Go services route to OBI, or to otelc if you can rebuild (the process-start section of [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|part 1]] has the background). Two routes remain.
## What OBI covers, and the span-depth cost
OBI instruments [13 Go libraries](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/v0.10.0/SUPPORT_MATRIX.md) out of the box, including `net/http`, gin, gRPC and `database/sql` (the what-it-actually-instruments-in-go section of [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI eBPF Auto-Instrumentation|part 2]] has the list). It attaches from outside the process: no `LD_PRELOAD`, no rebuild, and in DaemonSet mode no restart. It also skips services it sees publishing their own OpenTelemetry telemetry (`exclude_otel_instrumented_services`, on by default), so an OTel Java or Python agent shouldn't double up; the docs describe this for OpenTelemetry SDK calls, so test a pod with a vendor agent first. On Kubernetes that DaemonSet comes from a Helm chart, and the defaults of chart 0.10.0 (`open-telemetry/opentelemetry-ebpf-instrumentation`) are enthusiastic: every namespace, privileged, exporting to a collector on the node IP (`${HOST_IP}:4317` for traces, `:4318` for metrics). Point it elsewhere before installing it on a cluster you care about.
The cost is span depth. Custom spans and business-logic events still need code changes when they are not at the HTTP/gRPC boundary. OBI observes what crosses library boundaries and can't see inside our logic, so when the slow part is in our own functions, we need the other route.
## What otelc covers: one changed line in the build script
otelc takes the compile-time path: [`otelc go build`](https://opentelemetry.io/docs/zero-code/go/compile-time/) is a drop-in for `go build`. It instruments your code, its dependencies, and parts of the standard library at the AST level, and the injected spans cost the same as manually written OTel code. Here is the entire change: install the tool, swap the build command, run the binary with an endpoint and a service name.
```bash
go install go.opentelemetry.io/otelc/tool/cmd/
[email protected]
otelc go build -o ./myapp .
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_SERVICE_NAME=my-go-service \
./myapp
```
That is what "without a single line" means in practice: not one line of application source, one changed line in the build script. The endpoint says 4318 on purpose. otelc's runtime exports OTLP over HTTP (`http/protobuf`) by default, and 4317 is the gRPC port. Point the default protocol at 4317 and you get `malformed HTTP response` errors instead of traces; to use 4317, set `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.
The tradeoff is build-time coupling: we can't attach otelc to a running binary, or instrument services we are not rebuilding. We can put it in the release pipeline, though. The [getting-started guide](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/v1.1.0/docs/getting-started.md) covers CI/CD and calls it stable and ready for production use as of v1.0.0. Two tools, two sets of costs, and a choice an impatient human will get wrong at 3 a.m., which is why the next section writes it down so an agent can follow it.
## The agent skill and per-library lookups
A decision tree in a blog post doesn't shorten anyone's loop. As the [talk](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/) puts it, an agent on its own still has to pick a mechanism and know what it costs. The routing logic lives in an agent skill called [`collect-go-telemetry`](https://github.com/kakkoyun/zeroins/blob/v0.2.0/skills/collect-go-telemetry/SKILL.md), installed with `npx skills add kakkoyun/zeroins --all`. It encodes the decision tree and wraps it in a fixed workflow: preflight, coverage lookup, dry-run plan, confirmation gate, bounded attach, confirm traces, session audit. The tree is fixed; the context (which service, which libraries, which environment) is what an agent inspects. Disclosure: the author is one of otelc's maintainers, and [zeroins](https://github.com/kakkoyun/zeroins/tree/v0.2.0), `kubectl-obi` and the skill are the author's own projects; the skill and plugins live in the zeroins repository.
The coverage step is the clever bit. Pasting OBI's support matrix and otelc's library catalog into context for every request is wasteful, so the skill uses per-library lookups (`zeroins obi lookup <library>`, `zeroins otelc lookup <import-path>`) that return only the relevant rows. They are offline, reading a snapshot embedded in the binary (OBI v0.10.0 and otelc v1.0.1 in zeroins v0.2.0), so they change only when zeroins does. Treat documentation as a retrieval problem, not a context-stuffing problem.
## kubectl obi on Kubernetes: lifecycle, guardrails, and a safe try-out
`kubectl-obi` is one of two experimental `kubectl` plugins in zeroins, written for the talk. `go install` puts an executable named `kubectl-obi` on your `PATH`, and `kubectl` discovers it as `kubectl obi`. It deploys privileged eBPF observers and is not a production installer, so the first thing to do is read the dry-run. It shells out to `helm` and `kubectl` (zeroins asks for helm v4), and `--endpoint` is required. Here is the lifecycle, dry-run first:
```bash
go install github.com/kakkoyun/zeroins/cmd/
[email protected]
# Review the plan first (changes nothing)
kubectl obi attach --dry-run --endpoint=https://otel-collector.observability.svc:4318
# Instrument everything on every node, detach automatically after 15 minutes
kubectl obi attach --duration=15m --endpoint=https://otel-collector.observability.svc:4318
# Check what's being instrumented
kubectl obi status --all-namespaces
# Stop instrumenting
kubectl obi detach
```
The endpoint must be an OTLP HTTP(S) base URL, so no `/v1/traces` suffix. In DaemonSet mode the plugin installs the pinned OBI chart (0.10.0, OBI v0.10.0, the release pinned for the talk on 13 August 2026) into the `obi-system` namespace with Helm. Sidecar mode also exists; it patches one deployment and restarts it, so the "no restart" promise holds only for the DaemonSet. Sidecar mode needs its own `kubectl obi detach <deployment> --mode=sidecar`; a bare `detach` removes only the DaemonSet. The whole thing is experimental: CI runs unit tests and Helm contract checks, and the real-eBPF integration test on a disposable K3s cluster last passed shortly before the v0.2.0 tag.
Handing an agent a command that attaches privileged eBPF programs to a cluster is exactly as fun as it sounds, hence the guardrails. The messy operational checks deserve names: kernel version, capabilities, collector endpoint, rebuild access, Go version, and span depth. The skill does not silently attach eBPF to production. It produces a plan first (`attach --dry-run` prints the cluster context, the privileges, the exact commands and the rendered values), then stops at a confirmation gate until a human approves. Exploratory attaches take `--duration` and detach themselves, and `zeroins sessions list` and `zeroins sessions reap` show and clean up what is still attached. Boring, reversible, logged.
Nothing in the try-out touches a cluster. First, a local trace backend for the otelc example above, using [Jaeger](https://www.jaegertracing.io/docs/2.21/getting-started/). Jaeger only accepts traces, so the metrics flush logs a 404 when the process shuts down; the traces arrive anyway:
```bash
docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:2.21.0
# build and run any small net/http service as shown in "What otelc covers", send a few requests, then:
curl -s localhost:16686/api/v3/services
```
The services list should include `my-go-service`. Then the zeroins side, which only looks things up and prints a plan:
```bash
zeroins obi lookup net/http
zeroins otelc lookup net/http
kubectl obi attach --dry-run --endpoint=https://otel-collector.observability.svc:4318
```
The two lookups say which route covers `net/http`, and the dry-run prints the plan a human would be asked to approve. Versions, links and commands checked on 2 October 2026.
Where to go from here: back to the loop. The bug that won't reproduce locally used to cost a log line and a redeploy, then another when the first guess landed in the wrong place. Now the first move is the boring question from the top. If the service has to stay untouched, the plan is one dry-run away and, in DaemonSet mode, nothing restarts. If the answer lives in our own functions, the question sends us to otelc, and the rebuild carries no source edit. That is the series; the [talk page](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/) has the slides and links, and the [FOSDEM 2026 post](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/) has the write-up of the earlier version of the talk. To help out, OBI, otelc and the eBPF profiler each have an OpenTelemetry SIG (see the [community repository](https://github.com/open-telemetry/community)), and Ollygarden's [agent skills](https://github.com/ollygarden/opentelemetry-agent-skills) are a good next stop for more OpenTelemetry skills.
## FAQ
### Should I use OBI or otelc for my Go service?
If the service has to stay untouched (already running, rebuild or redeploy unacceptable), OBI is the default. If you control the build and need function-level spans, otelc is the default. The tiebreaker: can it be rebuilt and redeployed, or does it have to stay untouched?
### Why can't Go use LD_PRELOAD auto-instrumentation like Java?
A pure-Go build is static, so `LD_PRELOAD` has nothing to attach to. The OpenTelemetry host injector preloads agents for Java, Node.js, .NET, Ruby and Python (disabled by default), not Go. The Operator's `inject-go` annotation runs a per-pod sidecar with `go.opentelemetry.io/auto` instead, off by default, needing a privileged root container.
### Which port does otelc export to, 4317 or 4318?
1. otelc exports OTLP over HTTP (`http/protobuf`) by default, and 4317 is the gRPC port; pointing the default protocol at 4317 produces `malformed HTTP response` errors. To use 4317, set `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.
### What Go version does otelc need?
Go 1.25 or newer. OBI, in turn, needs a BTF-enabled Linux kernel and a specific capability set.
### How do I attach OBI in Kubernetes safely?
Review the dry-run first: `kubectl obi attach --dry-run --endpoint=<OTLP HTTP(S) base URL>`. Then attach with `--duration=15m` so it detaches itself, check `kubectl obi status --all-namespaces`, and `kubectl obi detach` when done. It deploys privileged eBPF observers and is not a production installer.
### Can an AI agent attach eBPF instrumentation on its own?
The `collect-go-telemetry` skill produces a plan first (`attach --dry-run` prints the cluster context, the privileges, the exact commands and the rendered values), then stops at a confirmation gate until a human approves. `zeroins sessions list` and `zeroins sessions reap` show and clean up what is still attached.
### How do I check whether OBI or otelc covers a given Go library?
`zeroins obi lookup <library>` and `zeroins otelc lookup <import-path>` return only the relevant rows. They are offline, reading a snapshot embedded in the binary (OBI v0.10.0 and otelc v1.0.1 in zeroins v0.2.0), so they change only when zeroins does.
### Does OBI double-instrument services that already have an OTel agent?
OBI skips services it sees publishing their own OpenTelemetry telemetry (`exclude_otel_instrumented_services`, on by default), so an OTel Java or Python agent shouldn't double up. The docs describe this for OpenTelemetry SDK calls, so test a pod with a vendor agent first.
## Related
- [[resources/documentation/Go/Instrumentation/Go Instrumentation|Go Instrumentation]] (series hub)
- [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched|Why Go Cannot Be Monkey Patched]] (part 1: the process-start injector background)
- [[resources/documentation/Go/Instrumentation/Go Instrumentation - OBI eBPF Auto-Instrumentation|OBI eBPF Auto-Instrumentation]] (part 2: what OBI instruments in Go and what it requires)
- [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation|otelc Compile-Time Instrumentation]] (part 3: the compile-time tool)
- [[resources/documentation/Go/Instrumentation/Go Instrumentation - Continuous Profiling Without Code Changes|Continuous Profiling Without Code Changes]] (part 4: the eBPF profiler layer)
- [[resources/documentation/Go/Instrumentation/Go Instrumentation - Flight Recording and USDT|Flight Recording and USDT]] (part 5: what Go itself might add)
- [[resources/documentation/Go/Go - uretprobes|Go - uretprobes]] (related)
- [[resources/documentation/Go/Go - Internals|Go - Internals]] (related)
## Sources
- [How to Instrument Go Without Changing a Single Line of Code (talk, GopherCon UK 2026, GoLab 2026)](https://kakkoyun.me/talks/instrument-go-without-changing-a-single-line/)
- [How to Instrument Go Without Changing a Single Line of Code (series)](https://kakkoyun.me/series/how-to-instrument-go-without-changing-a-single-line-of-code/)
- [FOSDEM 2026: auto-instrumenting Go](https://kakkoyun.me/posts/fosdem-2026-auto-instrumenting-go/)
- [OpenTelemetry host injector README at v0.10.1](https://github.com/open-telemetry/opentelemetry-injector/blob/v0.10.1/README.md)
- [OpenTelemetry host injector design doc at v0.10.1](https://github.com/open-telemetry/opentelemetry-injector/blob/v0.10.1/DESIGN.md)
- [OpenTelemetry Operator: automatic Go instrumentation](https://opentelemetry.io/docs/platforms/kubernetes/operator/automatic/)
- [OBI support matrix at v0.10.0](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/blob/v0.10.0/SUPPORT_MATRIX.md)
- [otelc: compile-time Go instrumentation docs](https://opentelemetry.io/docs/zero-code/go/compile-time/)
- [otelc getting-started guide at v1.1.0](https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/blob/v1.1.0/docs/getting-started.md)
- [zeroins repository at v0.2.0](https://github.com/kakkoyun/zeroins/tree/v0.2.0)
- [zeroins: collect-go-telemetry skill at v0.2.0](https://github.com/kakkoyun/zeroins/blob/v0.2.0/skills/collect-go-telemetry/SKILL.md)
- [Jaeger 2.21 getting started](https://www.jaegertracing.io/docs/2.21/getting-started/)
- [OpenTelemetry community repository](https://github.com/open-telemetry/community)
- [Ollygarden OpenTelemetry agent skills](https://github.com/ollygarden/opentelemetry-agent-skills)