# Go Instrumentation - otelc Internals otelc is the OpenTelemetry compile-time instrumentation tool for Go. Its hook templates and AST rewrites fail in ways where the woven program still compiles but behaves differently: double execution, wrong evaluation order, and silently skipped rules. The build is equally constrained: the bundle is byte-reproducible only with the repo-pinned toolchain, shared caches corrupt builds, and nested invocations can deadlock. ## Key points - The before-hook trampoline recovers a hook panic and still runs the original function body. `SetSkipCall(true)` before the panic has no effect, because the named `skipCall` result is assigned only by the final return. A hook that serves the request itself makes a panicking handler run twice and hides the panic from the caller. - A `wrap_call` rule that moves the call into a closure changes when Go evaluates the arguments. A fasthttp rule that read `conn` inside the goroutine crashed in 5 of 5 probe runs when a later `conn = nil` dereferenced nil. Evaluate the call arguments at the statement, as the net/http client rules do. - Alias rewriting runs after the original application expression is inserted, so it traverses application code too: the woven program can compile and call the wrong code, and a file's import alias can bind to a shadowing parameter. - `applyCallReplace` deep-clones the replacement, and the cloned selector gets no entry in the parser's node map, so `FindPosition` returns an empty position and later `method_call` rules silently skip the call. Two rules that wrap one call leave only the first. - An iota-cascade guard that gates on the explicit `kind` field misses const specs where `kind` is omitted, because the AST finder still returns the spec. Rewriting `B` in `const ( A = iota; B; C )` silently changes `C` from 2 to 99. - dave/dst leaves the `Path` of same-package identifiers empty, because `ResolveLocalPath` is not set, so a `function-call` aspect cannot target an unqualified local call with a qualified path. Use the bare symbol name under `all-of: import-path`. Rendering one AST node twice in a template panics the restorer with a duplicate-node error, so `.Copy` the node for the second use. - `add-struct-field` has no precedent for direct generic instantiations. Use a named wrapper type around the generic, for example a struct that embeds `atomic.Pointer[T]`: it weaves identically to existing named types. Verify empirically before you commit. - `closeEarlyDBIdle` in the database/sql early-connection shim closes the live idle connections of every DB opened before init: `:memory:` sqlite data, temp tables, `SET` state, and prepared statements are lost, and a connection checked out during init stays untraced. `SELECT 1` and counting-stub tests cannot catch it. - A blanket init-order guard regresses deliberately init-safe hooks. The logrus instrumentation initializes its bookkeeping lazily, because the hook can run before the package's initializers, and skipped calls are never replayed, so loggers created during init silently lose trace correlation. Init-safe hooks need an explicit rule policy. - The committed `tool/data/otelc-bundle.tgz` is not byte-reproducible across Go toolchains: the gzip layer differs by about 7 KB between a go1.27.3 build and the pinned go1.26.10, while the raw tar stream stays identical, so local content checks pass and CI's byte-comparing Verify Bundle fails. Regenerate with the toolchain that CI resolves from `.tools/go.mod` (`GOTOOLCHAIN=go1.26.10 gmake package`): a newer local toolchain, or a stale `.bin/bundle` built by one, produces bytes that Verify Bundle rejects, and the pinned result is deterministic on re-run. If platform bytes still differ, the `/regenerate-bundle` PR comment makes CI regenerate the bundle. - Bundle-in-sync recovery per CONTRIBUTING: `make setup-git` registers the `otelc-bundle` merge driver, then rebase onto `upstream/main`, `make package`, amend, force-push. Only the binary bundle conflicts. The `.gitattributes` merge driver keeps the current bundle, so a merge proceeds and then needs regeneration. GitHub reports merge conflicts on PRs whose source merges cleanly under `git merge-tree`: the unblock path is bundle regeneration, not manual conflict resolution, and a main advance that touches the bundle forces a re-merge plus regeneration. - Use `gmake` (GNU make 4.x), not the macOS system make 3.81, for targets that use `.ONESHELL` or package the bundle. Pin the toolchain for lint too: local go 1.27.3 formats differently than the pinned toolchain and makes golangci-lint panic on export data. Toolchain mismatch is this repo's standing "CI red, local green" cause. - Go's build cache key (`buildActionID`) never hashes `-toolexec`, and the link importcfg is populated by a compile-time side effect that a cache hit skips, so `-a` was load-bearing. Cross-app cache sharing corrupts builds when two apps that import but do not instrument the same package collide on a shared cache. Per-app cross-run caching is the only safe shape. - `syncDeps` is the only writer of the `replace` directives for the unpublished embedded modules and runs solely on the generate path, so a committed instrumentation tool file fails at import resolution with "instrumentation/basic is not part of a module". CI stayed green only because no test app commits a tool file: the untested path is exactly the one the docs advertise. - `EnableNestedToolexec` puts `-toolexec` into `GOFLAGS`, so the nested `go list -export -deps` inside import-config updating already resolves `add_file` imports on unmodified main: a PR can fix a scenario that its own e2e test cannot reach. - Lock renaming covers module and workspace entries only, so a nested child whose work directory equals the parent's module root takes the parent's module lock as its never-renamed work-dir lock, and each invocation waits for the other. - Module drift breaks unrelated PRs twice over: a fixed module list in the dependency-update tool lets MVS select a version whose tidy rewrites a module's go.mod and fails the tidy check, and merging upstream/main can pull in an `otelc.yaml` `version:` line that breaks `TestPublishedRuleFilesCompatibleWithReleasedOtelc` until it is stripped and `make package` re-runs. A clean conflict check does not mean a clean guard test: run the merge itself. - In otelc CI, `make all` runs build, format, lint, and test. Run the strict Go linter on just the changed packages first: it surfaces most findings before the heavier targets. `otelc go test -race -count=1` runs the race detector under the otelc toolchain. - `otelc go test -json` keeps stdout a pure test2json stream, valid for `jq`: setup messages and stack traces go to stderr on both success and failure, so pipe stdout to gotestsum and keep stderr for the log. - Any `otelc.yaml` rule-target change needs `make manifest` before commit, or `make verify-manifest` fails at the head: the Check Consistency CI workflow runs exactly that check. Run it at the PR head before you approve. - `otelc go` uses `SkipFlagParsing`: global flags placed after the `go` subcommand reach the Go toolchain verbatim. Put global flags before the subcommand. - otelc (open-telemetry/opentelemetry-go-compile-instrumentation) is not a donation of the earlier Orchestrion tool. A new OTel SIG built it from scratch with the same `-toolexec` plus AST-rewriting mechanism in a separate codebase. v1.0.1 (2026-07-14) is the effective first stable release, because v1.0.0 was retracted the same day. - Flat rule schemas, where matching and modification keys coexist at the top level, become ambiguous as the rule set grows. Real-world `orchestrion.yml` files show it: matching predicates (`function-pattern`, `import-path`) interleave with modification directives (`inject-before`, `inject-after`). Group matching under `where` and modification under `do`: the two concerns stay explicit, and each group extends without collision. ## How to apply 1. Export the repo-pinned toolchain and use GNU make: `export GOTOOLCHAIN=go1.26.10`, then `gmake` for every otelc target. 2. Regenerate the bundle with `GOTOOLCHAIN=go1.26.10 gmake package`. If bytes still differ, ask for `/regenerate-bundle` on the PR. 3. Put otelc global flags before the `go` subcommand: `otelc --rules ... go build`. 4. Run the strict Go linter on the changed packages first, then `make all`. Pipe `otelc go test -json` stdout to gotestsum, and keep stderr for the log. 5. After an `otelc.yaml` rule-target change, run `make manifest`, then `make verify-manifest` at the PR head. 6. Give each new rewrite pattern an adversarial test: double execution, argument evaluation order, and rules that silently skip. 7. Keep a nested child's work directory off the parent's module root, or the invocations deadlock. ## Observations - [codegen] The before-hook trampoline can run a panicking handler twice and hide the panic. #otelc (source: agent sessions, 2026-10) - [toolchain] The bundle is byte-reproducible only with the pinned Go toolchain. #ci (source: agent sessions, 2026-10) - [caching] The build cache key never hashes `-toolexec`, so `-a` was load-bearing. #go-build (source: agent sessions, 2026-07) - [schema] Group rule matching under `where` and modification under `do`. #yaml (source: agent sessions, 2026-03) - [review] `make verify-manifest` at the PR head before you approve rule-target changes. #otelc (source: agent sessions, 2026-11) ## Relations - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation]] (parent index) - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation]] (same tool, usage and mechanism background) - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation - Why Go Cannot Be Monkey Patched]] (why compile-time rewriting is the only hook point in Go) ## Sources Personal agent-session notes, March 2026 to November 2026