# Go Instrumentation - OpenTelemetry SDK and Bridge Pitfalls OpenTelemetry integration in Go fails in repeatable ways: no-op API imports, options configured but never wired, and hook lifecycles that leave spans unfinished. The `ddtrace/opentelemetry` bridge (the bridge) adds silent failures such as integer attributes that arrive as doubles. ## Key points - `StopIfEnabled` checked the wrapper `GetGlobalLoggerProvider()` while shutdown re-checked the concrete `globalLoggerProvider`. Two variables always set together create a TOCTOU gap and duplicate logging. One delegation point eliminates both. - A `sync.Once` reset is valid for re-initializable global providers. Mark the reset as intentional. - The tracer behind the bridge stores numeric span tags as `float64`, so the OTLP exporter emitted integer semconv attributes (`http.response.status_code`, `*.port`) as `doubleValue`, and `server.port` arrived as `8080.0`. Fix with an attribute-name allowlist plus `int64(value)`, exact to 2^53 under IEEE 754. - `WithTemporalitySelector()` set a config field that the provider constructor never read, so OTLP exporters always used the default temporality. The fix appends the option to both exporter option slices just before construction, and last-wins keeps user options winning. "Configured-but-not-wired" hit twice in one run. - Signal-specific OTLP env vars shadow the generic one. Gate a generic-value warning with `!p.IsSet("OTEL_EXPORTER_OTLP_METRICS_PROTOCOL")`, or the warning fires for a value that never reaches the exporter. `validateTraceProtocolVersion` accepts only `0.4` and `1.0`, and invalid values silently fall back. - The double root span in dd-trace-go issue #3369 is a layering bug, not a router bug. Build-time instrumentation wraps `http.Server.Serve` inside the library, and router middleware calls the same `httptrace.StartRequestSpan`, which auto-parents to the outer span. The outer root stamps a generic `error.msg` that masks the router's message, and both `http.request` spans double-count volume. - An attribute-loss report had two root causes. `httptrace.NewServeMux` with the OTel provider alone lost the same attributes. The write-through gap lives in the bridge, so the fix belongs in dd-trace-go. - Lost AppSec events under OTLP were a transport-capability mismatch: the tracer forwards the agent's `meta_struct` support answer even in OTLP mode, so WAF events land in `meta_struct["appsec"]`, which the OTLP exporter stringifies into an attribute the backend never reads. The "not force-kept" symptom was retention filtering. - The bridge violated OTel status precedence (OK > Error > Unset): a later `SetStatus(Error)` overwrote an earlier `Ok`. The fix uses a package-level `okStatus` map keyed by `weak.Pointer[tracer.SpanContext]` plus `runtime.AddCleanup`, so `Ok` keeps precedence across borrowed views without holding span references. - Under the experimental span pool, a retained borrowed view is not identity-stable: recycling resets `finished` and replaces the `SpanContext`, so the view can record again, expose another span's IDs, or end that span. The guard compares `ddSpan.Context()` against a view-creation snapshot. - A library that imports only `go.opentelemetry.io/otel/trace` is a documented no-op until the application wires the SDK, the `log/slog` pattern. The API pulls about 7 dependencies, mostly test-only, against about 16 for the SDK. pgx's zero-dependency `Tracer` interface is the gold standard. - `gen_ai.system` was removed from the semconv registry without formal deprecation. Use `gen_ai.provider.name`. `gen_ai.response.time_to_first_chunk` is a span attribute (double, seconds), not a metric. A registry check caught a wrong name and unit in one PR. - `MaxMessageBytes` in the Kafka exporter config is not enforced at the SDK level. The Kafka driver rejects oversized payloads. A restructuring enabled the fix, but the enforcement logic is still missing. - kafka-go's `(*kafka.Writer).WriteMessages` returns `kafka.WriteErrors` on partial batch failure: an `[]error` index-aligned with the input messages. Map each element to its own span, or successfully written messages are marked as failed too. The hook test covered only a 1-message batch. - moby's client never ends the `attach` otelhttp span after a 101 hijack. otelhttp ends spans in the wrapped body's `Close`, but the hijack path skips the body close (`fixLength` returns 0 for 1xx in Go 1.27). Closing is safe because hijack attaches `http.NoBody`. - In the `mcp-go` contrib, `llmobs.WithSessionID` at `StartToolSpan` time and the `mcp_session_id` tag set via `Annotate` are complementary: the option populates the session field that groups tool spans under their MCP session, the tag only powers span search. Only the tag leaves spans searchable but ungrouped. ## How to apply 1. Triage the bridge before the exporter or the router. Reproduce with the provider alone. If the symptom stays, fix the bridge. 2. Audit every option against the code that reads it. Wire options just before construction. 3. Assert that integer attributes serialize as integers and that `Ok` survives a later `SetStatus(Error)`. 4. Test batch hooks with multi-message batches and partial failures. Match each error to its message index. 5. Check the semconv registry for each attribute name, type, and unit before you submit a PR. ## Observations - [otel] One delegation for the wrapped global logger removes the TOCTOU gap and duplicate logging. #opentelemetry (source: agent sessions, 2026-01) - [data-model] Numeric tags stored as `float64` turn OTel integers into `doubleValue`. #otlp (source: agent sessions, 2026-07) - [wiring] `WithTemporalitySelector()` set a field that no constructor read. #otlp (source: agent sessions, 2026-07) - [transport] Lost AppSec events under OTLP were a `meta_struct` capability mismatch, not malformed data. #otlp (source: agent sessions, 2026-10) ## Relations - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation]] (parent index) - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation - otelc Compile-Time Instrumentation]] (stacked spans from build-time wrapping) - relates_to [[resources/documentation/Go/Instrumentation/Go Instrumentation - Flight Recording and USDT]] (`sync.Once` reset context) - relates_to [[resources/documentation/Open Telemetry - Prometheus]] (OTLP transport) ## Sources Personal agent-session notes, January 2026 to October 2026