Talks

A benchmark moved that could not have moved. At Datadog our code runs inside the customer’s process: a false red wastes our time, a false green costs the customer CPU. This talk is the set of checks we run before we believe a number, and each check earns its place through an incident. Every section opens on a real incident and asks one of three questions. Is the benchmark measuring real work? Is the sample stable enough? Is the difference large relative to the noise? On a laptop, the compiler deletes the work you meant to measure and a single run is a point. In CI, one workload is not representative and shared runners move under you. No maths on stage: each section says why a check matters and ends with one action you can take on Monday. ...

A production bug, a service you cannot touch, and a debugging loop measured in deploys. This talk follows one Go process through its lifecycle and finds an instrumentation point at each station: at build time, at process start, and in the kernel. At build time, otelc rewrites the code during compilation, so the binary ships already instrumented. At process start, the OpenTelemetry injector loads an agent into the process without a rebuild. In the kernel, OBI attaches eBPF probes to a static Go binary and collects traces and metrics from outside the process. Along the way we pick up CPU profiles, see where request correlation across goroutines and connections breaks down, and when the baseline still cannot answer the question, a live debugging probe captures the missing logs and variables within explicit limits. ...

The debugging loop is slow. You cannot reproduce the bug locally, so you add a log line and redeploy. Wrong place. You do it again. An agent alone does not fix this: it still has to pick a mechanism, and it has to know what that mechanism costs. This talk is about three approaches that take the source edit out of that loop. Each acts at a different point in the software lifecycle, and each inherits the strengths and constraints of its intervention point. ...

A benchmark is a measurement system. It can report a precise number while measuring removed work, an unstable sample, or an uncontrolled machine. This talk argues you should trust a Go benchmark only after answering three questions: Is the compiler measuring real work? Is the sample stable enough? Is the difference large relative to the noise? We work through three layers. Compiler honesty covers dead-code elimination, sinks, constant folding, inlining, timer ordering, and the B.Loop construct, which manages the timer for you and keeps the loop body from being optimized away. Statistical interpretation covers repeated samples, benchstat, coefficient of variation, run-count discipline, and the p-hacking traps that inflate false positives. Environment control covers both local machines and CI: Linux frequency and isolation controls, perflock, benchdiff, and the bare-metal runners that shared CI instances are not. ...

Jonathan Hall and Shay Nehmad had me on Cup o’ Go for episode 167, straight after their GopherCon US recap. We spent about forty minutes on instrumentation: what an APM tool is actually measuring, and the three ways you can get telemetry out of a Go program without asking every team to rewrite their handlers. The part I enjoyed arguing about most was blast radius. Manual instrumentation is precise and tedious, and it never finishes. eBPF buys you one agent covering every language on the box, at the cost of running code in the kernel, where a bad program can take the whole host down with it. Compile-time instrumentation rewrites the syntax tree behind go build through the -toolexec hook, which keeps failures scoped to one service and works anywhere Go compiles. People love this dark magic, which I find either reassuring or alarming depending on the day. ...