Observability

The debugging loop is slow. You cannot reproduce the bug locally, so you add a log line and redeploy. Wrong place. You do it again. An agent alone does not fix this: it still has to pick a mechanism, and it has to know what that mechanism costs. This talk is about three open-source approaches that take the rebuild out of that loop. Each acts at a different point in the software lifecycle, and each inherits the strengths and constraints of its intervention point. ...

A benchmark is a measurement system. It can report a precise number while measuring removed work, an unstable sample, or an uncontrolled machine. This talk argues you should trust a Go benchmark only after answering three questions: Is the compiler measuring real work? Is the sample stable enough? Is the difference large relative to the noise? We work through three layers. Compiler honesty covers dead-code elimination, sinks, constant folding, inlining, timer ordering, and the B.Loop construct that fixes the non-terminating-timer case. Statistical interpretation covers repeated samples, benchstat, coefficient of variation, run-count discipline, and the p-hacking traps that inflate false positives. Environment control covers both local machines and CI: Linux frequency and isolation controls, perflock, benchdiff, and the bare-metal runners that shared CI instances are not. ...

Note This post was originally published on the OpenTelemetry blog on 2026-07-16. I wrote it for the OpenTelemetry community; this is a mirror on my personal blog with some added commentary. Why this milestone matters to me I have been involved with the Go Compile-Time Instrumentation SIG since it kicked off at the start of 2025, when Alibaba and Datadog decided to pool their efforts into a single vendor-neutral project. Watching it reach v1 last week felt genuinely satisfying — not because “v1” is a magic number, but because it marks the point where we stopped calling it experimental and started calling it something people should actually run in their builds. ...

Every now and then a project surprises you by remembering something you said years ago. This week was one of those weeks. A talk I gave at PromCon Online 2020 — The Zen of Prometheus — has quietly become part of the official Prometheus documentation. I am still sitting with it. Where it started The talk was born in the strangest year of my career. PromCon 2020 was online, like everything else. I was a few years deep into running Prometheus in anger, collecting scars from instrumenting services that didn’t want to be instrumented and writing alerts that kept me up at night for the wrong reasons. I wanted a way to package those lessons that wasn’t another forty-slide deck of bullet points. ...

A Loose Cable That Broke Physics In 2006, a team of physicists began building the OPERA experiment — a 730-kilometer underground tunnel from CERN in Switzerland to Gran Sasso in Italy, designed to measure the speed of neutrinos. Five years of construction. Roughly 100 million euros. The most rigorous experimental physics on the planet. In September 2011, the results came back. Neutrinos were traveling faster than the speed of light. The team had just broken the laws of physics. ...