Observability

We’re going to put a probe on one small Go function, twice. First the way OBI (OpenTelemetry eBPF Instrumentation) does it today, then the way a USDT site would. On the way we’ll price a single probe hit, and find that USDT’s pitch is not a discount. This is a companion to the How to Instrument Go Without Changing a Single Line of Code series, between Part 2 and Part 3. The series accompanies my GopherCon UK talk. Part 2 showed what OBI sees. Here we look at how it gets its probes onto Go code, and why USDT keeps coming up. Our function is below. The //go:noinline keeps the compiler from folding it into main, so there is something left to probe. ...

Take one HTTP request, GET /orders/42, headed for a Go service nobody is going to rebuild this quarter. We want a span for it, another for its database call, and the trace context handed on to the next service, all without touching the service. “Zero code changes” travels far in conference talks and vendor docs. Today we watch that one request from the kernel and find out where the phrase holds. ...

Start a Java service with one extra flag and it gets traced, with no source change and no rebuild: 1 java -javaagent:agent.jar -jar app.jar The JVM offers each class’s bytes to the agent’s ClassFileTransformer before it defines the class, so the agent can rewrite a method on its way in. Python lets you swap a method at runtime, which is the monkey-patching of the title. That flag is Java’s agent slot, and Python doesn’t even need one. Go has no such slot, and that follows from how Go was designed. ...

Every now and then a project surprises you by remembering something you said years ago. This week was one of those weeks. A talk I gave at PromCon Online 2020 — The Zen of Prometheus — has quietly become part of the official Prometheus documentation. I am still sitting with it. Where it started The talk was born in the strangest year of my career. PromCon 2020 was online, like everything else. I was a few years deep into running Prometheus in anger, collecting scars from instrumenting services that didn’t want to be instrumented and writing alerts that kept me up at night for the wrong reasons. I wanted a way to package those lessons that wasn’t another forty-slide deck of bullet points. ...

A Loose Cable That Broke Physics In 2006, a team of physicists began building the OPERA experiment — a 730-kilometer underground tunnel from CERN in Switzerland to Gran Sasso in Italy, designed to measure the speed of neutrinos. Five years of construction. Roughly 100 million euros. The most rigorous experimental physics on the planet. In September 2011, the results came back. Neutrinos were traveling faster than the speed of light. The team had just broken the laws of physics. ...