Six hours. I had planned to spend thirty minutes on a proof-of-concept. Engineers overrun estimates all the time. But this wasn’t a planning failure, it was something I’d spent my whole adult life recognizing too late: hyperfocus. The kind where time doesn’t slow down, it simply stops existing. My coffee went cold. I missed lunch. I blew past my 6pm hard stop with more working code than I’d shipped in the previous three days combined. ...
At 9:14 a.m. on a Wednesday, I open the laptop. The daily briefing for today is already there. Above the fold: a Whoop recovery score of 62 (fine), a Wakatime row showing 4h 21m on the OpenTelemetry Go compile-time instrumentation repo yesterday, three GitHub PRs waiting on review, and a Things 3 task that has been sliding forward for nine days: Review that thing from X. Each previous day the task migrated, and the journal entry from that day is one click away. The reason it kept slipping is in those entries somewhere. ...
Jonathan Hall and Shay Nehmad had me on Cup o’ Go for episode 167, straight after their GopherCon US recap. We spent about forty minutes on instrumentation: what an APM tool is actually measuring, and the three ways you can get telemetry out of a Go program without asking every team to rewrite their handlers. The part I enjoyed arguing about most was blast radius. Manual instrumentation is precise and tedious, and it never finishes. eBPF buys you one agent covering every language on the box, at the cost of running code in the kernel, where a bad program can take the whole host down with it. Compile-time instrumentation rewrites the syntax tree behind go build through the -toolexec hook, which keeps failures scoped to one service and works anywhere Go compiles. People love this dark magic, which I find either reassuring or alarming depending on the day. ...
You can’t reproduce the bug locally, so you add a log line and redeploy. Wrong place. You add another one and redeploy again. That is the debugging loop from my GopherCon UK talk, and in this last part we’re going to shorten it: a decision rule for two zero-touch tools, then a runbook an AI coding agent (not a Java-style one) can execute. The runbook starts with one boring question: is this Go service already running, or can I rebuild it? If the service has to stay untouched, OBI (OpenTelemetry eBPF Instrumentation) is the default. If I can rebuild it and need function-level spans, otelc (OpenTelemetry Go compile-time instrumentation) is the default. The rest of the decision tree is checking privileges, Go version, collector endpoint, and whether RED metrics are enough. ...
A benchmark bot once told me that one of my pull requests made a benchmark 6–9% slower. A same-machine comparison said the pull request made it faster. Both results were stable, and they disagreed. In this last part we’ll find out how that happens, and leave with three questions and three small Go tools that tell us how far to trust a number. A loose cable Physicists have been fooled the same way, at a much larger scale. In September 2011, the OPERA collaboration announced that muon neutrinos appeared to travel faster than the speed of light. Months of rechecking found nothing wrong. The root cause, eventually, was an improperly seated fibre-optic connector in the GPS timing chain, which introduced a ~73 ns bias that made neutrinos appear to arrive early (Science called it a loose cable). A second fault, an oscillator defect, pushed the other way and partly masked the first. Once both were corrected, the 2012 re-measurements showed neutrino speed consistent with the speed of light. ...