A benchmark moved that could not have moved. At Datadog our code runs inside the customer’s process: a false red wastes our time, a false green costs the customer CPU. This talk is the set of checks we run before we believe a number, and each check earns its place through an incident.

Every section opens on a real incident and asks one of three questions. Is the benchmark measuring real work? Is the sample stable enough? Is the difference large relative to the noise? On a laptop, the compiler deletes the work you meant to measure and a single run is a point. In CI, one workload is not representative and shared runners move under you. No maths on stage: each section says why a check matters and ends with one action you can take on Monday.

A war story from dd-trace-go ties the layers together: a benchmark measured code the pull request did not touch, the same-machine run reversed the CI sign, and code layout was the likely explanation for the small shift. The closing message is that benchmarking is an investment, in tooling and in discipline, and both Datadog and the Go team pay it. If you ship a library, a tool, or a service on a hot path, pay it too.

Slides

Links

  • golab-26 — both decks, the script, and captured results
  • public-content — the talk archive; this deck is indexed there
  • benchlab — the honestbench, benchgate, and benchenv CLIs

Events

Related