A sponsored keynote, which is a genre that usually means a product pitch. This one is about how Red Hat ran Prometheus across a large fleet, and why the patches we needed went upstream rather than into a fork we would have to carry forever.

Past a certain scale, Prometheus stops being one binary you put on a box. It becomes a set of components you assemble, with Thanos or something like it behind it, and the interesting problems move from “how do I scrape this” to how you keep the whole assembly cheap, queryable, and boring to operate. Same ideas, more moving parts.

Upstream-first is the constraint that shapes the rest. Carrying a patched fork is faster this quarter and more expensive every quarter after it, so the work went into the projects themselves. The talk covers what that looked like in practice, and how the same instinct showed up in the other projects we ran alongside Prometheus.

This is from May 2021 and reflects the Red Hat setup of the time, so read the specifics as a snapshot.

Recording

Links

Events