Prometheus

Every now and then a project surprises you by remembering something you said years ago. This week was one of those weeks. A talk I gave at PromCon Online 2020 — The Zen of Prometheus — has quietly become part of the official Prometheus documentation. I am still sitting with it. Where it started The talk was born in the strangest year of my career. PromCon 2020 was online, like everything else. I was a few years deep into running Prometheus in anger, collecting scars from instrumenting services that didn’t want to be instrumented and writing alerts that kept me up at night for the wrong reasons. I wanted a way to package those lessons that wasn’t another forty-slide deck of bullet points. ...

Brussels Again, But Make It Unplugged The day after FOSDEM, about a hundred of us gathered at Sparks Meeting on Rue Ravenstein in Brussels for OTel Unplugged EU 2026 — an unconference dedicated entirely to OpenTelemetry. Purple stage lights, a mid-century auditorium with wood paneling, and the familiar buzz of people who spend their days thinking about telemetry pipelines. If you know, you know. The format is simple: no prepared talks, no slides. Morning session brainstorming, dot-voting on topics, then self-organizing into nine rooms across four breakout slots. You vote with your feet. If a conversation isn’t working, you move. It’s chaotic, it’s honest, and it produces the kind of discussions that polished conference talks rarely achieve. ...

Observability is crucial for understanding how your application operates in real-time. Among various observability signals—such as logs, traces, and continuous profiling—metrics play a significant role. They provide sampled measurements throughout the system, essential for ensuring service quality, improving performance, scalability, debuggability, security, and enabling real-time, actionable alerting. Building observable applications begins with proper instrumentation. While Prometheus tooling simplifies this process, there are still numerous opportunities for mistakes or misuse. In this talk, Jéssica Lins and Kemal Akkoyun present several useful patterns, best practices, and idiomatic methods for instrumenting critical services. They discuss common pitfalls, failure cases, and instrumentation strategies, sharing valuable insights and methods to avoid these mistakes. Additionally, they provide tips for writing simple, maintainable, and robust instrumentation facilities using real-life examples. The talk also demonstrates how to enrich metrics by correlating them with other observability signals and discusses how to best use recent changes in client_golang, the Go client library for Prometheus. ...

Metrics are powerful tools in the cloud-native space, enabled by Prometheus. However, using facilities to enable monitoring requires instrumenting the code. Everyone wants observability, but nobody wants to go the extra mile to instrument their clusters or applications. This is where eBPF comes in. eBPF, a promising technology for observability tooling, is not news. To observe the infrastructure and applications, eBPF-based system-wide agents can help us to capture events without requiring recompilation or redeployment of applications. In this talk, attendees will discover alternative ways to collect metrics from applications and infrastructure using system-wide eBPF agents. The presenters will demonstrate what level of observability could be accomplished without instrumentation. ...

Live Website: The Zen of Prometheus In modern days, we run our applications as loosely coupled microservices on distributed, elastic infrastructure as (mostly) stateless workloads. Under these circumstances, observability is the key to understanding how our applications run and behave in action to deliver highly available and resilient service. Prometheus is born in such an atmosphere as a solution to satisfy the observability needs of the cloud-native era. Among many other observability signals like logs and traces, metrics play the most substantial role. Sampled measurements observed throughout the system are crucial for monitoring the health of the applications and they enable real-time, actionable alerting. Although Prometheus tooling makes life a lot easier, there are still numerous possibilities to make mistakes or misuse them. ...