Observability

Measuring software performance reliably is remarkably difficult. It’s a specialized version of a more general problem: trying to find a signal in a world full of noise. A benchmark that reports a 5% improvement might just be measuring thermal throttling, noisy neighbors, or the phase of the moon. In this talk, we walk through the full stack of reliable performance measurement — from controlling your benchmarking environment (bare metal instances, CPU affinity, disabling SMT and dynamic frequency scaling) to designing benchmarks that are both representative and repeatable. We cover the statistical methods needed to interpret results correctly (hypothesis testing, change point detection) and show how to integrate continuous benchmarking into development workflows so regressions are caught before they reach production. ...

Profiling Python with eBPF: A New Frontier in Performance Analysis Profiling Python applications can be challenging, especially in scenarios involving high-performance requirements or complex workloads. Existing tools often require code instrumentation, making them impractical for certain use cases. Enter eBPF (Extended Berkeley Packet Filter)—a versatile Linux technology—and the open-source project Parca, which together are reshaping how Python profiling works. I’ll cover how eBPF enables continuous profiling, the challenges of stack unwinding in Python, and what modern profiling tools can do with all of this. ...

Most Python profilers ask you to decide in advance. You import something, or you wrap the process, or you restart it with a flag. That works on your laptop and helps little at 3 a.m., when the profile you want is of a process that has already been running for six hours. eBPF changes the bargain. An agent samples from the kernel on a timer, and the application never knows it is there. For compiled code that is close to free: read the registers, walk the frames, resolve the addresses against DWARF. Python is where it stops being easy. ...

Prometheus is the second-oldest project in the CNCF and the default answer for metrics in Kubernetes. Which means most people meet it already running, configured by someone who has since left, and never get a chance to ask how the thing works. This is the maintainer track session for people in that position. We start from the beginning for anyone who has only ever touched Prometheus through a Grafana panel: what it scrapes, what it keeps, and what the query engine is actually doing while your dashboard spins. Then we go under the covers into the storage layer, the write path, and the places the memory tends to go. ...

Observability is crucial for understanding how your application operates in real-time. Among various observability signals—such as logs, traces, and continuous profiling—metrics play a significant role. They provide sampled measurements throughout the system, essential for ensuring service quality, improving performance, scalability, debuggability, security, and enabling real-time, actionable alerting. Building observable applications begins with proper instrumentation. While Prometheus tooling simplifies this process, there are still numerous opportunities for mistakes or misuse. In this talk, Jéssica Lins and Kemal Akkoyun present several useful patterns, best practices, and idiomatic methods for instrumenting critical services. They discuss common pitfalls, failure cases, and instrumentation strategies, sharing valuable insights and methods to avoid these mistakes. Additionally, they provide tips for writing simple, maintainable, and robust instrumentation facilities using real-life examples. The talk also demonstrates how to enrich metrics by correlating them with other observability signals and discusses how to best use recent changes in client_golang, the Go client library for Prometheus. ...