Observability

Profiling Python with eBPF: A New Frontier in Performance Analysis Profiling Python applications can be challenging, especially in scenarios involving high-performance requirements or complex workloads. Existing tools often require code instrumentation, making them impractical for certain use cases. Enter eBPF (Extended Berkeley Packet Filter)—a versatile Linux technology—and the open-source project Parca, which together are reshaping how Python profiling works. I’ll cover how eBPF enables continuous profiling, the challenges of stack unwinding in Python, and what modern profiling tools can do with all of this. ...

Observability is crucial for understanding how your application operates in real-time. Among various observability signals—such as logs, traces, and continuous profiling—metrics play a significant role. They provide sampled measurements throughout the system, essential for ensuring service quality, improving performance, scalability, debuggability, security, and enabling real-time, actionable alerting. Building observable applications begins with proper instrumentation. While Prometheus tooling simplifies this process, there are still numerous opportunities for mistakes or misuse. In this talk, Jéssica Lins and Kemal Akkoyun present several useful patterns, best practices, and idiomatic methods for instrumenting critical services. They discuss common pitfalls, failure cases, and instrumentation strategies, sharing valuable insights and methods to avoid these mistakes. Additionally, they provide tips for writing simple, maintainable, and robust instrumentation facilities using real-life examples. The talk also demonstrates how to enrich metrics by correlating them with other observability signals and discusses how to best use recent changes in client_golang, the Go client library for Prometheus. ...

The CNCF Incubated Thanos project with the large open-source community continues to push boundaries regarding observability and monitoring using Prometheus-based metrics. Together with the Prometheus community, it improves the metric story for Kubernetes clusters and beyond. Things like improved performance, better scalability, debuggability, security, metrics backfilling and query QoS is only the tip of the iceberg. As we know, observability nowadays comes in many flavours. Bunching them together is not a trivial side, given many shapes and collection points. Aside from metrics, we have logs, traces or even continuous profiling. In this talk, Kemal and Bartek, Thanos maintainers, after a quick overview of Thanos, will explain how Thanos can be integrated with those non-metric observability signals. The audience will learn an example, end-to-end ways to correlate multiple observability backends with Thanos for enhanced observability and monitoring experience. ...

Metrics are powerful tools in the cloud-native space, enabled by Prometheus. However, using facilities to enable monitoring requires instrumenting the code. Everyone wants observability, but nobody wants to go the extra mile to instrument their clusters or applications. This is where eBPF comes in. eBPF, a promising technology for observability tooling, is not news. To observe the infrastructure and applications, eBPF-based system-wide agents can help us to capture events without requiring recompilation or redeployment of applications. In this talk, attendees will discover alternative ways to collect metrics from applications and infrastructure using system-wide eBPF agents. The presenters will demonstrate what level of observability could be accomplished without instrumentation. ...

For years Google has consistently been able to cut down multiple percentage points in their fleet-wide resource usage every quarter, using techniques described in their “Google-Wide Profiling” paper. Ad-hoc profiling has long been part of the developer’s toolbox to analyze CPU and memory usage of a running process, however, through continuous profiling, the systematic collection of profiles, entirely new workflows suddenly become possible. Matthias and Kemal will start this talk with an introduction to profiling with Go and demonstrate via Conprof - an open-source continuous profiling project - how continuous profiling allows for an unprecedented fleet-wide understanding of code at runtime. Attendees will learn how to continuously profile Go code to help guide building robust, reliable, and performant software and reduce cloud spend systematically. ...