Observability

Another Year, Another FOSDEM FOSDEM — the annual Brussels pilgrimage. If you’ve been, you know the drill: too many talks, too little time, questionable coffee, and the kind of conversations that only happen when you pack thousands of open-source developers into a university campus in the dead of winter. This year was different for me, though. Two talks in two devrooms, three sessions at OTel Unplugged — and this time, I brought the whole family. My wife and our toddler (who has graduated from “can barely walk” to “can absolutely destroy a hotel room in under four minutes”) came along, and we turned it into a proper trip — FOSDEM, then a few days exploring Ghent and Antwerp before heading home. ...

Zero-touch observability for Go is finally becoming real. In this talk, we walk through the different strategies you can use to instrument Go applications without changing a single line of code, and what they cost you in terms of overhead, stability, and security. We compare several concrete approaches and projects: eBPF-based auto-instrumentation using OpenTelemetry’s Go auto-instrumentation agent and OBI (OpenTelemetry eBPF Instrumentation), compile-time manipulation using tools like Orchestrion and the OpenTelemetry Compile-Time Instrumentation SIG, runtime injection via Frida/ptrace, and USDT (User Statically-Defined Tracing) probes — both via libstapsdt and a custom Go runtime fork. ...

Measuring software performance reliably is remarkably difficult. It’s a specialized version of a more general problem: trying to find a signal in a world full of noise. A benchmark that reports a 5% improvement might just be measuring thermal throttling, noisy neighbors, or the phase of the moon. In this talk, we walk through the full stack of reliable performance measurement — from controlling your benchmarking environment (bare metal instances, CPU affinity, disabling SMT and dynamic frequency scaling) to designing benchmarks that are both representative and repeatable. We cover the statistical methods needed to interpret results correctly (hypothesis testing, change point detection) and show how to integrate continuous benchmarking into development workflows so regressions are caught before they reach production. ...

Profiling Python with eBPF: A New Frontier in Performance Analysis Profiling Python applications can be challenging, especially in scenarios involving high-performance requirements or complex workloads. Existing tools often require code instrumentation, making them impractical for certain use cases. Enter eBPF (Extended Berkeley Packet Filter)—a versatile Linux technology—and the open-source project Parca, which together are reshaping how Python profiling works. I’ll cover how eBPF enables continuous profiling, the challenges of stack unwinding in Python, and what modern profiling tools can do with all of this. ...

Observability is crucial for understanding how your application operates in real-time. Among various observability signals—such as logs, traces, and continuous profiling—metrics play a significant role. They provide sampled measurements throughout the system, essential for ensuring service quality, improving performance, scalability, debuggability, security, and enabling real-time, actionable alerting. Building observable applications begins with proper instrumentation. While Prometheus tooling simplifies this process, there are still numerous opportunities for mistakes or misuse. In this talk, Jéssica Lins and Kemal Akkoyun present several useful patterns, best practices, and idiomatic methods for instrumenting critical services. They discuss common pitfalls, failure cases, and instrumentation strategies, sharing valuable insights and methods to avoid these mistakes. Additionally, they provide tips for writing simple, maintainable, and robust instrumentation facilities using real-life examples. The talk also demonstrates how to enrich metrics by correlating them with other observability signals and discusses how to best use recent changes in client_golang, the Go client library for Prometheus. ...