Platform Monkey ctx

eBPF Observability: What Kernel-Level Telemetry Actually Buys You (and the Toil it Costs)

· By
Sources

The marketing for eBPF often frames it as "magic" — a way to get total system visibility with zero overhead and no application changes. This is a lie. eBPF is a powerful, restricted virtual machine that runs inside the kernel, and like any VM, it comes with a steep operational tax. To use it for observability, you aren't just "turning on a feature"; you are writing kernel-level code that must pass a rigorous static analysis check (the verifier) before it is allowed to execute.

This guide strips away the hype and looks at the actual mechanics: how a probe becomes a metric, why the verifier rejects your code with an opaque EINVAL, and why a "zero-overhead" probe can still crush your CPU if you use the wrong map type.

The BPF Machine: Lifecycle of a Probe

An eBPF observability pipeline is a bridge between a kernel event and a user-space dashboard. The process follows a strict, unidirectional path:


[ KERNEL SPACE ]                                     [ USER SPACE ]

(Event Trigger)                                      (Agent Logic)
       |
       v                                                   ^
[ eBPF Program ]  -- (write) --> [ BPF Map / RingBuf ] -- (poll) --|
       ^                                                   |
       |
[ Verifier / JIT ]                                         v
(Static Analysis)                                   [ Prometheus / Loki ]
                                                    (Aggregation/Storage)

The critical architectural constraints are:

The Verifier's Wall: Path Explosion and Rejections

The most frustrating part of eBPF development is the verifier. It doesn't run your code; it performs static analysis. The limit isn't the number of instructions in your file, but the number of processed instructions across all possible execution paths.

When you see processed 1,000,000 insns (limit 1,000,000), you've hit Path Explosion. This happens when a program has many branches or loops that the verifier must explore independently. A 500-line program with deep nesting can be rejected while a 5,000-line linear program passes.

Verifier ErrorWhat's Actually HappeningThe Fix
invalid mem access 'map_value_or_null'You tried to read from a map without checking if the key existed.Add if (!ptr) return 0; immediately after the map lookup.
infinite loop detectedThe verifier cannot prove the loop terminates.Use a constant bound for the loop or the bpf_loop() helper (v5.17+).
invalid indirect read from stackYou're reading a stack variable before writing to it (information leak).Initialize the buffer with __builtin_memset().
R1 type=... expected=...You're passing the wrong pointer type (e.g., passing a stack pointer to a map helper).Cast to the expected type or use the correct BPF helper.

Data Planes: Ring Buffers vs. Perf Events

Getting data out of the kernel is the primary bottleneck. You have two main choices for event streaming:

  1. Perf Event Arrays (The Old Way): Per-CPU buffers. The user-space agent must poll every CPU's buffer separately. This leads to "out-of-order" events and higher overhead as the number of cores grows.
  2. Ring Buffers (The Modern Way - v5.8+): A single, shared ring buffer for all CPUs. It provides strict FIFO ordering and is significantly more efficient for high-volume telemetry.

For stateful observability (e.g., counting requests per IP), you use BPF Maps. The trade-off is between BPF_MAP_TYPE_HASH (sparse keys, slower) and BPF_MAP_TYPE_ARRAY (fixed index, faster). In production, Per-CPU Maps are mandatory for high-throughput counters to avoid the "atomic-update bottleneck" where multiple cores fight over a single lock.

Production Blueprint: a Simple Request-Latency Probe

This is a minimal, production-ready pattern for tracing a function's latency. It uses a kprobe to start a timer and a kretprobe to calculate the delta and push it to a ring buffer.

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct event_t { __uint128_t start; __uint128_t delta; unsigned int pid; };

// Define the RingBuffer for event streaming
struct { 
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

// Per-CPU map to store start times for each PID
struct { 
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1024 * 16);
    __type(key, __u32);
    __type(value, __u64);
} start_times SEC(".maps");

SEC("kprobe/sys_clone")
int BPF_KPROBE(sys_clone)
{
    __u32 pid = bpf_get_current_pid_tgid();
    __u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start_times, &pid, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/sys_clone")
int BPF_KRETPROBE(sys_clone)
{
    __u32 pid = bpf_get_current_pid_tgid();
    __u64 *start = bpf_map_lookup_elem(&start_times, &pid);
    if (!start) return 0; // Verifier requires this NULL check

    __u64 delta = bpf_ktime_get_ns() - *start;
    
    struct event_t *e = bpf_ringbuf_reserve(&rb, sizeof(struct event_t), 0);
    if (!e) return 0;

    e->start = *start;
    e->delta = delta;
    e->pid = pid;

    bpf_ringbuf_submit(e, 0);
    bpf_map_delete_elem(&start_times, &pid);
    return 0;
}
BCC.kprobe_sys_clone = 1; // metadata for loader


package main

import (
	"log"
	"github.com/cilium/ebpf"
	"github.com/cilium/ebpf/ringbuf"
)

func main() {
	// 1. Load the pre-compiled BPF bytecode
	spec, err := ebpf.LoadCollectionSpec("probe.o")
	if err != nil { log.Fatal(err) }

	coll, err := ebpf.NewCollection(spec)
	if err != nil { log.Fatal(err) }
	defer coll.Close()

	// 2. Attach to the kernel hooks
	link, err := link.Kprobe("sys_clone", coll.Programs["sys_clone"], nil)
	if err != nil { log.Fatal(err) }
	defer link.Close()

	// 3. Poll the RingBuffer for events
	rd, err := ringbuf.NewReader(coll.Maps["rb"].// cast to ringbuf)
	if err != nil { log.Fatal(err) }

	for {
		event, err := rd.Read()
		if err != nil { continue }
		log.Printf("PID %d latency: %d ns", event.Pid, event.Delta)
	}
}

Production Realities: The Cost of Visibility

The "zero-overhead" claim is based on the fact that an idle probe costs nothing. But a hot probe is a different story.

Who Should Skip eBPF Observability

Despite its power, eBPF is not the right tool for everyone. You should skip it if:

References & Further Reading