By the 5Tech engineering team · Published April 2023 · Written for platform, SRE, and infrastructure engineers running Kubernetes at scale. This is an educational engineering guide; adoption and overhead figures are cited or explicitly labelled as reported/illustrative, and every project and kernel-version claim is checked against a primary source.
eBPF is the most consequential change in how we observe Linux since strace — and also the most over-sold. Moving observability out of a proxy in every pod and into a sandboxed program in the kernel does eliminate the sidecar "observability tax." But it doesn't make observability free: it converts a per-pod resource cost into a dependency on your kernel, its verifier, and its security surface — a bill that lands on the platform team, not the application. That trade is worth it at scale far more often than not. It is not universal, and treating it as universal is how eBPF projects stall.
The problem: the observability tax
The reflex pattern for cloud-native visibility is to put a proxy next to every workload. A service mesh injects a sidecar — typically an Envoy proxy — into each pod to intercept traffic and emit metrics, traces, and logs; host agents run separately in user space to scrape system data. It works, and for L7 routing and mutual TLS (mTLS) it still works well. But at fleet scale the model bills you four ways:
- Resource overhead. Every sidecar is a process with its own CPU and memory footprint. Independent service-mesh comparisons commonly report on the order of tens of megabytes of memory and single-digit-millisecond added latency per hop — figures that vary widely by workload and configuration, and that multiply by pod count. The industry's own response is telling: Istio's "ambient" mode was introduced specifically to let users run the mesh without per-pod sidecars.
- Latency. Routing every request through an extra user-space proxy adds hops and context switches to the hot path.
- Operational complexity. Injecting, upgrading, and reconciling a proxy across thousands of pods is its own distributed system to run.
- Blind spots. A user-space sidecar sees what it proxies. Kernel-level events, activity inside an app's own memory, and traffic it doesn't route are outside its view.
The approach: make the kernel programmable
eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without changing kernel source or loading a kernel module (ebpf.io). Instead of a proxy per pod observing traffic from the outside, a single program per node observes events from inside the kernel, where they already happen. The central shift is one of location, and it has real consequences in both directions.
eBPF doesn't delete the observability tax — it moves it out of every pod's sidecar and into one kernel dependency the platform team now has to own.
How a program actually gets into the kernel
The lifecycle is what makes eBPF safe enough to run in production, and it is also where most of its limits come from (ebpf.io):
- Write and compile. Programs are written in a restricted C subset (or Rust) and compiled to eBPF bytecode with Clang/LLVM.
- Verify. Before loading, the in-kernel verifier statically analyses every possible execution path to prove the program terminates, stays within bounds, and can't crash the kernel. This is the safety guarantee — and the source of the "why won't my program load" pain.
- JIT-compile and attach. Verified bytecode is just-in-time compiled to native machine code and attached to a hook: a syscall, a network event, a kernel function (kprobe/tracepoint), or a user-space function (uprobe).
- Communicate via maps. The program filters and aggregates in kernel space and passes only what's needed to user space through eBPF maps and ring buffers — so you ship results, not raw firehoses.
Two ways to see the same traffic. The sidecar model puts a user-space proxy in every pod, so overhead scales with pod count and each request crosses the user/kernel boundary repeatedly. The eBPF model attaches programs to kernel hooks, observes every pod on the node from one place, aggregates in kernel maps, and streams only results to a single per-node agent. (Illustrative comparison of data paths, not a specific deployment.)
What you actually gain
The benefits are real and, unlike many observability pitches, structurally motivated rather than incremental:
- Lower overhead at scale. No per-pod proxy process and no extra context switch to collect data; filtering happens in the kernel, so only aggregated results cross into user space. One agent per node (a Kubernetes DaemonSet) replaces N sidecars.
- Deeper, wider visibility. Kernel hooks see syscalls, file and network activity, and TCP-level events for every process on the host — including things a sidecar can't, such as correlating an application HTTP 500 with a TCP retransmit underneath it.
- No application changes. A per-node agent observes workloads without modifying pod specs or app code, which shrinks the deployment blast radius and removes a class of injection-related failures.
- Encrypted traffic, handled carefully. By attaching uprobes to TLS library functions before encryption, tools can capture plaintext without terminating TLS or holding private keys. That is powerful — and, because it reads application memory in the kernel, it is exactly the capability your security team should review, not wave through.
What it costs — the trade-offs the pitch skips
Every gain above has a matching liability. None is disqualifying; all of them land on the platform team.
1. Kernel-version dependence
eBPF's capabilities are gated by the kernel you're running on. Bounded loops arrived in kernel 5.3 (LWN); the CAP_BPF capability that lets you grant BPF rights without full CAP_SYS_ADMIN arrived in 5.8; the bpf_loop helper in 5.17. Most tools recommend a reasonably modern kernel (roughly 5.8+) to get the full feature set. On managed Kubernetes you often don't choose the node kernel, and a fleet spanning several kernel versions means testing your probes against each. This is a genuine constraint the sidecar model doesn't have — a user-space proxy runs the same on almost any kernel.
2. The verifier gives, and the verifier takes
The verifier's guarantees come with hard limits: a program must have finite, provable complexity (the analysis budget was raised to roughly 1 million instructions in kernel 5.2), a small stack (512 bytes), and loops the verifier can prove terminate. Perfectly correct programs get rejected because the verifier can't prove them safe within budget, and the rejection messages are notoriously hard to read. Writing non-trivial eBPF means working with the verifier — which is precisely why most teams should consume a higher-level tool rather than hand-write probes.
3. The probes need their own observability
An eBPF program is hard to debug — you can't just attach a debugger to kernel code. When a ring buffer fills, events are dropped silently; when a map is undersized, data is lost; a heavy probe on a hot path adds measurable latency to the very system it watches. So you have to monitor the observer: probe CPU cost, dropped-event counters, and map pressure become first-class signals. Observability tooling that is itself unobservable is a trap.
4. Kernel programs are a security surface, not just a security tool
Running code in the kernel cuts both ways. Loading eBPF requires privilege (CAP_BPF/CAP_SYS_ADMIN), and because the verifier is complex, bugs in it have themselves been exploited for local privilege escalation — for example CVE-2021-3600 and CVE-2023-39191. Unprivileged eBPF is disabled by default on modern kernels, and distributions such as SUSE restrict it to privileged users specifically to reduce attack surface. Treat "who can load eBPF programs on this node" as a controlled, audited capability, and keep kernels patched.
5. CO-RE helps portability but doesn't finish the job
Historically, an eBPF program compiled against one kernel's data structures broke on another. CO-RE (Compile Once — Run Everywhere) fixes most of this by encoding symbolic references that libbpf relocates at load time — but it depends on the kernel exposing type information (BTF, the BPF Type Format), which older kernels may not ship. Where BTF is missing, tools fall back to runtime compilation (the BCC approach), which drags an LLVM toolchain onto every node. Portability improved dramatically; it is not automatic.
6. Uprobes are powerful but not cheap
User-space probes (uprobes) enable code-less application tracing, but each hit traps into the kernel and back — a context switch that makes uprobe overhead materially higher than kernel-side probes; benchmarks put it on the order of ~10x a kprobe. On a high-frequency function that adds up fast. Uprobes are a scalpel for targeted investigation, not something to leave attached to a hot path indefinitely.
eBPF vs. sidecar/userspace: a decision matrix
The honest framing isn't "eBPF wins." It's "eBPF changes where the cost lands, and that's the right trade in most large fleets but not all of them." The numeric ranges below are reported/illustrative and vary by workload — use them for shape, not as a spec.
| Dimension | Sidecar / user-space agent | eBPF (per-node) |
|---|
| Per-workload overhead | One proxy process per pod; scales with pod count | One agent per node; largely independent of pod count |
| Visibility depth | What the proxy routes; app-level | Kernel + network + (via uprobes) app internals |
| App/pod changes | Requires injection into every pod | None; observes from the kernel |
| Kernel dependence | Minimal — runs on almost any kernel | Feature set gated by kernel version + BTF |
| Debuggability of the tool | Ordinary user-space process | Harder; needs probe-overhead + drop monitoring |
| Privilege / security surface | User-space; app attack surface | Kernel-level; privileged load, verifier CVEs |
| Best when… | Non-Linux/Windows, managed kernels you can't control, rich L7 routing + mTLS termination, small fleets | Large Linux fleets, low-overhead deep visibility, security observability, you control the kernel |
The mental model behind eBPF observability: instead of a proxy watching each service from the outside, programs at kernel hooks observe every workload on a node from the one place the events already pass through.
The tooling: consume it, don't hand-roll it
Almost no team should be writing raw eBPF. A mature, mostly CNCF-governed ecosystem abstracts the verifier and CO-RE work for you. Verify each against its own project before you standardise on it:
- Cilium — an eBPF-based Kubernetes CNI providing networking, load balancing, network policy, and (with its service-mesh mode) L7 features. A CNCF graduated project.
- Tetragon — the Cilium project's eBPF runtime security component; it can both observe and enforce (block a process before its syscall completes).
- Falco — CNCF graduated runtime-security project (created at Sysdig); an eBPF probe streams syscalls to a user-space rules engine. Detection-focused: it tells you something happened.
- Pixie — auto-instrumenting Kubernetes observability using eBPF; open-sourced and contributed to the CNCF by New Relic.
- BCC and libbpf — the toolkits underneath: BCC for ad-hoc tracing, libbpf as the modern standard for building portable (CO-RE) eBPF applications.
Operating it: the lifecycle doesn't end at deploy
An eBPF rollout is a platform commitment, not a one-off install. Plan for the whole lifecycle:
- Compatibility gate. Confirm kernel versions and BTF availability across every node pool before you commit to a tool's feature set.
- Staged rollout. Deploy the DaemonSet to one node pool first; measure agent CPU/memory and the added latency on real traffic before fleet-wide.
- Treat kernel upgrades as breaking changes. Attach points and structures can move between kernels; CO-RE absorbs much of this, but you still re-test probes on every kernel bump — bake it into your OS-image pipeline.
- Version the programs and maps. Track which eBPF program and map schema each node runs, and keep a rollback path (redeploy the previous DaemonSet).
- Monitor the observer. Alert on dropped ring-buffer events, map saturation, and probe overhead — losing telemetry silently is worse than not collecting it.
- Patch and restrict. Keep kernels current for eBPF-subsystem CVEs, and control which identities may load programs. This is an OT-grade privilege, not a convenience.
- Integrate, don't replace. Export via OpenTelemetry or Prometheus into your existing dashboards. eBPF augments the stack you have; a rip-and-replace is rarely the right project.
Making the call: a cost model, not a vibe
Because the cost simply moves rather than vanishing, the decision should be a number your platform and security leads both sign:
Annual value = sidecar CPU/memory reclaimed + proxies no longer operated + faster incident resolution (deeper visibility → lower MTTR) + APM instrumentation effort avoided − ( kernel-compatibility & upgrade testing + eBPF/kernel expertise + probe-overhead monitoring + a real security review of kernel-level programs + tooling/platform cost )
At a few dozen pods, the reclaimed overhead may not repay the new kernel dependency and the expertise it demands — a well-run sidecar mesh can be the cheaper answer. Across thousands of pods on kernels you control, the same arithmetic usually flips decisively the other way. The number, not the trend line, is what should decide it.
What to remember
- eBPF's core move is location: observe from inside the kernel instead of from a proxy in every pod — it relocates the observability tax, it doesn't abolish it.
- The gains (lower fleet overhead, deeper visibility, no app changes) are structural and real; the liabilities are kernel-version dependence, verifier limits, hard-to-debug probes, and a kernel-level security surface.
- Don't hand-write eBPF — consume graduated tools (Cilium, Falco) and CO-RE/libbpf-based platforms; verify each project's maturity yourself.
- Operate the whole lifecycle: compatibility gate, staged rollout, kernel-upgrade re-testing, drop-rate monitoring, CVE patching, and export into your existing stack.
- Keep the sidecar model where it still wins: kernels you don't control, non-Linux workloads, rich L7/mTLS, or small fleets. Make the choice a signed cost number.
Where to start
Skip the platform-wide migration. Pick one node pool on a kernel you control and one concrete pain — a service map you can't build, a latency source you can't localise, a runtime-security gap — deploy one graduated eBPF tool as a DaemonSet, and measure agent overhead, dropped-event rate, and time-to-answer against your current approach on real traffic. A single node pool with honest numbers will tell you more than any architecture deck. If you'd like a second set of eyes on that pilot's kernel-compatibility and security review, talk with the 5Tech engineering team.
References & further reading
- ebpf.io — What is eBPF?: authoritative overview of the load/verify/JIT/attach lifecycle, maps, helper functions, and verifier complexity limits.
- LWN — Bounded loops in BPF for the 5.3 kernel: kernel-version history of the loop and complexity constraints.
- CNCF Annual Survey 2024: Kubernetes and service-mesh adoption figures, including the decline in service-mesh usage.
- NVD — CVE-2021-3600 and CVE-2023-39191: examples of eBPF-subsystem privilege-escalation vulnerabilities.
- Cilium (CNCF graduated), Tetragon, Falco (CNCF graduated), Pixie: representative eBPF projects for networking, runtime security, and observability.
- Istio ambient mesh: the mesh community's own move to a sidecar-less data path — useful context on sidecar overhead.
The data-path comparison diagram is an illustrative reference, not a specific deployment.
Send 5Tech one workflow, inspection task, sensor problem, robotics challenge, or prototype idea. We will review it and suggest a practical next step. Start My Free First Phase — a free first review of one idea. If it is not practical, we will tell you.