Mythbusters: Sidecars, eBPF, and ambient mesh

An engineer's guide to cutting through marketing hype, from your engineering friendsat team Linkerd

Myths

eBPF allows you to remove sidecars. False.Sidecars and eBPF are two orthogonal choices, andsidecar proxies can use eBPF just as easily asnon-sidecar proxies. The limitations of eBPF meanthat every service mesh still requires user-spaceproxies; the decision is where to place those proxies

Sidecars are slow/complex/bulky. False. There isnothing inherently slow about the sidecar pattern,any more than adding a library to your application isinherently slow. Speed and resource consumptiondepend on what is running in the sidecar.

Sidecars introduce extra latency. Technicallytrue. Sidecars require more network hops thanother models, but except in extreme cases, thisadditional latency is negligible compared tolatency of application, database, and othercomponents of the system.

Truths

Sidecars have clearly-defined security andfailure domains. Sidecars are deployed in eachpod, which means they can hold the mTLS secretsjust for that pod, and that any failure or leak of anindividual proxy only affects that pod.

Sidecars are simple. Because they bind to anindividual pod, the failure and security domain forevery sidecar proxy is clear and very small: a failureor leak in an individual proxy affects only that pod.

Per-host proxies are less secure and morecomplex than sidecars. Because traffic andmTLS secrets from arbitrary sets of pods aremixed in the same process, per-host proxies havesignificantly worse security and operationalcharacteristics than sidecars.

Sidecars can be extremely fast and light. Whilemost service meshes use the general-purpose Envoyproxy, Linkerd's Rust-based linkerd-proxy is designedspecifically for the service mesh use case, and canrun in as little as 3mb of RSS with a p99 of <1ms.

What do all these words mean?

Sidecar proxy. A proxy deployed as a container ina pod, alongside the application container. Thisproxy intercepts traffic to and from the pod,proxying it to and from the application container.

Per-host proxy. Proxy deployed at the host level(e.g. via a DaemonSet). This proxy intercepts traffic toand from whichever pods are scheduled on the node.

eBPF. A Linux feature that allows applications to docertain types of work in the kernel. By avoiding theuserspace/kernel barrier, eBPF providesperformance advantages in some situationsincluding processing of network packets. However,eBPF is highly limited and cannot handle complexoperations such as HTTP/2 or TLS parsing, whichmust be done in userspace.

eBPF service mesh. A service mesh model inwhich sidecar proxies are replaced by per-hostproxies, with some eBPF features. (Primarily amarketing term, as sidecar-based service meshescan and do use eBPF.)

Ambient mesh. A service mesh model in whichsidecar proxies are replaced by a combination ofper-host and per-namespace proxies.

Model Comparison

Security

Operational
complexity

Performance

Sidecar service mesh

Good
Low
Good

eBPF service mesh

Poor
High
Good

Ambientmesh

Good
High
Good

Sidecar service mesh

E.g. Linkerd, Istio. Good security, operability, andperformance when sidecar implementation is fast andsmall (e.g. Linkerd's Rust-based proxies).

eBPF service mesh

E.g. Ciilum service mesh. The reliance on per-host proxiesresults in significantly worse security and operationalcharacteristics than sidecar-based models.

Ambient mesh

E.g. Istio Ambient mode. Splitting into per-namespaceproxies allows this model to maintain the securitybenefits of sidecars. However, multiple layers ofproxies and reliance on Envoy create significantoperational complexity.

Sidecars are the future of the service mesh

Sidecars aren't perfect, and we at team Linkerd know their downsides first-hand. But we continue to believethat the operational and security benefits of sidecars far outweigh these downsides, and that if your goal is toreduce operational complexity and security risk a sidecar-based approach is superior by far to the alternatives.

Further reading

Want the deep dive? Lots more reading online: