Applying L4 network policies with a service mesh

(Note: if you’d like to read more about eBPF and sidecars, you might like our blog post on eBPF, sidecars, and the future of the service mesh by William Morgan.)

In this tutorial, you’ll learn how to run Linkerd and Cilium together and how to use Cilium to apply L3 and L4 network policies to a cluster running Linkerd.

Linkerd is an ultralight, open source service mesh. Cilium is an open source CNI layer for Kubernetes. While there are several ways to combine these two projects, in this guide we’ll do something basic: we’ll use Cilium to enforce L3/L4 network policies on a Linkerd-enabled cluster.

What are Kubernetes network policies?

Kubernetes network policies are controls over which types of network traffic are allowed to happen within a Kubernetes cluster. You might put these in place for reasons of security, or simply as a safeguard against accidents.

The terms “L3” and “L4” refer to layers 3 and 4 of the OSI network model, and refer to the policies that can be expressed in terms of IP addresses (layer 3) and ports (layer 4). For example, “requests between 192.0.2.42:9376 and 192.0.2.43:80 are forbidden” is a layer 4 policy. In orchestrated environments like Kubernetes, policies about individual IP addresses are quite brittle, so typically these policies will instead be expressed in terms of label selectors, e.g. “any pod with label app=egressok can send packets from port 80”. Under the hood, Cilium will track the pod assignments that Kubernetes does and translate this from label selectors to IP addresses.

L3 and L4 policies stand in contrast to L7 policies, which are expressed in terms of protocol-specific information. For example, “Pods with label env=prod are allowed to make HTTP GET requests to the /foo endpoint of pods with the label env=admin” is a layer 7 policy, because it requires parsing the protocol sent over the wire.

Linkerd supports rich L7 policies that take into account things like service identity. So we’ll use Cilium to implement L3 / L4 policies, and leave the advanced stuff to Linkerd. Let’s see how this works in practice.

Getting ready

What you’ll need to follow along:

• Kind, which we’ll use as a Kubernetes sandbox environment
• A modern Linkerd release
• A modern Cilium release

Creating the test cluster

As a first step, we’ll configure our kind cluster via a configuration file. Make sure you disable the default CNI and replace it with Cilium:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true

Installing Cilium

Now that we’ve got the cluster up and running, let’s install Cilium. Here are the steps:

# Add the Cilium repo
helm repo add cilium https://helm.cilium.io/
# Load the image onto the cluster
kind load docker-image cilium/cilium:v1.9.0

helm install cilium cilium/cilium --version 1.9.0 \
   --namespace kube-system \
   --set nodeinit.enabled=true \
   --set kubeProxyReplacement=partial \
   --set hostServices.enabled=false \
   --set externalIPs.enabled=true \
   --set nodePort.enabled=true \
   --set hostPort.enabled=true \
   --set bpf.masquerade=false \
   --set image.pullPolicy=IfNotPresent \
   --set ipam.mode=kubernetes

To monitor the progress of the installation, use kubectl -n kube-system get pods –watch.

Installing the sample workloads

To showcase what Cilium can do, we’ll use Podinfo and Slowcooker to simulate a client issuing requests to a backend API service:

# Create the test namespace
kubectl create ns cilium-linkerd
# Install podinfo
kubectl apply -k github.com/stefanprodan/podinfo//kustomize -n cilium-link

Now that we have a server, it’s time to install the client:

cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: cilium-linkerd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: buoyantio/slow_cooker:1.3.0
        command:
        - "/bin/sh"
        args:
        - "-c"
        - |
          sleep 5 # wait for pods to start
          cat <> url_list
          http://podinfo:9898/env
          http://podinfo:9898/version
          http://podinfo:9898/env
          http://podinfo:9898/metrics
          http://podinfo:9898/healthz
          http://podinfo:9898/readyz
          http://podinfo:9898/headers
          EOT
          /slow_cooker/slow_cooker @url_list
        ports:
        - containerPort: 9999
EOF

Applyinging ingress policies

The workloads are running now, so let’s go ahead and apply a Layer 4 ingress policy based on labels that restrict the packets that reach our Podinfo workload.

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "ingress-policy"
  namespace: cilium-linkerd
specs:
  - endpointSelector:
      matchLabels:
        app: podinfo
    ingress:
    - fromEndpoints:
      - matchLabels:
          app: client
      toPorts:
      - ports:
        - port: "9898"
          protocol: TCP
  - endpointSelector:
      matchLabels:
        app: podinfo
    ingress:
    - fromEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": linkerd
EOF

This policy has two ingress rules that apply to services labeled app: podinfo:

1. The server can accept traffic from workloads labelled with app: client only on port 9898. All other ports are blocked.
2. Workloads from the Linkerd namespace can communicate with the server.

The second rule is essential for the correct operation of Linkerd. Much of the functionalities like tap and top rely on the control plane components connecting to the proxy sidecar that runs in the meshed workloads. If this connectivity is blocked by Cilium rules, some of the Linkerd features might not work as expected.

Applying egress policies

Our Podinfo server is now conforming to the network policies. To allow our client pod to only communicate with the Podinfo backend we can use an Egress Cilium policy:

cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "egress-policy"
  namespace: cilium-linkerd
specs:
  - endpointSelector:
      matchLabels:
        app: client
    egress:
      - toEndpoints:
        - matchLabels:
            "app": podinfo
  - endpointSelector:
      matchLabels:
        app: client
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": kube-system
          "k8s:k8s-app": kube-dns
  - endpointSelector:
      matchLabels:
        app: client
    egress:
    - toEndpoints:
      - matchLabels:
          "k8s:io.kubernetes.pod.namespace": "linkerd"
EOF

This policy has three egress rules that apply to workloads labeled with app: client:

1. The server can initiate traffic to workloads labelled with app: podinfo.
2. Initiate traffic to kube-dns for DNS resolution.
3. Unblock traffic to Linkerd components.

Here again, outgoing traffic to Linkerd components is essential. The Linkerd proxy uses the identity and destination services to obtain TLS certificates and perform service discovery. If this connectivity is blocked, the proxy will not be able to work correctly, rendering the meshed workload unusable.

Observing traffic with Linkerd

Now that our traffic is obeying ingress and egress policies, we can go ahead and install Linkerd following the installation guide. Ready? Then, let’s mesh our workloads:

kubectl get deploy -n cilium-linkerd podinfo -oyaml | linkerd inject - | kubectl apply -f -

Once workloads are meshed, we can see the requests being issued to Podinfo:

$ linkerd top deployment/podinfo --namespace cilium-linkerd

Source                   Destination               Method      Path       Count    Best   Worst    Last  Success Rate
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-j96td  GET         /env           2   531µs   847µs   847µs       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-cbt98  GET         /metrics       2     2ms     2ms     2ms       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-cbt98  GET         /env           1   507µs   507µs   507µs       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-j96td  GET         /healthz       1   664µs   664µs   664µs       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-j96td  GET         /readyz        1   469µs   469µs   469µs       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-j96td  GET         /headers       1   586µs   586µs   586µs       100.00%
client-5c69b9d757-tzbng  podinfo-5fc5cb5f59-j96td  GET         /version       1   491µs   491µs   491µs       100.00%

Similarly, we can observe the live stream of all requests going out of the client workload:

$ linkerd tap deployment/client --namespace cilium-linkerd

req id=5:0 proxy=out src=10.244.0.215:54990 dst=10.244.0.211:9898 tls=true :method=GET :authority=podinfo:9898 :path=/env
rsp id=5:0 proxy=out src=10.244.0.215:54990 dst=10.244.0.211:9898 tls=true :status=200 latency=1069µs
end id=5:0 proxy=out src=10.244.0.215:54990 dst=10.244.0.211:9898 tls=true duration=30µs response-length=1008B

Note that the tls=true indicator shows that mTLS is applied to all traffic between workloads. To ensure that the policies work:

$ kubectl exec deploy/client -n cilium-linkerd -- curl -s podinfo:9898
{
  "hostname": "podinfo-5fc5cb5f59-j96td",
  "version": "5.0.3",
  "revision": "",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v5.0.3",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.15.3",
  "num_goroutine": "8",
  "num_cpu": "4"
}

Reaching an alternative destination from the ones allowed is not possible:

# The request will simply hang...
$ kubectl exec deploy/client -n cilium-linkerd -- curl  'https://postman-echo.com/get?foo1=bar1&foo2=bar2'

Congrats! At this point you’ve successfully enforced L3/L4 policies using Cilium on a Linkerd-enabled cluster.

What’s next?

In this post, we’ve demonstrated how to use Cilium and Linkerd together, and how to apply L3/L4 policies in a Linkerd-enabled cluster. Everything in this blog post can be used in production today. In upcoming releases, Linkerd will add L7 support, we’ll be able to extend these same ideas to protocol-specific policies as well. Until then, go forth and make L3/L4 policies with Linkerd and Cilium!

Buoyant makes Linkerd awesome

Buoyant is the creator of Linkerd and of Buoyant Cloud, the best way to run Linkerd in mission-critical environments. Today, Buoyant helps companies around the world adopt Linkerd, and provides commercial support for Linkerd as well as training and services. If you’re interested in adopting Linkerd, don’t hesitate to reach out!

‍

‍