If you’re running Kubernetes 1.6 or later, you can optionally make use of Kubernetes’ new support for RBAC (role-based access control), which allows you to restrict who can access the Kubernetes API on the cluster and what they can do with it. However, when upgrading to an RBAC-enabled cluster you can run into issues, as many Kubernetes examples do not take into account the fact that certain API calls may be restricted.
In this post, we’ll show you how to use Linkerd, our open source service mesh for cloud-native applications, with RBAC-enabled Kubernetes clusters.
What is RBAC?
First, it’s helpful to understand what RBAC actually does. RBAC works by defining a role that describes a set of permissions, and by then assigning that role to relevant users/service accounts. In Kubernetes RBAC, these roles restrict which Kubernetes verbs can be used (e.g.
create), and which Kubernetes resources they can be applied to (e.g.
services). So, for example, we can create a
Role (called, for example, “read-only”) that only allows
watch on pod resources. And we can then create
RoleBindings to assign this “read-only” role to whichever “subjects” need them, e.g. the “qa-bot” service account.
In order for Linkerd to operate in an RBAC-enabled cluster, we need to make sure that the types of access that Linkerd needs to the Kubernetes APIs are allowed. Below, we’ll walk through how to do this. If you just want the completed config, you can skip to the bottom—or just use linkerd-rbac-beta.yml (stored in our linkerd-examples repo).
We’ll be setting up the permission by creating a
ClusterRole and a
ClusterRoleBinding, illustrated below.
Granting Linkerd access to an RBAC Kubernetes Cluster
When used with a Kubernetes cluster, Linkerd uses its
io.l5d.k8s “namer” to do service discovery against the Kubernetes API. (Of course, this namer can be used in conjunction with other service discovery mechanisms, allowing Linkerd to bridge Kubernetes and non-Kubernetes systems—but that’s a later blog post).
Linkerd only requires read access, and only needs access to access the
endpoints Kubernetes resources. We can capture this access via the following Kubernetes config:
--- # grant linkerd/namerd permissions to enable service discovery kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-endpoints-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["endpoints", "services", "pods"] # pod access is required for the *-legacy.yml examples in linkerd-examples verbs: ["get", "watch", "list"]
For simplicity’s sake, at this point we could just assign this role to the
default service account (which is the account Kubernetes assigns to you when you create a pod if you don’t specify one):
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-role-binding subjects: - kind: ServiceAccount name: default # change this to your service account if you’ve specified one namespace: default roleRef: kind: ClusterRole name: linkerd-endpoints-reader apiGroup: rbac.authorization.k8s.io
Linkerd now has the access it needs to function in a Kubernetes environment. In production, however, you might want to use a dedicated service account—see below.
If you’re using Namerd as a control plane to dynamically change routing configuration across all Linkerd instances (see here for why you might want to do this), you’ll need some additional permissions. Namerd needs access to a Kubernetes
ThirdPartyResource to store its routing rules (“dtabs”). In our example namerd.yml, we’ve added this resource as
d-tab.l5d.io. We can allow Namerd read and write access to this resource using the following role:
--- # grant namerd permisisons to third party resources for dtab storage kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: namerd-dtab-storage rules: - apiGroups: ["l5d.io"] resources: ["dtabs"] verbs: ["get", "watch", "list", "update", "create"]
Similar to above, we’ll assign the role to the
default service account with a role binding:
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: namerd-role-binding subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: namerd-dtab-storage apiGroup: rbac.authorization.k8s.io
Running Linkerd with a specified Service Account
In the previous sections, we used the default service account to run Linkerd. For some use cases, however, you may want to create a dedicated service account and assign the permissions to that account. You’ll also want to consider whether you want roles to be cluster-scoped (
ClusterRoleBinding) or namespace-scoped (
Let’s go through how to configure permissions for a specific service account,
linkerd-svc-account, starting from the linkerd.yml config in linkerd-examples. We’ll add a
ServiceAccount config, and assign that service account to the pod. Here’s part of the file, with
--- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: app: l5d name: l5d spec: template: metadata: labels: app: l5d spec: volumes: - name: l5d-config configMap: name: "l5d-config" serviceAccount: linkerd-svc-account containers: - name: l5d image: buoyantio/linkerd:1.1.2 env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP args: - /io.buoyant/linkerd/config/config.yaml ports: - name: outgoing containerPort: 4140 hostPort: 4140 - name: incoming containerPort: 4141 - name: admin containerPort: 9990 volumeMounts: - name: "l5d-config" mountPath: "/io.buoyant/linkerd/config" readOnly: true - name: kubectl image: buoyantio/kubectl:v1.4.0 args: - "proxy" - "-p" - "8001" --- apiVersion: v1 kind: ServiceAccount metadata: name: linkerd-svc-account
Then we’ll change the subject in your linkerd-rbac-beta.yml to reference this new service account:
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-role-binding subjects: - kind: ServiceAccount name: linkerd-svc-account namespace: default roleRef: kind: ClusterRole name: linkerd-endpoints-reader apiGroup: rbac.authorization.k8s.io
And that’s it! The Linkerd pods now use the
linkerd-svc-account and have the right permissions.
Putting it all together
For a complete Kubernetes config file that uses all of the above, just use this file: linkerd-rbac-beta.yml. This config will allow Linkerd and Namerd to have all the access needed to the Kubernetes API with the default service account. If you’d like to set this up using a dedicated service account, you’ll need to modify linkerd-rbac-beta.yml, as described in the previous section.
We hope this post was useful. We’d love to get your thoughts. Please join us in the Linkerd Discourse forums and the Linkerd Slack channel! And for more walkthroughs of how to use Linkerd’s various features on Kubernetes, see our Service Mesh For Kubernetes blog series.