Relevant articles:
Abhimanyu Selvan
July 31, 2023
This blog post was orginially publishes on DigitalOcean's docs by Abhimanyu Selvan
Introduction
Multi-cluster Kubernetes refers to managing multiple Kubernetes clusters, which are groups of nodes working together to orchestrate and run containerized applications. These clusters can be distributed across different data centers, regions, or cloud providers to ensure high availability, fault tolerance, and better performance.
Multi-cluster Kubernetes offers several advantages in specific scenarios:
- Scaling and High Availability: By distributing workloads across clusters, it can handle increased traffic and workload demands. If one cluster fails, applications seamlessly switch to another, minimizing downtime.
- Geographical Distribution: Clusters in different locations ensure localized data processing, compliance with regulations, and meeting data sovereignty requirements.
- Isolation and Resource Allocation: Applications with varying resource needs, critical workloads, or distinct security/compliance requirements can be deployed in separate clusters. This ensures optimal resource usage and prevents interference between workloads.
- Application and Environment Segmentation: Multi-cluster Kubernetes supports separating development, testing, staging, and production environments. This segregation allows better control, isolation, and independent work among teams. Changes in one environment have minimal impact on others, further minimizing risks.
Service Mesh Solution
Service Mesh frameworks are dedicated infrastructure layers to simplify the management and configuration of micro-service-based applications in a distributed system. Service meshes introduce a sidecar container as a proxy to provide multiple features (i.e., secure connections with mutual TLS, circuit breaking, canary deployments).
Some of the most popular service mesh architectures like Istio and Linkerd have multi-cluster support to embrace multi-cluster microservices applications. The interconnection among different clusters uses a dedicated proxy to route traffic from the mesh of one cluster to another. Similarly, Istio and Linkerd can create an ad-hoc mutual TLS tunnel across clusters and provide primitives to expose services across the clusters, enabling cross-cluster communication and traffic-splitting features.
Multi-cluster DOKS setup using Linkerd
In this POC, we will use the Multi-cluster extension of Linkerd to set up secure cross-cluster communication between regions.
Prerequisites
We need to install some CLI tools to set up the POC successfully.
- doctl - CLI for DigitalOcean API
- kubectl - K8s CLI
- step - CLI for configuring Public Key Infrastructure
- linkerd - CLI to interact with Linkerd service mesh
Prepare the environment
Open a terminal and let’s go!
DOKS cluster creation
Set Trust Anchor
Linkerd requires a shared trust anchor between the installations in all clusters that communicate with each other. This encrypts the traffic between clusters and authorizes requests that reach the gateway so your cluster is not open to the public internet.
Install Linkerd(+CRDs) and Extensions
Note: The commands in the following section will assume that you renamed the contexts to west and east.
Multicluster and Visualization extension
Two components implement Linkerd’s multi-cluster functionality:
- Service mirror: This component watches a target cluster for updates to services and mirrors that service updates locally on a source cluster. This provides visibility into the service names of the target cluster so that applications can address them directly.
- Gateway: This component provides target clusters with a way to receive requests from source clusters.
linkerd-viz is a monitoring application based on Prometheus and Grafana, autoconfigured to collect metrics from linkerd.
To link west to east, a credentials secret, Link resource, and service-mirror controller are created. The credentials secret includes a kubeconfig for accessing the east cluster’s Kubernetes API. The Link resource configures service mirroring with gateway settings and label selectors. The service-mirror controller utilizes the Link and secret to identify and copy matching services from the east cluster to the west cluster.
In this case, we would like cluster west to mirror the services from cluster east so the service mirror component is installed only on the west cluster.
Service mirroring in action
The service mirror is a Kubernetes Operator. Once installed, it mirrors a remote cluster’s services locally to provide service discovery and allow pods to refer to the remote services. It also manages configuring endpoints so that traffic goes to the correct IP address.
Services are not automatically mirrored in linked clusters. Only services with the mirror.linkerd.io/exported label will be mirrored by default.
Secure cross-cluster communication
mTLS (mutual Transport Layer Security) is an extension of regular TLS (Transport Layer Security) that ensures the server and the client are authenticated. By default, TLS only authenticates the server, but mTLS establishes mutual authentication, making the authenticity of both parties symmetric.
By default, requests will be going across the public Internet. Linkerd extends its automatic mTLS across clusters to ensure that the communication going across the public internet is encrypted.
To verify mTLS:
Output(check tls=true):
Linkerd enables mutually-authenticated Transport Layer Security (mTLS) for all TCP traffic between meshed pods. There are exceptions where non-mTLS traffic may still exist, such as traffic to or from pods outside the mesh (e.g., Kubernetes health checks) or traffic on ports designated as skip ports, bypassing the Linkerd proxy.\
Activate service traffic
To demonstrate the traffic flow between the two clusters across two different regions, we need to make the following changes:
Now the deployment/vote-bot forwards the requests to svc/web-svc-east whose endpoints are configured to point to the EXTERNAL-IP of the linkerd-gateway on the east cluster.
Monitoring the traffic flow
We will now use the linkerd-viz dashboards to monitor the cross-cluster service communication.
The built-on linkerd dashboard below shows the live calls and route metrics between the deployment/vote-bot and svc/web-svc-east (mirrored) on the cluster east:
The built-on linkerd dashboard below shows the live calls and route metrics between the deployment/linkerd-gateway and svc/web-svc on the cluster west:
End-to-end flow
The following figure depicts the secure end-to-end flow between the services in cluster west to the services in cluster east:
Key takeaways
- A comprehensive and practical guide for setting up a secure, efficient multi-cluster, multi-region environment using DigitalOcean Kubernetes (DOKS) and Linkerd.
- Leverage the power of multi-cluster Kubernetes to scale the applications, ensure high availability, meet data sovereignty requirements, and optimize resource allocation.
- Establish secure cross-cluster communication using mutual TLS encryption, and enable service mirroring for efficient service discovery.
- Insights into monitoring and visualizing the traffic flow between clusters using Linkerd’s built-in visualization dashboards.
As technology evolves, multi-cluster setups are increasingly crucial for handling modern application requirements. With the guidance provided in this tutorial, readers can navigate the complexities of multi-cluster environments and unlock the full potential of their Kubernetes deployments.
Useful Resources
