The path to FedRAMP Authorization to Operate (ATO) is a significant undertaking for any SaaS provider who wants to tap into the U.S. federal government. And vendors running their applications on Kubernetes face a unique set of challenges and opportunities.
In a recent episode of Anti-Complex Kubernetes, John Hamilton, Principal Software Engineer at Zscaler, shared his experience getting FedRAMP ATO on Kubernetes. In his conversation with Buoyant CEO William Morgan, John offers lessons learned for peers embarking on a similar path. This article covers key takeaways from their conversation and highlights important considerations for anyone seeking FedRAMP ATO within a Kubernetes environment.
The Kubernetes: simplicity vs. complexity
First, we need to talk about Kubernetes and its dual nature within the FedRAMP context, because it can both simplify and complicate the compliance process.
Simplifying compliance: When implemented correctly, containerization can streamline certain security controls. For example, the isolation and resource management capabilities of containers can contribute to a more secure and auditable platform. Additionally, FedRAMP requirements like FIPS-validated encryption in transit can be achieved quickly and with low operational overhead by using Buoyant Enterprise for Linkerd as the service mesh.
Introducing complexities: The dynamic nature of Kubernetes— including ephemeral containers, mutable network identities, and complex orchestration — can create friction with traditional compliance controls designed for more static infrastructure. You'll need careful planning and new approaches to map these dynamic elements to established security requirements.
Zscaler's Cloud Browser solution
Zscaler required a FedRAMP ATO to sell its Cloud Browser isolation solution to the U.S. government — a cloud-based, isolated environment where user web browsing sessions are sandboxed for security. Instead of directly rendering web content on the user's local browser, the website renders and executes JavaScript remotely. The user then sees an interactive video or stream of pixels representing the webpage in their local browser. Zscaler’s product provides two main benefits: cyber threat protection by isolating potentially malicious websites and data protection by controlling access to internal applications without requiring an agent.
The application's architecture
Zscaler's cloud browser isolation solution runs almost entirely on Kubernetes—a global deployment with multiple Kubernetes clusters in various regions, sometimes multiple clusters within a single region, each cluster running thousands of pods.
Additionally, the solution requires ephemeral pods that exist only for the duration of a single user session, resulting in pods being created and destroyed at a much higher rate than what is seen in many clusters. This increases load not just on Linkerd, which secures all communications between pods, but also on the Kubernetes control plane itself.
Maintaining this complex platform and environment, composed of several different open source tools and container images, requires careful management of dependencies and vulnerabilities. The Zscaler team is continually pushing the boundaries of Kubernetes, even encountering scaling limits of the control plane due to the sheer number of pods.
Zscaler's FedRAMP process
Zscaler had already established their cloud browser isolation solution in their commercial clouds and integrated it with other products that served FedRAMP customers. Those FedRAMP customers were now asking for the browser isolation feature, making the FedRAMP ATO for that product a business opportunity they were eager to implement.
A key challenge was achieving FIPS-140 validation for all encryption, especially with the stricter requirements of the FedRAMP Rev. 5 baselines. This required understanding all data in flight and at rest, as well as ensuring all crypto operations used validated modules.
Zscaler used Linkerd with commercial support for FIPS-validated modules to cover a significant portion of the traffic. They also needed to ensure that other components communicating with the Kubernetes control plane and external APIs used FIPS-validated modules. This involved both purchasing commercial builds of some open source tools and building some components themselves.
Overall, the team decided to adopt a "low-risk" approach, both technically and in terms of compliance. They didn't want to leave any "open holes" where someone could point out data in transit that wasn't covered. They were also proactive in identifying and addressing vulnerabilities, especially with the continuous monitoring process.
Collaborating with the compliance team
The Zscaler platform team sought to collaborate with the compliance team in a way that was productive and interactive. Initially, there were weekly meetings where the compliance team guided John on the major aspects of FedRAMP compliance that needed to be addressed. This involved a lot of back-and-forth communication, with John asking questions about the requirements and the compliance team providing clarification and guidance.
John also educated the compliance team on Kubernetes and containerized workloads — a relatively new area for them. The compliance team researched and navigated the complex government documentation. Together, they navigated the challenges of applying FedRAMP controls, which were not always written with containerization in mind. Zscaler was one of the first Kubernetes solutions to get FedRAMP certified and possibly the first under the newer Rev. 5 requirements.
FedRAMP lessons learned
When asked whether John had any lessons learned he'd like to share with the audience, he mentioned 5 points:
- Use your resources: Collaborate with your teams. John was lucky to have an internal compliance team, but he also worked closely with security on threat modeling and internal testing. He also involved all engineering teams, from application development to the ops and infrastructure teams.FIPS touches everything, so everyone needs to be on board: you’ll be asking them to take a more active role in security, and they’ll need to understand what you’re working on and why you’re working on it in order to make it all successful.
- Understand FIPS first: Your most significant architectural changes will be driven by FIPS requirements, so do your research first. FIPS isn’t a matter of just importing new cryptographic modules; if you are new to FIPS, you'll likely need to fill some big knowledge gaps before being able to even do the right Google searches. John did a lot of reading and wrote an internal Wiki to help the rest of the team understand just how much they’d need to think about. He also created a few guides on approaches for different programming languages. Check out John's LinkedIn post, where he shares some of what he has learned.
- Aim for zero drift between commercial and fed environments: Your life will be simpler if you don’t have to maintain separate “FIPS” and “non-FIPS” architectures or builds. As far as you can, unify everything. For example, using feature flags for non-FIPS-compliant capabilities for your commercial customers will almost certainly be simpler than having two entirely separate source bases.
- Balance technical and compliance risk: While you'll generally want to avoid architectural changes to avoid introducing technical risks, sometimes it might be better to take on that risk upfront and not risk running into compliance issues down the road.
- Take continuous monitoring seriously: The challenge here isn’t scanning for vulnerabilities or the like, but managing how you’ll fix the issues you find within the FIPS-mandated 30-day window. Being proactive about the process here will be a big help down the road
Getting FedRAMP ATO
Successfully achieving FedRAMP ATO on Kubernetes was a significant milestone for Zscaler. It validated their commitment to security and opened up opportunities to serve U.S. government clients. Their journey highlights the importance of careful planning, proactive vulnerability management, and strong collaboration between the technical and compliance teams. Zscaler was able to navigate the challenges of FedRAMP Rev 5 in a dynamic Kubernetes environment, and we hope their experience provides guidance for those beginning their FedRAMP authorization process.
Fast-track your FedRAMP ATO journey with Buoyant
Navigating the complexities of FedRAMP, especially within a dynamic Kubernetes environment, can be challenging. At Buoyant, we understand the requirements firsthand. Just as we partnered with Zscaler on their successful journey, we're here to be your trusted FedRAMP partner. Book a meeting today to explore how we can help you obtain your FedRAMP ATO.
LinkedIn