.svg){kind=small}
Feb 23, 2024
Note (Oct 23, 2024): Please see our eight-month retrospective post on the change in stable release policy discussed in this post.
A few days ago, we announced Linkerd 2.15 with mesh expansion, native sidecars, SPIFFE support, and some other great new features—and also with a statement that we would no longer be publishing open source stable releases.
That last point provoked some great discussion on various forums. We knew that there were a lot of Linkerd adopters out there. We were reminded this week that our adopters are very, very passionate about the project!
Feedback was positive overall. But we also learned that there are a couple of things that we need to clarify and a few things we need to consider changing.
Let's start with the clarifications.
Yes. Linkerd remains fully open source under the Apache v2 license.
No. Linkerd remains fully open source under the Apache v2 license.
No. Every line of code we're adding to Linkerd—every feature, improvement, and bugfix—all continues to go into the Linkerd GitHub repo and is all Apache v2 licensed, just as it has been since 2016 when we first contributed Linkerd to the CNCF.
We will continue to address all bugfixes, security patches, and CVE fixes as they always have been in Linkerd. We fix these issues as quickly as possible and ship them in the next release, escalating the timeline for that release if necessary. We have a formal security policy (SECURITY.md) that documents how we handle these situations.
Linkerd has a stellar security record that I'm very proud of—better than other projects in this space, in my biased opinion—with concrete examples to back it up. We take our commitment to security seriously, and nothing about that has changed or will change.
We are no longer producing open source release artifacts marked "stable".
Buckle up, I'm going to go into a lot of detail here.
In addition to writing Linkerd code, helping users debug problems, triaging issues, and so on, we produce release artifacts. These artifacts are how most people consume Linkerd—they comprise the binaries, Docker images, Helm charts, etc, necessary to get Linkerd on your Kubernetes cluster. These release artifacts go into the open source repo, alongside the code. Of course, as a fully open source project, you're always welcome to build your own release artifacts from the code, or use the code in any other way, including for commercial purposes.
Note that these artifacts are a convenience for users, not some intrinsic requirement for an open source project. There's a timely blog post by Helm maintainer Matt Farina on this very topic: Open Source Doesn't Require Providing Builds.
Historically, we've produced two types of artifacts, which we've called "edge" and "stable". Edge artifacts are produced every week or two; stable artifacts less frequently. Edge contains all the code in the repo at the time they were created, while stable contained only specific changes—typically, bugfixes—that we decided were important.
Edge releases have been easy to produce. Stable releases, however, required a lot of work. To produce stable releases (outside of the dot-zero releases like 2.14.0), we have to cherrypick specific changes from the main branch of development and backport them on top of the previous stable release branches, altering the code as necessary to account for the intervening changes, and also validating that the fix still makes sense. As the number of these releases grows, this burden only escalates.
So, the change is: for 2.15 and onwards we are not continuing the work to produce those stable releases. We are continuing to produce edge artifacts.
Two reasons. First, this allows us to significantly increase the pace of development in Linkerd. We have an incredible amount of work ahead of us to deliver on an insanely ambitious roadmap—and to do it with the same laser focus on simplicity and security that makes Linkerd so great today; that allows it to take on trillion-dollar behemoths like Google and Microsoft and still come out ahead. To accomplish what we need to do, our top priority has to be low-latency iterative improvements to Linkerd that can be consumed readily.
The backporting and cherry-picking work that goes into producing a stable release is really hard. It's time consuming, and it is distinct from the work of developing Linkerd or fixing bugs. Removing the work of stable releases allows us to simplify our eng process dramatically. This isn't theoretical; we'll be blogging about some of the concrete improvements unlocked by this change soon.
The second reason—and I want to be upfront about this, even though normally we try to maintain a distinction between our commercial work and our open source work—is that this change makes space for Buoyant to build a sustainable business around Linkerd. The reality is that for the past eight years, billion-dollar businesses have been built on these public stable Linkerd releases without any reason to fund the project or contribute back in any way. That's not a moral judgment or a statement of fairness, but it is a statement of fact. If we want Linkerd to grow and improve dramatically, as we all do—we need to fix that now. This change is key to unlocking that opportunity for us.
This is the second half of the story. Over the past year, we actually have been building out a stability-focused distribution of Linkerd, with rigorous testing in specific environments, regression testing, backports and cherry-picking, and some additional features designed for sustained production use. The infrastructure here is incredibly expensive to build and maintain—but this is an easy expense to justify when it is tied directly to customer revenue.
At this point, one option would have been for us to simply say: ok world, use edge releases, pay us for enterprise, or find another path. That didn't really feel right. Instead, we also made our enterprise distribution of Linkerd freely available for production use at companies with fewer than 50 employees. (We may change that threshold. See below.)
Our intention here, to be frank, is to give you something significantly better than the stable releases of the past; to make it free for the majority of the community; and, as discussed, for larger companies, to use it as a mechanism to fund the project.
One big source of confusion we saw was around edge releases, and what, exactly, running one entailed. Are these experimental? Are they safe? Do they have security fixes? So, to clarify:
If you're using Linkerd, we'd love for you to run the edge releases, and we're really incentivized to make this work for you. For the health of the project, we need more people running these releases and helping us shrink the development cycle. By using edge releases and reporting any issues you find, you contribute back to the community in a really significant way, and we'd like to make this as viable an option as possible for you.
Another point of confusion was around the grace period. We want to ensure that everyone has at least 90 days of safety and stability to decide what they want to do, by decoupling the immediate reliability engineering decisions from financial decisions.
So, while most companies shouldn't have to pay for BEL, if you are large enough that you would have to pay, you have a 90-day grace period (until May 21st, 2024) before you have to commit to paying. And when you are ready to commit, we are happy to work around budgeting timelines and process—we are all too familiar with how these processes work. In short, we want to make this easy for you. If you're in this situation, or any unique situation, please get in touch and I promise we'll do our best to make it work for you.
Based on feedback from the community, there are a couple things we are considering changing.
We're considering shipping marker releases that denote the announcement of 2.15, 2.16, etc. We typically work towards big feature milestones (like mesh expansion!) and it feels a little empty to not have a corresponding artifact to show off, especially given how little work this would be. This same idea could potentially be applied in other situations, as long as the additional work imposed on the maintainers was minimal. Input welcome on this.
One clear piece of feedback was that the pricing model for BEL doesn't work for many Linkerd adopters at smaller companies with substantial Linkerd deployments. For these adopters, the current pricing model simply wouldn't allow them to make the move to BEL, and I consider this a serious problem. We've got a couple of ways we could address this and we're actively discussing them now. Stay tuned for updates on this. (And if you have a specific datapoint about your company that we can use as input, I am all ears.)
One thing that's a little unclear in our announcement is what Linkerd 2.14 users should expect going forward in terms of security patches. We don't want to leave anyone out in the cold here, so we probably need an explicit statement about what kind of commitment we have to patching security issues specifically. Feedback welcome.
Feedback from the community has been incredibly valuable in helping us refine the plan already. We've created a #215-changes channel on the Linkerd Slack for this purpose, or as before you can email me directly at william@buoyant.io. (Sadly, the Reddit and other social media threads where I was active have started to become a little saturated with... suspicious... activity, so I will mostly be listening in the Slack.)
A complete guide to everything the modern enterprise architect needs to know about service meshes, including mutual TLS, zero trust security, eBPF, sidecars, and more.
.svg)