Security Holes Persist in Kubernetes: KubeCon 2016 Recap
November 28, 2016 | DevOps | Dustin Collins
Earlier this month I had the privilege of attending KubeCon, a two-day conference all about the latest and greatest developments in the Kubernetes ecosystem. Held in downtown Seattle, the event sold out with 1,000 attendees. Next year, KubeCon will come to Austin TX, in an even bigger venue.
Excitement was high. Kubernetes 1.4 had shipped shortly before the event and much of the crowd seemed focused on either upgrading their existing clusters and/or taking advantage of new features in the release. As Conjur’s developer advocate and resident DevOps cheerleader, I was most eager to learn about how Kubernetes users addressed their various organizations’ governance and compliance requirements.
Learning Kubernetes at KubeCon is like learning to drink water from a firehose, but as I looked over my pages of notes (and recalled discussions from previous ContainerDays events) a few points about Kubernetes and the direction of its ecosystem became clear. Excitement is high, the ecosystem is exploding, but as is often the case in the early days of emerging technologies, security is a second-class citizen. The following points are my big takeaways from the conference.
1. The ecosystem is exploding with vendor and open-source tooling
The Cloud Native Computing Foundation (CNCF), the new steward of Kubernetes and of a growing list of related tooling, is working on a Cloud Native Landscape reference that maps the technologies and tools available in the “cloud native computing” space. A work in progress, this reference will surely grow over time. If you’re been following the DevOps space lately, many of these logos will be familiar.
Analysis paralysis could become a problem here; there are simply a lot of vendors and technologies in this space. Some are competing head-to-head, but several of these technologies overlap in their feature sets. Expect to spend a lot of time evaluating several technologies to find the one that fits your use case.
It is interesting to note that 19 of the 37 sponsors for KubeCon are offering a ‘wrapped’ or hosted version (PaaS) of Kubernetes. Most are targeted towards enterprise customers that require additional features not implemented by Kubernetes itself (SSO, multi-datacenter dashboards, etc). Key selling points of these PaaS versions of Kubernetes are ease of deployment and minimizing operational maintenance. However, you do run the risk of lock-in or feature drift from Kubernetes core.
Some of the new open-source tooling that people were talking about about at KubeCon:
- Anchore – Image security scanning CLI
- Kompose – Renders Kubernetes resource YML from a docker-compose file
- Calico – L3 network mesh with built-in traffic authorization
- jsonnet – JSON templating tool, useful for Kubernetes resource files
- linkerd – Service mesh for cloud-native applications
- Helm – Kubernetes package manager, just reached 2.0
2. Setting up a proper production cluster is still pretty difficult
Kubernetes is composed of a number of technologies that work together to provide its feature set. This modular architecture makes it easy to scale your cluster efficiently at the cost of increased operational complexity. Several projects are now available that make it easier to stand up and maintain a Kubernetes cluster. They each have their own assumptions, benefits and drawbacks.
For local Kubernetes development, minikube is simple to get up and running. This project allows you to run and manage Kubernetes on a local virtual machine. minikube is a great way to try out Kubernetes locally. Be aware, though, that all of Kubernetes’ components are running on one machine – some features of Kubernetes like load-balancing across nodes won’t be possible. CoreOS has also released tooling for running single and multi-node clusters locally with Vagrant.
Setting up a production Kubernetes cluster can be a pain, and everyone knows it.This is why we see so many wrapper/PaaS versions of Kubernetes being made available. There are many different tools and projects that make launching a highly-available cluster easier. kops, from the Kubernetes team, provides cluster CRUD operations on top of AWS. CoreOS maintains kube-aws, a similar tool. Most platforms now have tooling for running Kubernetes clusters. Launching a cluster from scratch is also an option if you’re curious about how everything fits together.
3. Kubernetes Secrets has an Achilles Heel, and it is etcd
Secrets were added to Kubernetes in v1.3. Users create secrets and specify the pods that can read them. Sensitive credentials are made available via a temporary file system running on the cluster nodes running the pods that need them. Secrets are easy to use and provide a basic level of security for credentials that shouldn’t be built into images or passed via environment variables. That said, there are a number of potentially serious risks to consider when using Kubernetes secrets.
From the Kubernetes secrets documentation:
In practice, the engineers I talked to at KubeCon were either locking down their etcd cluster to only senior operations staff or searching for a better solution for providing secrets to their Kubernetes applications. The coarse root/non-root authorization of the Kubernetes API is also a big issue for many companies. When least privilege isn’t enforced you end up with an unreliable system.
Conjur is currently planning support for making secrets available to Kubernetes. Robust role-based access control and immutable audit are standard features for any secrets management solution.
4. Governance and security are still in the back seat
There are a number of security problems the community will need to address before Kubernetes can become a viable strategy for many organizations. Secrets insecurely stored in etcd, underdeveloped authentication/authorization for interacting with the API, escalation issues on the nodes, lack of an immutable audit log; any one of these can be a deal-breaker for organizations that want to allow more than a handful of engineers to use Kubernetes.
There are a few interesting projects attempting to tackle these issues. Calico allows you to declaratively define which pods can talk to each other at the network level. dex is an OpenID Connect provider that plugs into Kubernetes to provide authentication from a number of pluggable sources. Securing Kubernetes will be a hot topic in 2017. Look for lots of activity in this area from Conjur and others.
KubeCon was a fantastic experience. I came to the event to hear stories from people about running Kubernetes in production; how they set it up, what challenges they face, and where they’re going from here. The development workflow and resource utilization Kubernetes enables are compelling reasons for many companies to rethink how they will deliver software in the coming years.
Governance and security are still ‘in the wild west’, as I heard a few times at the conference, but work is being done to hold Kubernetes to the same standard we require for other parts of our infrastructure. If secrets management, authentication, and authorization of your Kubernetes deployments is an issue you’re thinking about, I’d love to hear from you at [email protected]. Conjur is building out our support for Kubernetes, and we welcome design partners.