Scaling Enterprise DevOps Security With Machine Identity – Secrets
| DevOps |
The rise of DevOps has coincided with an explosion in the number of worldwide software applications and services. Agile software methods, the development of mobile applications and microservices are pushing the number of deployable artifacts ever higher, while DevOps and continuous delivery methods are shortening release cycles from monthly to hourly. Enterprise operations and IT are concurrently developing and deploying hundreds of applications.
DevOps and continuous delivery personnel must avoid being inserted directly into the middle of large-scale and frequently changing environments, or they become overwhelmed with work in the form of support tickets, deployment requests, access control changes, security updates, network reconfiguration, loading new secrets, secrets rotation, etc. Therefore, we look to deploy automated solutions for these common security and access control problems. Agile methodologies encourage us to start with “minimally viable” solutions, even for for security, access control, auditing, etc. However, these initial solutions have hard limits in terms of scalability and enterprise suitability.
Once an organization reaches about 50 developers and 500 machines, the initial “DIY” solutions start to break down, and the need for more scalable access control and security solutions becomes clear.
Specifically, the developers, operations, and security teams together face three infrastructure challenges : management and distribution of secrets, machine login access control (e.g. SSH), and authorization of web requests between applications and services. In each of these domains, it’s essential that:
- Developers have agility and freedom regarding architecture choices and code updates
- Operations are managing a well-defined deployment architecture which is operated by a small number of stable supporting systems
- Security has visibility into the development and operational processes, and the ability to see their security suggestions implemented in a timely and transparent manner
These objectives are helped greatly by a strong and uniform system for assigning, tracking, and managing machine identity. Let’s explore how this is so, by taking the use cases one by one. In this post we wil discuss Secrets Management.
First, let’s frame the problem by looking at small-scale and large-scale secrets management:
Small-scale Ops is manually configuring a single machine, and placing the key on it by hand.
Large-scale Dev is pushing the code, and ops is managing the deployment through the control plane. The control plane is spinning up new applications, which are fetching code from the repository. Security is managing the secrets, which are being distributed out to the applications.
If you’ve implemented secrets management at scale, you’ve had to deal with these five requirements:
- Encryption; at rest and in transit.
- High availability; making sure that the secrets service is “always on”.
- Audit; detailed recording of the secrets activity.
- Granular distribution; sending secrets where they need to go, and nowhere else.
- Collaboration; developers, operations and security each need to be able to use and understand the system.
It turns out that requirements 4 and 5 are by far the most challenging problems. That’s why you’ll see the following flaws in many secrets management tools:
- Lack of scalability; due to reliance on manual intervention, or encrypting each secret uniquely for each client machine.
- Lack of granularity; for example, reliance on a single “API password” which can be used by any client machine to get any secret.
- Reliance on PKI; PKI is notoriously difficult to implement, and it provides authentication but not authorization (it’s not a good way to manage granular access).
- Obfuscation; only one of the three teams involved (dev, ops, security) understands how the system works.
Assuming we can rule out systems which are non-scalable, rely on a single master key, or provide authentication but not authorization, we’re left with one inevitable conclusion : a good secrets management system must provide both machine authentication and granular access control (e.g. via role-based access control).
That’s why machine identity and RBAC is the foundation of enterprise secrets management. So when you evaluate a secrets management tool, make sure you dig into its capabilities for authorization and granularity. Make sure that developers, operations and security teams all understand how a secrets management rollout will affect them.
In the next post we will break down SSH Management and how it fits into the Enterprise DevOps Security equation.