PaaS-ing Secrets: Security Considerations for Pivotal Cloud Foundry Deployments
November 7, 2016 | DevOps | dustin byrne
In working with our customers who use Pivotal Cloud Foundry’s Platform-as-a-Service (PCF’s PaaS), Conjur’s technical field teams have identified common security concerns for production deployments. These challenges are sufficiently serious to block important applications that handle sensitive data from going into production. The issues described below should be considered by developers and application architects early in the PCF application development process, and they are likely common to other cloud PaaS platforms too.
Organizations that deploy on PCF’s PaaS are optimized for speed of application development. By virtue of PCF’s low-friction deployment workflow, applications can germinate, evolve, and scale quickly, sometimes at breathtaking speed.
But two operational problems for PaaS’s exist around introducing sensitive access credentials like database passwords and cryptographic keys into the ephemeral servers that get created:
- How are the credentials’ environment variables populated, without delay, in a trusted, audited manner?
- What are the properties of each credential relating to trust, isolation, and control?
Pivotal’s Chief Security Officer offered a compelling view earlier this year on how to make enterprise security more effective by making infrastructure more dynamic. He called for faster repaving of servers and applications (restoring them to a known good state every hour), repairing vulnerable operating systems within hours of a patch being available, and rotating data center credentials frequently — as often as every minute! As he noted at the end of his post, PCF doesn’t offer automated credential rotation yet, and that gap is proving problematic for some customers trying to get security-sensitive applications into production.
The question of credential rotation sounds deceptively simple, but when you dig into the requirements that large, security-conscious enterprises have, it’s much more complex as the rest of this post will illustrate.
Population of Credentials
The simple separation of secrets into configuration environment variables is a good start (see Heroku-founder Adam Wiggins’ 12-factor app best practices), but that is only the beginning. For larger organizations that have certain compliance requirements, an added goal is also the separation of responsibility. A credential-trusted operator should be separate from a deployment-trusted operator. Binding credentials in environment variables either crosses those concerns because the deployment-trusted operator now needs to have access to the credentials. The naive alternate would be to have a two-operator rule for every deployment action. Unfortunately, that would likely slow deployments and erase some of the agility benefits of a PaaS, per Factor X: Dev/Prod Parity.
A slightly better approach is Cloud Foundry’s concept of user-provided services (via cf cups and cf uups)
http://docs.cloudfoundry.org/devguide/services/user-provided.html#user-cups that can make shared credentials available outside of the lifecycle of an app instance. In short, that provides for some separation of concerns, and allows developers to release without hinderance. That’s a good next step. However, there’s no audit trail, and every application in that service group has visibility to all credentials. There’s no finer-grained policy mechanism available.
Properties of Credentials
More broadly, some questions that affect PaaS credentials include:
- Is there a clear, declarative policy for each class of credential enforcement? Can it be audited and trusted? How are least-privilege policies implemented?
- What is the authentication and authorization workflow for programmatically issuing new credentials? What are the delegation and trust mechanisms around that?
- For elastic hybrid clouds, how do you isolate public vs. private cloud credentials? Are there different policies in different environments?
- What are the audit control points that can keep up with the rapid nature of a PaaS? How tamper-resistant are audit logs?
- What are the secrets rotation policies and how do they get implemented? Can routine rotations be handled with minimal disruption?
- Can you securely, automatically issue credentials and secrets to supporting components like a load balancer’s SSL private key with the same level of policy and audit control?
- What is the maximum velocity at which you can issue unique credentials? In a standing-start reboot scenario, is your credential system a bottleneck?
- Where are root credentials stored and accessed for disaster recovery?
- Is there a break-glass procedure for emergencies?
- What are high availability / resiliency options in the event of partial failures?
Conjur PaaS Design Pattern
Figure A. 1. Security team authors and publishes declarative, reusable security policy. 2. Dev team authors code and pushes to PCF as normal. 3. Application is instantiated as normal. 4. Conjur Summon, included in asset, authenticates to PCF/Conjur authenticator. 5. Authenticator retrieves token from Conjur Server (or a follower) and returns it to Summon. 6. Token is submitted to Conjur Server for a particular secret, e.g. a database password, per security policy.
A Conjur implementation can provide solutions to all of these needs. In this application design pattern, application credentials are issued from a Conjur master server or a remote follower if in a hybrid cloud. A PaaS instance authentication service ties the PaaS orchestration mechanics to the credential service. For each requesting application instance, the authentication service compares the metadata provided by the orchestration service upon app provisioning with an incoming request for new credentials. If all criteria are met, an ephemeral token is issued that can be submitted to the Conjur server for the needed credentials, say a database server.
The presentation of “identity”, the fetching of the token, the subsequent exchange for a valid database credential and then the injection into the running environment variables is done automatically using Summon, Conjur’s open source secrets injection tool. Incidentally, Summon can be used to inject secrets into any application’s environment, so a standard, policy-driven, audited process can be implemented for supporting components too.
By using Conjur both to create a declarative policy and enforce that policy at run-time, the design pattern effectively governs trust uniquely to each instance. A clustered Conjur set-up provides for resiliency and high availability with a tamper-resistant audit trail. The Conjur cluster can be easily sized to handle extreme scaling for high-volume application or standing-start situations. And by having a control point for credentials, rotation of credentials can be done programmatically to minimize disruptions.
If desired, different policies can be applied for different categories of instances (e.g. internal vs. external) in order to manage exposure and risk in less trusted environments.
– !variable database/url
– !variable database/username
– !variable database/password
privilege: [ read, execute ]
Figure B. Sample Conjur policy
At Conjur, we’ve observed that for enterprises with high-value applications like banks, infrastructure providers, and governments, the rapid deployment authority of PaaS creates a new class of security challenges. Access to account databases and sensitive functions like treasury servers is necessary for applications. But that access must be meted out in the smallest measure, and then each access credential must be carefully isolated for risk, audit, and governance compliance.
A manual provisioning process for those access credentials won’t cut it. Manually creating a few credentials a day doesn’t stand a chance against a team of developers that’s releasing apps and refreshing several times an hour… or worst case, in a infrastructure re-start requiring the provisioning of potentially millions of secrets per minute.
Conjur’s core architecture is designed for robustly handling these and other authentication and authorization challenges. Not only does it automate the management of secrets at the speed and scale demanded by developers of modern applications, but it also creates a visual representation of the security architecture and provides an audit trail that enables the the business to deploy with confidence.
If you want to learn more, check out our whitepaper on Privileged Access Management for the Cloud & Container Era or see it in action by requesting a demo here.