Continuous Secrets Delivery
June 5, 2015 | DevOps | Josh Bregman
Hanging out with the IT Crowd in London.
I’m sitting in the airport lounge in Madrid on my way back to Boston after a whirlwind 2 days at the Infosec Europe conference. I was summoned into service at the last minute when Elizabeth unfortunately wasn’t able to make the trip, and I took her speaking spot at the DevOps exchange. I’ve been at Conjur for just a few weeks so this was my first opportunity to really interact with the DevOps community personally.
First of all, I want to thank Alan Schimmel from DevOps.com and Wai Man Yau from Sonatype for both being exceptionally gracious hosts for the events. Wai Man and the entire Sonatype crew including Josh Corman were a blast to hang out with. Thanks to them (and a few pints) – project Cauldron is dead – long live project Summon. A great example of failing fast and an judgement free post mortem on my talk.
I also had the great pleasure of meeting Justin Arbuckle from Chef and watching his presentation. I’ve always been a fan of Jim Collins’ – Good to Great – so I loved Justin’s talk entitled “Velocity is your Hedgehog, Compliance is your Fox”. Justin’s idea is that one thing that DevOps enabled enterprises know is velocity – faster, faster, faster. This idea is what drives the flywheel….keep pushing, and pushing, until it takes off. Compliance is the fox means that compliance concerns will constantly come up and will need to be addressed, but NOT at the expense of velocity. Easier said than done to be sure.
After my talk, Justin said that he liked the concept of Continuous Secrets Delivery that I discussed in my talk. Continuous Secrets Delivery is a methodology that we’ve found works across customers and industries to ensure that secrets management isn’t the hurdle that trips up the team and ruins your velocity.
By following this approach, teams can avoid adding to their security debt that ultimately makes the information security folks call a timeout on their project.
Here it is:
Define a Policy
Much has been said about the cultural challenges of bring Dev and Ops and Security together towards a common goal, but there needs to be basic alignment on which people and code, based upon their role, need access to which secrets. This policy should be collectively reviewed and committed to source control. The configuration of that policy should be strictly codified – as in the case of Conjur’s Policy DSL – but regardless of format, this step is critical. Information security doesn’t always want to say “No”, so it’s their responsibility to be proactive, and give guidance on how to apply their risk management framework and best practices to secrets management.
“Get Your Secrets into Source Control”
This is the working tagline of project Summon. The idea is that you create a secrets.yml file for each project (application or service). This file declares the secrets that the software needs access to, and into which environment variable each piece of secret data should be “summoned”. The secrets.yml file has no actual secrets in it, and can safely be checked into source control.
The “summon” command line tool can then be used to launch the application, container or service. For example:
# Start Rack-based server app (e.g. in an upstart script)
$ summon rackup
# Run a Dockerized app
$ summon docker run
Create Host Factories
A host factory is a mechanism for “lifting” a new host (machine, container, or PaaS application) into a privileged computing role. For example, Conjur’s Host Factory generates a new host (machine) identity and grants it a specific set of roles. Getting this right is of critical importance in achieving velocity with compliance. If there are any manual steps in this process – like changing network settings or manual approvals – then you can’t go fast. If every machine has the same identity, that’s obviously fast, but is it secure? All of those machine’s audit records will have the same identifier. This calls out for something dynamic like a token service (OAuth Dynamic Client Registration) or Conjur Host Factory to allow machines and or tool chains to automate in a secure fashion the assigning of the correct identity.
Make it Highly Available
In the Summon model, the system that provides the secret is called a “secrets provider”. This service needs to be highly available and performant across entire infrastructure, including multi-cloud, multi-region, and hybrid cloud environments. To be fast, the secrets need to be securely distributed to the processing environment of the application.
Integrate with the DevOps Toolchain
The final step in scaling out the application it to integrate with the other tools in the toolchain; for example Puppet, Chef, SaltStack, Ansible, AWS, Docker, Heroku, Kubernetes, etc. Conjur provides integration libraries for many of these.
Ironically, many teams start with the last step, and try to turn their configuration management system or cloud storage service into a secrets server. Configuration management tools weren’t built for this task. There is an expression “Sometimes when all you have is a hammer, everything looks like a nail.” Using configuration management for secrets management is a sure fire way to slow the velocity of your project. Gaining alignment up front can be organizationally and culturally challenging, but it’s the only path forward to maintain velocity and compliance. It’s a much easier conversation to have than AFTER something bad happens or information security won’t allow the application to go into production because they find plain text database passwords in Jenkins log files, Chef data bags, Puppet Hiera, Github repos or an S3 bucket.
This 5 step process of Continuous Secrets Delivery has been born out from years of experience solving this problem across customers and technologies, but we want to hear from you. What’s your process? Did we miss anything?