Secrets and Source Control: A Maturity Model
| DevOps |
Today I want to talk about secrets and source control. Earlier this week, Carlo from humankode posted this article: How a bug in Visual Studio 2015 exposed my source code on GitHub and cost me $6,500 in a few hours. In short, there is a bug in VS2015 that publishes your source code to a public repo even though you explictly say you want the repo to be private. His AWS keys were in the repo, a bot scanned them and launched a bunch of instances. That post has received a lot of attention, and it should. We should have full control over the visibility of the software we write, for a variety of reasons. However, public or private, if we check our credentials into source control we bear some responsibility for their misuse. In this post, I’ll cover three different ways to handle secrets in source control, framed in a maturity model.
From least to most refined, this is the maturity model for secrets in source control:
- Checked directly into source control.
- Tracked outside source control.
- Referenced by source control.
Let’s cover them one-by-one.
Checked directly into source control
Most agree that this is a bad idea, for several reasons. I’m going to to focus on the friction it causes in collaborative environments.
- Visibility into who/what has access to credentials is difficult. If you want to manage access to those credentials, you have to start locking down source control. And your CI system. And your Chef server. The list goes on. Now these tools that are meant for collaboration have a mess of rules to navigate. This limits their effectiveness and introduces bottlenecks (“file a ticket for access to that Jenkins job”).
- Credential rotations are now tied to application/infrastructure deploys. Tying deploys to rotations has the effect of slowing everything down and introducing interesting chicken-and-egg “opportunities”. Let’s face it: many of us have not achieved continuous delivery yet. Here’s what happens. An engineer has pushed a feature to a cookbook, but is still working on the tests, to be committed soon. We need to rotate a database password. So, do we push the code without tests or roll back the change for now to get the rotation deployed? Neither option is desirable.
- Encrypting credentials and checking them in is not a solution. It creates another problem, maintaining and distributing decryption keys. These keys are just new secrets.
Tracked outside source control
We have removed all secrets from our code and manage them in a system outside source control. This is a step in the right direction, but it only solves some of our problems and adds new ones to the mix.
- One of the learnings from the annual State of DevOps reports is that in order to be a high-performing IT organization, you need to be able to recreate your production environments entirely from source control. Credentials are a part of your production environment, but now they have been removed from source.
- Auditing becomes more difficult. We’ve now lost a key benefit of having your secrets in source: a history of who changed what, when. If someone rotates an API token or accesses a SSL cert, we should know when it happened and who did it. A security framework that doesn’t include an immutable audit log is not yet mature.
- We’ve placed an extra burden on developers. Without proper tooling, devs are now responsible for maintaining the credentials they need to develop features. This usually means that they are in written plain-text into environment variables or files like .gitignore. Rotations require extra communication. Getting access to new credentials is often not straightforward. If security is standing in the way of someone delivering value to your organization, they will work around it.
Referenced by source control
Credentials are dependencies of the code we write. If we fail to install the Postgres library or make the Postgres password available to our application, the end result is the same: our application does not run. Library dependencies are easy, we can list them directly in source. We should treat secrets the same way, but instead of storing their values we store where they can be retrieved from. To enable this pattern, we need a format to define credential references and tooling to support retrieval.
Earlier this summer, we (the devteam at Conjur) released a format and tooling to enable exactly this workflow. They are both open-source and you can use them whether or not you are a Conjur customer. We wrote them because existing OSS solutions didn’t meet our needs, namely: works with everything we use and is easy to deploy (we wrote this tooling in Go for easy distribution).
secrets.yml is a format that maps environment variables to where credentials are stored. An example:
summon is a command-line tool that will resolve the references in secrets.yml and inject them as environment variables into any child process. When the process exits, the secrets are not left on the system. You only need to prefix your command with ‘summon’:
summon chef-client --once or
summon bundle exec rackup or
summon aws s3 cp ...
Summon works with any tooling you already use that accepts environment variables (Chef, Puppet, Docker, any programming language, etc). It is also not Conjur-specific, we wrote it to work with pluggable providers (systems that hold your secrets). We’ve written AWS S3, OSX keychain and Conjur providers so far, but we’d love to see more providers in the community. Take a look and tell us what you think.
In conclusion, managing secrets for complex infrastructure and distributed applications is hard. We as engineers bear the responsibility of coming up with a solution that works for our organizations. Checking secrets into source control sets you up for future headaches, even if the respository is private. Removing secrets from source control creates new problems to solve. Together, we can work on better security tooling that reinforces collaboration and doesn’t significantly impact our velocity.