Securing Heroku Apps
April 23, 2015 | DevOps | Dustin Collins
In this post I will go into detail about how the Demo Factory works and what systems it needs access to, the Heroku-recommended approach to give access and its problems, and finally a friendlier approach using Conjur.
Demo Factory – how it works and what it needs
This is what happens when you request a Conjur demo:
- an EC2 instance spins up
- it is provisioned with Chef
- in ~10 minutes you receive an email with instructions on how to access your demo environment.
These are the credentials the app needs to be able to access to do its job:
- AWS keys to create and destroy instances
- Sentry DSN to capture any exceptions that may occur
- Hipchat API token to post messages to a room when demos launch
- Hubspot API token to create contacts
- Mandrill API key to send emails
- Keen API key to capture metrics for later aggregation
The Heroku Way
In accordance with the Twelve-Factor App manifesto, Heroku apps receive their configuration via environment variables, “config vars” in Heroku parlance. You set your config vars via the Heroku CLI or UI and access them as environment variables in your application.
When developing an app that requires secrets, you create a .env file and run the app via foreman. You can also use the heroku-config plugin to pull variables from Heroku into your .env file. You add the .env file to the project’s .gitignore file. This is where the workflow starts to unravel.
If any part of your secrets workflow depends on gitignoring files, this should be a red flag.
Why? You now have secrets sitting in plain text on your dev system. If you forget to gitignore that file or your machine is compromised, your secrets are public. We are working so hard in the DevOps community to achieve parity between environments – this lets us catch problems earlier when they’re less expensive. So why should our secrets workflow be different for development, test/CI and production? The way our applications and engineers access secrets should be the same no matter what environment they are in.
Rotating your secrets is a good idea. This should not affect your developer workflows. By gitignoring parts of your application, you’re putting the obligation of keeping secrets up to date on your developers. This is a distraction from what they do to deliver value – shipping features.
Finally, there is no trail of access to secrets in this workflow. Lack of transparency is the number one obstacle to compliance. If you have compliance obligations, the Heroku Way of handling secrets rules out using Heroku. It’s also worth noting that handing the keys to your kingdom to a third party with minimal transparency on how they’re used is not ideal.
The Conjur Way
The Conjur workflow is based on assigning permissions to identities. Users, hosts, third-party platforms; they all have an identity. Our goal here is transparency: which identities have access to which secrets and how are they using them.
This starts with a policy, a security topology that can be checked into source control. Conjur has a Ruby DSL for doing this. Here is the policy file for the demo factory:
In the policy we define the names for our variables (secrets) and give the layer “service” execute (get the value) privilege on those variables. Note that there is nothing sensitive in this policy file, we can check it into Github and make it public.
We load this policy into Conjur with the Conjur CLI like so:
conjur policy load --collection production policy.rb
This operation is idempotent.
You can now add values to the variables you defined in your policy file through the Conjur CLl like so:
conjur variable values add mandrill/api-key 98phe12p89hda
Now we need to create an identity for our Heroku app. We’ll create a host identity and add it to the policy’s “service” layer so it can read the variables we’ve defined in our policy.
conjur host create production/heroku/demo-factory-conjur conjur layer hosts add production/demo-factory-1-0/service production/heroku/demo-factory-conjur
A neat feature of Conjur is that you can apply host identity via environment variables. We’ll use this capability to assign an identity to our Heroku app. We can use the heroku CLI to set the config vars.
Now our Heroku application can use its Conjur identity to fetch secrets at runtime. The API key is a single secret that can be managed by ops without interfering with development, instead of multiple secrets that are being used in code and .gitignored out of source control, troublesome to rotate or change.
The last part of the puzzle is creating a mapping of environment variables to secret names and updating our Procfile to use conjurenv. conjurenv is a subcommand in the Conjur CLI that allows you to use a mapping file to expose secrets as environment variables to any process.
Here is the mapping file for demo-factory:
Now we can run our app prefixed with conjurenv and it will be able to access secrets via environment variables. Demo Factory is a Sinatra app, so we update our Procfile like so:
web: bundle exec conjur env run -- bundle exec rackup config.ru
How is this better?
For starters, we don’t have secrets on our systems. Once the command wrapped by conjurenv exits, the environment variables are gone. Dust in the wind.
By definition, the policy is minimally scoped to the application. It is the least privilege that still gets the job done, the smallest possible turtle.
Our Heroku app has a granular level of access to the secrets it needs through Conjur. Execute permission means it can only view the value of the variable. It does not have “update” or “read” permissions to change the value or view the access control relationships, respectively.
Our security policy is now checked into source control where our teammates can read and review it. Making a pull request on a permissions change? This is a bright new world.
Secret rotation is no longer disruptive. When someone updates the value of a Conjur variable the dev workflow is not affected. The next time the app pulls down a secret via conjurenv it will be the updated value.
Secrets are accessed in a consistent way across environments. Developers, your CI system and the Heroku app all use conjurenv when they need a secret.
We have a full audit log of secrets access for our app. You can view audit events in the Conjur CLI or UI.
The Conjur Way is not Heroku-specific. You can apply a Conjur identity to other services outside your direct control as well. Other PaaS solutions like Stackato or dotCloud, CI platforms like TravisCI or CircleCI, et al. As long as you can configure your application through environment variables you can use the pattern outlined in this post. The way you manage and distribute secrets should be the same, no matter your platform.
Our demo-factory is open-source, feel free to browse it on Github.