Dynamic Credential Access: Jenkins and Conjur, Revisited
July 31, 2015 | DevOps | Alan Potter
Kevin previously described how Conjur secured Jenkins when building our various artifacts. We currently use release-bot to publish our artifacts, but this approach hasn’t scaled well — it’s hard to get Heroku to run the tooling we need for publishing. This post will describe a new architecture for getting our build artifacts to their various destinations.
Security when releasing code is as important as when building it. Not having a secure release process means anyone can push anything and claim it’s a release. Because Jenkins is open, having a Jenkins job push an artifact is inherently insecure and also offers no chance of auditability.
Separating the release machine from the build machine allows for better security. Instead of having a Jenkins job publish the artifact itself, it interacts with a separate release server. The release server has the credentials to publish to various destinations (e.g. RubyGems.org, Heroku, Docker Hub, etc) but doesn’t have the ability to build artifacts or modify them in any way. This is consistent with least privilege: the Jenkins job can do builds but not push releases, the release machine can push releases but not do builds.
After a Jenkins build job runs, a Conjur employee starts the promotion process. A Makefile that’s part of the project bundles up the build artifacts, then invokes a script to kick off deployment. If the output of the build job is a set of files (e.g. the output of building a gem), the script uses scp to copy the artifact to the release machine. If the output of the build job is a docker image, the build job pushes it to our local docker repository.
Once the artifact is available to the release machine, the Jenkins promotion job uses ssh to connect to the release machine. The remote user can only execute a single command on the release machine: he can run a publishing script, but can’t do anything else. Because the release machine is granted powerful privileges, it’s important that access to it is very limited.
The publishing script labels the artifact with the build information, then asks Conjur for deployment credentials. Because it runs on a machine in the release layer, the script is authorized to use these credentials. It publishes the artifact to its destination, then writes a Conjur audit record indicating that it did so (or failed to do so).
This architecture will allow us to add new kinds of artifacts more easily than if we stuck with release-bot. We’ll be able to push them out securely, and show that only sufficiently privileged entities publish releases.
Click on the video below to see how Conjur helps give Jenkins jobs temporary access to the secrets they need to run tests and build artifacts.