Conjur and Jenkins
| DevOps |
Jenkins is a popular system for build and continuous integration. Here at Conjur, we use Jenkins extensively, and we would like to share how we have secured our Jenkins system using Conjur.
Securing Jenkins with Conjur has three great benefits:
- Both web and ssh login are powered by Conjur, via the PAM system
- The secrets that are used by Jenkins jobs are secure, organized, and kept off the hard drive at all times
- As a result of (1) and (2) above, the artifacts produced by Jenkins are highly verified and trustworthy
Our Jenkins system is utilized by the following user groups:
|build||Users who manage the build system|
|developers||Developers who use the build system, by pushing code and starting jobs|
We already had these groups defined in Conjur, with the appropriate members.
The next step is to assign a Conjur identity for all “robots” in a system; machines, VMs, jobs, processes, etc.
Because this project was focused on Jenkins, we created a layer for Jenkins:
$ conjur layer:create production/build-1.0.0/jenkins
Then we created the host identity for the Jenkins VM and added it to the layer.
$ conjur host:create ec2/i-74ab831f $ conjur resource:annotate host:ec2/i-74ab831f name=jenkins/master $ conjur layer:hosts:add "production/build-1.0.0/jenkins" "conjurops:host:ec2/i-74ab831f"
This set up works for a single Jenkins server; more complex setups would use a layer for the Jenkins master and additional layers for Jenkins slaves.
Conjur provides fine-grained authorization for user login, without the use of a custom login agent. The login system uses the standard toolchain of openssh and PAM with LDAP. LDAP is provided by a Conjur adapter which exposes the Conjur authorization rules over the
With the Conjur layers and groups already defined, all we had to do is assign privileges on the Jenkins layer to the
build managers are authorized to admin the Jenkins layer;
developers are authorized to login with regular user privileges.
Finally, we configured Jenkins to delegate login to the PAM system by selecting “Unix user/group database”.
When a user logins in to the Jenkins web UI, they are authenticated with their Conjur password, and then authorized via Conjur.
SSH access uses the same login procedure as the Jenkins UI. After openssh authentication, Conjur authorization determines whether the user is granted access and assigns the unix group. We use two user groups:
/etc/sudoers.d configuration allows sudo by
developers do not have any sudo capability.
Building and testing Conjur images such as our AWS Marketplace AMIs are used by Jenkins for these general purposes:
- Updating resources in Amazon S3, primarily for our web sites
- Pushing code to our continuous integration (CI) environment, via Heroku
- Pushing code to our continuous integration (CI) environment, via ssh and chef-solo
Before we secured the system with Conjur, secrets were primarily stored in text files on the filesystem, and were managed by the build team members.
We sought to improve Jenkins secrets management, to:
- Ensure that secrets are not captured in Jenkins backups, by keeping them off the filesystem.
- Achieve separation of duties in secrets management by moving secrets to secure, auditable storage with well-defined access control rules.
- Automatically rotate all secrets used by Jenkins.
To achieve this, we moved each secret from the filesystem to a Conjur variable. A variable is a version-managed, encrypted, access controlled data value; it’s perfect for secrets such as private keys, passwords, and cloud identities.
Each variable has two privilege levels: update and execute. update privilege confers the ability to change the variable value (actually, add a new version). execute privilege confers the ability to fetch the variable value.
Our Jenkins server uses the following variables:
|ec2/ami-build/identity||AWS identity used to launch a VM and capture an image||[group] cloud-managers||AMI build and test jobs|
|ec2/ami-build/private_key||AWS private key used by Vagrant to access and configure a VM||[group] cloud-managers||AMI build and test jobs|
|ec2/ci-push/private_key||AWS private key used to push code to CI servers via ssh + chef-solo||[group] cloud-managers||Jobs which push via Chef|
|s3/website/identity||AWS identity used to update Conjur web site assets in S3 via s3sync||[group] cloud-managers||Jobs which update web site properties|
|conjur-ci/admin-password||Admin password used by the CI deployment of Conjur||[group] ci-managers||All jobs which run tests against the CI stack|
We defined each secret, and populated it with the appropriate value. Then we granted execute access to the Jenkins layers.
With the secrets in Conjur, we needed a way to provide secrets to the tools used by the Jenkins jobs. For example, we use Vagrant to launch EC2 VMs and create AMIs. The Vagrant ec2 provider requires environment variables containing an AWS access key id, AWS secret access key, private key, and private key name. As a second example, we use s3sync to push static web site resources to S3. s3sync requires a file containing the AWS access key id and secret access key.
Exposing a Conjur variable as one or more environment variables is just a simple bit of Bash scripting:
$ cat get_variable.sh #!/bin/bash --login varname=$1 if [ -z "$varname" ]; then echo "Variable name not provided!" >&2 exit 1 fi # policy.version is production/build-1.0.0 policy=`cat policy.version` /opt/conjur/bin/conjur variable:value $policy/$varname
In Jenkins, this script is used in an “Execute shell” step
ADMIN_PASSWORD_FILE=$(get_variable.sh "ci/admin-password") bundle exec rake jenkins
For tools which expect credentials as files, we were a little more clever. Since we want to keep credentials completely off the filesystem, we fetch and store secrets to
/dev/shm, a memory mapped filesystem. In addition, a cron job periodically erases secrets from
At the completion of this project, our Jenkins server now contains no secrets on the filesystem, and we have separated the right to create and update secrets, from the right to fetch and use them. Furthermore, we have a complete activity record of authorization activity that affects the Jenkins system:
- Permission policy changes
- Uploads and downloads of secrets
- Users added to and removed from the
And we have powerful privilege access capabilities:
- Unified authorization of Jenkins UI and ssh access
- Instant de-comission of a compromised build server
- Instant de-authorization of user access to the build system
Moving secrets management into Conjur also affords system management and reliability benefits. For example, should a hardware problem with our Jenkins server occur, it will be very easy for us to recover. We regularly capture snapshots of the Jenkins data volume. To bring up a new Jenkins server, all we need to do is launch a new VM, run our Chef scripts which install and configure Jenkins, mount
/var/lib/jenkins to a volume created from the most recent snapshot, and assign a new host identity to the VM. ALL the Jenkins jobs and processes will then fetch the secrets they need automatically from Conjur, when they are needed. Freedom from worry about manually maintaining secrets is a big productivity boost as well as a great security and compliance benefit.