How To Securely Scale Continuous Integration


December 9, 2015 | DevOps | Alan Potter


At Conjur, we’ve developed the principles of the Ten-Factor CI Job to inform how we build and test our software. Applying these principles has allowed us to create our software rapidly, reliably, and reproducibly. 


Here’s how we’ve applied each of the ten factors:

1)  Codebase: All code, including the description of the Jenkins jobs, gets committed to GitHub repos. We use the Jenkins Job DSL Plugin to create our jobs and these definitions are stored in a repo, too. Each of our jobs has a single script that runs tests.

2)  Dependencies: Most jobs use Docker containers to manage dependencies. Slaves are all configured using a Chef cookbook so we know exactly what is installed. Slaves are conjurized: we use Conjur SSH to control who gets access to the slave and what commands can be run on it.

3)  Config: We use Summon to fetch any secrets required for build, test, and deploy from a Conjur server.

4)  Backing Services: Many of our jobs rely on a running Conjur appliance. We use a local Docker registry to store a preconfigured appliance image. Any job that needs an appliance can start a container from it.

5)  Build, Release, Run: Storing the DSL for our jobs in git means that they are versioned. If we need to recreate a build, we can go back and find the exact job that created the artifact.

6)  Processes: All of our Jenkins jobs delete the workspace before they start. Any required images are pulled from the local registry as part of initialization.

7)  Disposability: Docker containers make this easy, but there are two things to watch out for: transient, job-related images should build quickly, and containers should get run with –rm. Maximizing layer reuse is an important technique for making the transient images build quickly. For example, many of our projects use bundler for gem management. The Dockerfile used to build an image to test such a project has lines like these in it:

  • ADD Gemfile Gemfile.lock ./  # as well as the gemspec, if appropriate
  • RUN bundle install –deployment
  • ADD . ./ # add the rest of the source code.

Structuring the Dockerfile this way allows reuse of the layer created by the bundle install, as long as the Gemfile and Gemfile.lock remain unchanged…This can be a big time saver.

8)  Dev/Prod parity: That single script from Factor 1 will run on a dev laptop, as well as on the Jenkins slave. As mentioned above, our Jenkins slaves get created from a Chef cookbook. We use Vagrant to create the slaves, allowing us to spin up an AWS instance (connected to our Jenkins master), or a VirtualBox VM (for developer use).

9)  Logs: Jobs that run in containers write to stdout and stderr, each of which gets captured by Jenkins. Test output gets written in JUnit format and we use the Jenkins JUnit Plugin to publish them.

10)  Audit: All access of secrets by a Jenkins slave during build and test gets audited in Conjur. If other information needsto be captured,  custom audit records can be created.

Here are some other things to consider:

Make sure your Jenkins slaves are cattle, not pets. Sometimes, the expedient thing is to “fix the build machine” when a job breaks. But if this isn’t also accompanied by updates to the machine description (e.g. by changing the Chef recipe), you’re asking for trouble.

As your teams grow, you can use Jenkins folders along with the Conjur Jenkins plugin to manage permissions for Jenkins jobs. Even if you conjurize your Jenkins slaves, each still only has a single identity. If teams, each of which should have disjoint sets of permissions, share Jenkins slaves, it’s hard to manage and audit what the teams are doing. If you use the CloudBees folder plugin to segregate teams, you can use the Conjur Jenkins plugin to give an identity to each job run by a team.

Having separate identities for “build” and “publish” follows the principle of least-privilege and allows for better auditing of the provenance of an artifact.

Check here for more details on how Conjur helps secure jenkins.





Keep up-to-date on security best practices, events and webinars.

Share This