Cloud Security Step by Step – The Trusted Image
February 17, 2014 | DevOps | Kevin Gilpin
Organizations like Amazon, Canonical, and Red Hat provide VM images (e.g. AMIs) of popular operating system distributions. It’s common for application teams to launch these images directly “as-is” and install their software onto them after launch.
However, there are advantages to specializing one (or several) of these images and using it instead of the base OS image. A base image which has been supplemented with configuration and installed packages that suit the needs of your application is referred to as a “Trusted Image”.
Why Make a Trusted Image?
VM Identity and Authorization
Authorization is a big deal to us. It’s the capability that you need in order to build a layered, defensible infrastructure.
You can build some features into your trusted image that will help establish an identity for each VM, and help the VM to get permissions. For example, in EC2, a trusted image can expect to be launched with an IAM Role. IAM Role grants permission to read a specific file from S3 which contains bootstrap credentials. These bootstrap credentials can be used to access other resources which the VM needs to join the application, such as database credentials, system passwords, API tokens, and other sensitive information.
The more downloads that a VM requires in order to provision and configure, the more likely that something will go wrong. Network errors, outages at package repositories, and random download errors can all render your VM unusable or unstable.
By building base packages into our trusted image, you are minimizing the chance that network problems or problems at external sites will prevent your VM from launching successfully.
If there are configuration options that you want to ensure on every VM, you can build these settings into your trusted image. For example, firewall (iptables) configuration can be applied to the trusted image, and will be in place to protect every VM that you launch.
If you want to use SSL or TLS inside your application (and you should), you will need to establish trust in a root certificate. Inside your own cloud account, it’s a good idea to simply create your own master certificate, and use it to issue certificates that you distribute throughout the rest of your infrastructure.
DO NOT succumb to the urge to skip certificate validation! Cert validation takes a little work and know-how to set up, but disabling cert validation is really not acceptable, because it defeats one of the primary reasons for the existence of SSL: assurance that your code is talking to right server and not an imposter.
When you build your VMs from a trusted image, all you need to do to enable SSL cert validation is store root certificate on the image itself. Now you can establish fully trusted communication within your network, based on the trust relationship that you established when you “baked in” the root certificate.
Installing and configuring SSL certificates is not as simple as it should be. You’ll need to do some or all of the following:
- Place your certificate into the openssl certs dir (the command
openssl version -dwill tell you where to look)
- Run c_rehash to update certificate symlinks
- Append your certificate to the trusted certificate list, in the parent of the certs/ directory
Unfortunately the details vary from tool to tool, you’ll have to do some Googling and experimentation.
openssl s_client -connect host:port
is a useful command for testing the SSL negotiation.
Built-in Service “Wiring”
Do you want to ensure that the syslog of every VM you launch is sent to your logging server? Or to ensure that every VM is hooked up to a monitoring system? Or to ensure that a specific version of Python, Ruby, Java, Node.js, etc is pre-installed on every VM that you launch? How about client libraries like AWS tools or boot? How about security scanning (e.g. anti-virus)?
Build these into your trusted image.
If you are installing the same packages on every VM that you launch, you are wasting quite a bit of time waiting for those package installations to occur. Build them into your trusted image, and your VMs will launch much faster.
Not only will every VM come with these features pre-installed, you will also have these systems configured and runningbefore your configuration management scripts start running. So you can even log and monitor the configuration process itself, and use these facilities during the configuration process.
Security updates from vendors are frequent. Any new image that you build will have all the latest security patches built into it; but your images will quickly get out of date as patches are released by your OS vendor. It’s not uncommon to get more than 100 patches each month from Canonical (Ubuntu) and/or RedHat.
If you configure your image to apply security patches when it launches, your rapid launch will gradually slow down (and become more likely to fail) as the VM downloads and installs all its patches. But if you don’t patch, you are taking security risks (admittedly, it’s hard to know what those security risks actually are…).
Another approach is to configure VMs to patch themselves on a schedule, for example, nightly during a “quiet period”.
Patching is tough. We’d be interested to hear about your experiences with this problem!
How to build a Trusted Image
A trusted image starts with a base OS distribution such as CentOS, RHEL, Amazon Linux, or Ubuntu. On top of this image, you add the standardized packages and configuration that you want to ensure will be present and operational on all the virtual machines used by your application.
Here’s our process for AWS, using Vagrant, Chef, and Jenkins:
- We use Vagrant to launch local (VirtualBox) VMs for development. Develop and test the image locally, among the dev/ops team.
- Once we have a release candidate, we run a Jenkins job which launches an EC2 VM, configures it, and captures an image with vagrant-ami. This job also tags the AMI with metadata such as the job name and job number.
- A downstream Jenkins job launches a VM from the new image and performs acceptance testing.
- If all the tests are green, we “promote” the image and start using it for downstream processes.
In a more complex system, you might build several trusted images. For example, one for Java app servers, one for web servers, and one for databases. Just apply the same procedure as above, re-using Chef recipes across trusted images.
Once you have built a trusted image, you will need a way to manage them. Image tags and other metadata features provided by your cloud can be helpful here. Treat your images like any other product of your build system: version them strictly and keep track of the provenance (history).
Thanks to Zachary Strowe, Dave Treff, and Lucia Capano for reviewing and editing this post!