It’s almost January 2014. Do You Know Where Your System Access Credentials Are?


December 10, 2013 | DevOps |


Increasingly, corporate data and proprietary business processes are operating in the cloud and on Internet services. But many providers and services are antiquated in how they handle password repositories – and they’re even further behind in implementing modern Identity and Access Management Services.

For most organizations, access to the “crown jewels” has evolved from a few simple tokens that you shared with a small team, to a password quagmire of emails, wiki pages, sticky notes, and text files. Passwords are still the most important protection that you have on your valuable data and systems. Poor management of these “keys to the kingdom” can cripple small and large organizations alike – and is not something that you want to deal with reactively. It’s time to fix this problem – proactively.

What’s a system password?

A “system password” includes the keys, credentials and other secrets that protect your services, data and infrastructure – anything you have that you wouldn’t share with someone you don’t trust. For example:

  • The public and private keys (and passwords) that you use to log in to servers
  • SSL certificates that protect your web servers
  • Cloud identities that you use to make cloud API calls
  • API keys and tokens that your apps use to call third-party services
  • Encryption keys used for files and storage volumes

What’s wrong with ad-hoc management? (like I really have to explain 😉 )

What’s wrong with ad-hoc management? (like I really have to explain 😉 ) Of course, mismanaged access keys and credentials can lead to operational failures and data loss. But if I think about all the different ways this could possibly happen, the list would look something like this:

  • You can lose passwords, and lock yourself out of things.
  • Team members can accidentally apply damaging updates to the wrong environments, due to confusion over which credentials to use and excessive access privileges.
  • You can leave your system vulnerable to attacks in which an intruder gains an initially low privilege level, and gradually escalates by finding loose passwords and using them.
  • Your ex-employees still have shared passwords that you gave them – and may be using them
  • You can’t on-board people like temporary contractors easily because you don’t have a way to “sandbox” them appropriately. This costs you time and opportunity.
  • You’re unprepared when a big customer is interested in your technology and asks for a compliance audit. (Wouldn’t it be great to be ready when that day comes?)
  • You can’t easily rotate credentials since you have no way to track down all the places where the old credential is in use.
  • You can expire credentials that you think you aren’t using any more, but find out that you actually are still using them – when your important systems stop working.
  • You can’t decommission a user account, because that user is sharing a password that everyone else still needs to use.

Clearly, trying to protect yourself from each of these different scenarios – traditional ad hoc password management – is impractical. Where do you start?
Where’s the source? And how do you know?


So what’s the alternative?

Let’s look at this by analogy.

Not so long ago, it was common to find small software teams working without source control. They ran into nasty problems like:

  • Out-of-sync code between developers
  • Potentially valuable history being overwritten by new code and and lost forever
  • Loss of entire bodies of work due to hard drive crashes, computer damage, theft, etc
  • Difficulty in bug fixing because the team couldn’t figure out which code the customer was running.

Medium and large enterprises learned these lessons the hard way, and invested to solve them with enterprise source control solutions like Subversion, Git, ClearCase, Perforce and PVCS. Specifically, the community learned that, because code is so important, it must be managed with a system that is:

  • Centralized – and therefore authoritative
  • Durable – so that it’s always available and never lost
  • Multi-user – so that teams can collaborate effectively
  • Versioned – to preserve history
  • Role-based – for separation between code authors (developers) and code consumers (build + test infrastructure)
  • API-accessible – to facilitate automation
  • Searchable – so that things can be found

Today, there’s a universal understanding of the importance of source control in software development. And, thanks in part to today’s excellent, inexpensive source control systems (GitHub), even the smallest projects use source control from the start.

We need the same kind of approach for the management of access and key credentials.

What are good password management practices? Password management in the software industry is all over the place, with few good strategies and technologies. In most organizations, leaky password management is still the norm. It’s easy to think of good password management as simply locking away a single copy of a password. However, this strategy is laughably insufficient because passwords need to be recoverable and frequently used. What happens if you lose the only copy of a key, or the team member who has all the keys becomes unreliable?

To protect yourself adequately, your password management system should have the following characteristics:

  • Centralized location – so you can protect and secure your password “vault.”
  • Highly available and durable – so they are always there when you need them.
  • Encrypted – to avoid accidental exposure.
  • Group-accessible. Passwords should NOT be owned by one single user, they should be owned by a group. You don’t want critical system access to depend on one individual.
  • Versioned – so that critical keys won’t accidentally be over-written.
  • Searchable – to prevent loss, duplication, and general inefficiency.
  • No credential sharing to enable identity tracking, revocation, and rotation.
  • API-accessibility – so they can be distributed into the infrastructure via scripts and programs.
  • Separation of duties – so that one group can update the passwords, and another group can use them.

What can I do about it?

You can either build out your own modern password management system or you can outsource it to an expert – just like you’ve outsourced other important supporting functions of your business to the cloud or Internet services.

My suggestion: Go to, get the API and run it on your own AWS instance.

There’s NO excuse for not plugging this hole right now. Just do it.

Adapted from a blog post by Andy Palmer of KOA Labs.





Keep up-to-date on security best practices, events and webinars.

Share This