Codecov Breach Learning: Engage Developers to Protect the DevOps Pipeline

April 22, 2021 Chris Smith

Codecov Breach Secure DevOps Pipeline

Regardless of what industry you’re in, software is a driving force behind digital innovation. But what happens when the software your organization builds and uses to innovate isn’t secure? It’s a question that many organizations and government agencies are grappling with as they examine their development environments and digital supply chains post-SolarWinds — and with even more urgency since the recent Codecov breach.

This month, Reuters reported that attackers infiltrated code coverage reporting solution provider Codecov and by manipulating one of its software development tools gained access to hundreds of organizations’ networks. This breach, with the potential for far-reaching and yet-unknown impact, reinforces the fact that development environments, including continuous integration/continuous delivery (CI/CD) pipelines, are vulnerable to targeted attacks. To minimize risk, it is imperative that security teams “shift left” and become fully embedded into the software development process — from the very first planning discussion to coding and all the way through production.

Who’s Keeping Your Secrets Safe?

ZDNet reports that DevOps adoption has almost doubled in the last five years and since COVID-19.

As DevOps adoption accelerates, developers and IT teams are relying heavily on automation and cloud services to code faster and accelerate software releases. While critical, these advancements increase the attack surface by introducing new credentials and secrets that are exceptionally powerful and highly susceptible to compromise, including those used:

  • On developer workstations and laptops to write code, build, test, provision, run and operate applications
  • By applications, scripts, and other non-human identities to communicate with other applications and tools, as well as to securely access databases and other sensitive resources
  • By human users to access DevOps tool administrative consoles, as well as cloud consoles

Unfortunately, too often organizations don’t fully recognize the extent to which these secrets are being used in their DevOps environments or how effectively these secrets are being secured across code repositories and the entire software development lifecycle (SDLC). Why is that?

The Great DevOps Security Divide

In some organizations, this is because “ownership” of secrets management — from requirements to platform selection and operations — isn’t clearly defined. Instead, secrets management decisions are happening at multiple levels: 46% companywide, 30% at the business/division level, and 24% at the project team level, according to a CyberArk survey. This lack of clarity and project-level vs. holistic approach to security leaves gaps that lead to problems as DevOps adoption increases and applications move to production.

Additionally, developers are inadvertently — or deliberately — taking on more direct security responsibilities, either because security is not aware or not yet involved with the project. In this case, developers are often making the secrets management decisions early in the development process without fully understanding the risks and requirements — and without involving the security team until it’s too late, often when code is already in production.

Fundamentally, development and security teams have very different priorities. Security is charged with reducing risk and defending the organization from attacks, while development is focused on delivering software innovations on tight release schedules. Without security’s involvement, development teams tend to make decisions that prioritize speed over security. This could mean taking risky shortcuts such as relying on the DevOps tool’s native security capabilities, using code repositories without securing credentials, or worse — hard-coding privileged credentials directly into code — or simply ignoring security altogether.

If not secured, the credentials and secrets used across the DevOps pipeline become easy — and valuable — targets for attackers. CI/CD pipeline and other DevOps tools are Tier Zero assets; access them and you get more credentials. With compromised credentials, attackers can take confidential data, inject malware into the codebase, or steal code and valuable IP from repositories.

What’s more, when secrets management isn’t addressed throughout the process, security has to jump in to confront issues at the eleventh hour — sometime right before code is scheduled to go into production — putting deployments on hold until security issues are resolved. Developers feel the pain with last-minute code changes, which is costly and frustrating for everyone involved.

Shift Left to Embed Security from the Start

To reduce risk and accelerate innovation, organizations embed security into development and operations practices from inception to deployment. This concept, commonly known as “shifting left,” requires security teams to engage with developers from the very beginning of the development process, breaking down silos and working together to make critical security decisions. This means developers can avoid rework and delays later in the lifecycle. And by making it easy for developers to secure their application credentials and secrets, security teams can reduce objections, increase adoption and accelerate development cycles.

While many organizations recognize the value of embedded security, moving from idea to action can be challenging. Today, only 41% of security and DevOps teams report that security is integrated throughout the application development process, according to our research. There are numerous reasons for this — especially when it comes to secrets management. For example, our research found 60% of developers believe security teams lack the necessary technical expertise.

How Can Security Shift Left and Better Engage Developers?

Now more than ever, it is critical that security and development teams come into alignment to protect their DevOps environment and digital supply chain. Yet while security teams know they need to engage with developers, changing mindsets and cultural practices take time, and getting started can be half the battle.

Our new guide, “Six Practical Approaches to Engage Developers and Improve DevOps Security,” provides enterprise security teams with pragmatic approaches for creating a security-first culture, strengthening bonds, and engaging with developers, including:

  • Introducing self-service models that make securing application credentials easy for developers
  • Running cross-educational workshops on specific use cases and digging into developer concerns and how to address them
  • Encouraging developers to think like attackers by providing training on common attack vectors and hands-on demonstrations
  • Embracing Agile and DevOps methods within security processes
  • Demonstrating “quick wins” to make an immediate impact, quickly show value, and establish credibility

For broader guidance on securing development environments and the software supply chain, read “Securing Privileged Access in Development Environments – Three Critical Use Cases.”

Previous Article
Breaking Down the Codecov Attack: Finding a Malicious Needle in a Code Haystack
Breaking Down the Codecov Attack: Finding a Malicious Needle in a Code Haystack

Earlier this month, San Francisco-based technology company Codecov discovered that attackers had compromise...

Next Article
Between a Rock and a Hard Place: The IT Help Desk Manager’s Password Dilemma
Between a Rock and a Hard Place: The IT Help Desk Manager’s Password Dilemma

Long-time Saturday Night Live fans will likely remember Nick Burns – a.k.a. “your company’s computer guy” –...