EP 46 – Behind the Data Breach: Dissecting Cozy Bear’s Microsoft Attack

[00:00:00.280] – David Puner
You’re listening to the Trust Issues podcast. I’m David Puner, a Senior Editorial Manager at CyberArk, the global leader in identity security. Hello, and welcome to another episode of Trust Issues. In recent weeks, a significant breach involving Microsoft has captured our attention, showcasing the stealthy tactics employed by threat actors.

[00:00:38.660] – David Puner
Today’s guest is Andy Thompson, who’s CyberArk Labs’ Offensive Security Research Evangelist. Andy last joined us just a few months ago to break down another couple of high-profile breaches on MGM and Okta, respectively. Big names, big breaches, no organization is impenetrable. This time, Andy sheds light on the intricacies of this attack on Microsoft, and he highlights lessons organizations can learn from the incident. He also unravels the timeline of events, examining how the attacker, the notorious Cozy Bear threat actor, gained access to Microsoft systems through a legacy non-production test tenant account.

[00:01:22.060] – David Puner
We’ll explore the nuances of password spraying, a method that proved effective in this case by avoiding standard rate limits and detection mechanisms. As we explore the anatomy of the attack and the attack flow, Andy provides insights into the assumptions of misconfigurations that facilitated this breach and what organizations can do to mitigate similar risks. From protecting non-production environments to the importance of identity threat detection and response, ITDR, we’ll uncover practical steps organizations can take to bolster their cybersecurity posture.

[00:01:59.480] – David Puner
In a landscape where cyber threats continue to evolve, understanding these tactics is crucial for staying ahead of the curve. Here’s my conversation with Andy Thompson. Andy Thompson, CyberArk Labs’ Offensive Security Research Evangelist. Welcome back to the podcast, Andy.

[00:02:23.580] – Andy Thompson
Thank you, David. I really appreciate the invitation back. It seems that we keep coming back here with a data breach after data breach.

[00:02:31.980] – David Puner
Yeah. The last we spoke was back in November, and I did not expect to see you back so soon, but as is the nature of these breaches, I guess it’s not entirely surprising, unfortunately. But it’s so good to see you.

[00:02:47.620] – Andy Thompson
You as well.

[00:02:49.470] – David Puner
Anyway, today we’re going to talk about a recent attack. Or is it a recent breach, or are they one and the same? It happened in January to Microsoft. You and the CyberArk Labs team have done a great initial analysis of it, which our listeners will be able to find on the CyberArk blog by the time this episode comes out. We will link to it in the show summary. To start things off briefly, and then we’ll go into more depth as we go, what happened? Who did it? What do we know, and what don’t we know at this point? I guess let’s tackle that briefly. I also threw in there the difference between a breach and an attack. I don’t know what you want to touch upon first.

[00:03:35.490] – Andy Thompson
I’ll go through the whole process here, go from soup to nuts here.

[00:03:39.540] – David Puner
All right. Okay.

[00:03:41.490] – Andy Thompson
What we know so far is that Microsoft recently announced that they had been compromised and data had been exfiltrated by a nation-state threat actor, APT29, Iron Hemlock, Dark Halo, whatever you want to call them. Microsoft calls them Midnight Blizzard. This threat actor is tracked back to the Russian intelligence agency, the SVR, I think.

[00:04:10.450] – David Puner
Yeah, that sounds right.

[00:04:11.910] – Andy Thompson
Anyway, it appears that this nation-state actor, who’s incredibly well known for their covert actions, had somehow compromised a legacy or test environment that Microsoft was using, or wasn’t using, actually. Through some very simple attacks, they were able to get initial access and then take advantage of some over-permissioned OAuth applications. This allowed them to create new applications, and ultimately leveraging these privileges, were able to read the inboxes and attachments of some very, very highly privileged users.

[00:04:54.740] – David Puner
Executives within Microsoft.

[00:04:56.820] – Andy Thompson
Absolutely. Executives, legal, the IT, the threat hunting teams. All of this information was subsequently exposed and exfiltrated by Russian state actors. I mean, this is some really scary stuff. I don’t think the public really grasps the severity of this issue. This is legitimately a concern.

[00:05:21.230] – David Puner
We’ll get more into the details of the attack itself and the different steps in a few minutes. But I guess from what you’ve already said, there are a couple of things I want to ask. One is, what did… You threw out a lot of different names there. APT29, Cozy Bear, Midnight Blizzard, Midnight Oil. I don’t know. Why do they have so many different names, and what are they, it, whomever it may be? It’s a group, right? What did they exfiltrate?

[00:05:48.600] – Andy Thompson
This group, named by the federal government, goes by APT29. But like I said, they go by a ton of different names.

[00:05:56.640] – David Puner
The US federal government?

[00:05:58.650] – Andy Thompson
Yes, the US federal government. The reason they go by all these different names is because many different organizations are tracking them, and they have their own naming conventions and whatnot. Again, whoever is the one that’s facilitating their threat research and threat hunting calls them by different names. But I think the most famous moniker that they go by is Cozy Bear. Not to be confused with Fancy Bear. That’s the military arm of the Russian government. But this group is the intelligence agency. They are the ones that are imbued with the power to collect information for the best interest of the Russian government, whether that be proprietary data, government information, all sorts of information that provide the Russian government with information that serves their best interest.

[00:06:53.750] – David Puner
So this isn’t just some lone wolf in their basement with the hoodie, as we always see in those stock images?

[00:06:59.780] – Andy Thompson
No, these are not the standard script kitties that we’ve heard so often about in previous data breaches. But no, this one is a government-sponsored espionage group. They have been known to do some pretty incredible hacks. I mean, we’ve seen previous situations with the Democratic National Committee of the United States. There is the Republican National Committee, even SolarWinds. SolarWinds was actually the supply chain attack that happened back in 2019, 2020. I just remember that was the terrible, terrible New Year’s.

[00:07:37.350] – David Puner
I remember that, too. Must have been 2020.

[00:07:40.230] – Andy Thompson
Exactly. We learned to dread in the IT industry the last week of December because something inevitably is going to go blow up. But this is the organization that perpetuated that supply chain attack. Their motives, their TTPs, tools, or tactics, tools, procedures, whatever, they are above and beyond the standard threat actor that we see.

[00:08:04.290] – David Puner
I’ll go back to the actual attack in a moment, but to just dig into the name just a little bit more. Is there any reason for all these bears in the name?

[00:08:13.970] – Andy Thompson
I have my own speculation. The bear is a quintessential thing with Russia. I think the theme of the bears, Cozy, Fancy Bear, I think it relates to the Russian state animal, I think.

[00:08:28.630] – David Puner
Then going back to the attack itself or the breach. Actually, that’s something we haven’t discussed yet. Are attack and breach one and the same when you see this terminology here, there, and everywhere?

[00:08:42.930] – Andy Thompson
No. Actually, it’s not because an attack is more aptly named an incident. Somebody is actually trying to break in or acquire data or something, and that’s an incident. An incident doesn’t necessarily result in a data breach, which a data breach is sensitive information leaking out beyond the constraints of an IT organization. Yes, this was an attack, this was an incident, and this was a breach.

[00:09:13.220] – David Puner
The breach happens, and what was exfiltrated, and how do we know any of this?

[00:09:22.190] – Andy Thompson
What we’ve been able to determine based on the public statements by Microsoft is that certain information was disclosed, specifically regarding Cozy Bear. This appears that they were doing reconnaissance on themselves, seeing what the threat hunters actually knew about themselves. They were looking around to see what the bad guys, which actually, I guess, is the good guys in this circumstance. They wanted to see what the good guys knew about themselves. There’s also some additional information that we don’t know. We do know that they were able to access the mailbox of high-level executives, legal, and other departments. But that particular information wasn’t disclosed. There’s so many unknowns here.

[00:10:10.440] – David Puner
Was it potentially happenstance that they were able to get to the executives and the legal? Because if they were seeking information on themselves, why would they be going to Microsoft executives or Microsoft legal?

[00:10:24.520] – Andy Thompson
I think once they got that initial foothold, realized that they were able to do reconnaissance on themselves, they had unfettered access into the mail infrastructure. Why not? Why not go after those top executives and see what information? This is figuratively icing on the cake from a threat actor’s perspective. Why not fulfill your mission? But if you can get additional incredibly high-value information from senior executives, why wouldn’t you?

[00:10:56.180] – David Puner
Then back to the attack itself. What are the key events of the recent attack on Microsoft, starting with the initial breach?

[00:11:06.510] – Andy Thompson
The initial breach was instantiated through a very simple attack method called the password spring. This is done by slowly trying to authenticate over an extended period of time through different IP addresses and things of that matter. This is incredibly relevant because there has been a recent botnet that has been discovered by another nation state, and it’s using these bot nets to really, we call it, go low and slow to authenticate over time. The reason you do this is it goes under the radar of many SIEM solutions because they usually detect authentications in rapid succession, and the account gets locked out. None of this happens in a password spray.

[00:11:53.120] – David Puner
This password spray, I think we can get a pretty good visual of what it is, but maybe if you could just define what a password spray is.

[00:12:03.290] – Andy Thompson
I’m trying to think of the official way to explain it, but it is a slow, prolonged, distributed attack on an authentication system that is done in a way that prevents technical controls from blocking it. Preventing the account lockout, preventing even detection from it. If it tries to authenticate and log on one time, maybe an hour later, another time, maybe two hours the next time. It doesn’t do it consistently. It’s not going to do A-A-A-A, A-A-A-B. It’s going to come from a random array of usernames, random array of passwords. Again, it’s randomly spraying credentials up against the wall and seeing what sticks. That is a password spray.

[00:12:51.510] – David Puner
When it’s random, is it just 100% random, or is it potentially cleaning data from generative AI or something like that?

[00:13:02.180] – Andy Thompson
Oh, yeah. Great question, David. The answer is the quintessential IT answer. It depends. There’s different ways to facilitate a brute force attack, which is essentially what this is. You could go through a password list, maybe gleaning information from previous data breaches, which is very, very possible. You could do a brute force like you do A-A-A-A, A-A-A-B, but you could also, just with reconnaissance information that you gather, create custom word lists. This is where generative AI could actually really provide value in helping create dynamically created word lists based on all sorts of information that it automatically gathers. We don’t know exactly how the password spray occurred and from what data it was using. But what we do know was it was able to authenticate into a legacy non-production system. That’s a real problem.

[00:14:00.940] – David Puner
What is a legacy non-production system?

[00:14:04.720] – Andy Thompson
When organizations, data, IT departments are working, they don’t just have the system that everybody’s working on. That’s called production. Organizations will do testing, they’ll do development, they’ll do all sorts of things in other environments that are very similar to the production environment. What happens is often mistakes are made. You’ll see sometimes, and not in this circumstance, but production data, PII, might be used in these QA and dev environments. In the event that a QA or dev environment is compromised, then you have literal PII that’s being leaked.

[00:14:42.650] – David Puner
Personal identifiable information, right?

[00:14:45.670] – Andy Thompson
Yeah, personal identifiable information. You could have health information, credit card information. In previous jobs I used to work at, we actually had made these sorts of mistakes. We identified them and corrected them, but we previously had personal identifiable information in development environments because they used that data for testing. Now, what’s interesting to note here with the Microsoft event is that they weren’t using production data, but they were using a production password. The same password that was being used to log into this legacy non-production environment, in my opinion, was very much so a production credential because it allowed them to ultimately pivot into the production mail servers.

[00:15:37.280] – David Puner
Is that why there wasn’t a multifactor authentication part of the flow?

[00:15:42.500] – Andy Thompson
Quite possibly. Again, what we often see in these non-production environments is that not all the security controls are the same. Don’t get me wrong, they should be, but just for ease of access, there are reasons, operational reasons, ultimately, that certain things, like multifactor, very well may have been neglected in this circumstance. That was a real issue with this particular breach, because even if the password spray had been successful, the multifactor should have been able to prevent the authorization and the access.

[00:16:21.840] – David Puner
In your blog post, there’s a graphic that outlines APT29 or Cozy Bear’s steps in the Microsoft attack. How did each step contribute to the success of the operation? Maybe it’d be best to just briefly go through the steps.

[00:16:40.670] – Andy Thompson
Okay, let’s start with the initial step one. It really boiled down to reconnaissance first, identifying what information to target, the user, that sort of stuff. It could be done through several ways, using something to scan the IP range to find which applications are publicly accessible. You could also do something like using social media to find potential users that could be attacked. This is similar to what we saw with the MGM breach, where the threat actors use social media to zero in on their targets. Once they identified the credentials that they wanted to password spray and the targets, that legacy non-production application, that’s when they facilitated that password spray and subsequently got in. After they got in, they were able to bypass the multifactor, and that’s really where they started abusing OAuth.

[00:17:50.080] – David Puner
Maybe this is the right time to ask you this, maybe it isn’t. You can tell me. What’s Golden SAML, and what role did it play in this attack?

[00:17:59.430] – Andy Thompson
Golden SAML is a method. It’s a tactic that threat actors have used in the past to bypass SAML authentication and pivot from the cloud into on-prem data centers and vice versa. This was a tactic that was used by Cozy Bear in the SolarWinds attack back in, like I said, 2019, 2020. However, this was a tactic discovered by Shaked Reiner, a researcher here at CyberArk Labs, all the way back in 2017. What’s notable about Golden SAML is that although it wasn’t used in this particular Microsoft hack, that we at CyberArk Labs identified this and that Cozy Bear was the very first threat actor that was discovered using this tactic. That’s what’s particularly notable about CyberArk Labs and our relationship to Cozy Bear. It was specifically the abuse of the Golden SAML tactic.

[00:19:01.230] – David Puner
How many of the details that we’re talking about in this conversation are speculative and how much of it is stuff that we know? Then of what we know, how much of that is from Microsoft and how much of that is from you and the CyberArk Labs team?

[00:19:18.280] – Andy Thompson
Most of the information, if not all, actually it is, it was provided to the public by Microsoft’s announcement explaining the circumstance, explaining where the misconfigurations were. There is very little speculation there. However, the things that we had to assume were why there wasn’t multifactor, why there was an overprovisionment of a particular role. As much as Microsoft did a fantastic job, and don’t get me wrong, they actually did a really great job of disclosing a ton of information in a timely manner, no less.

[00:20:00.640] – David Puner
It’s an important point.

[00:20:02.050] – Andy Thompson
Yeah, they did it in a lot of information and in a timely manner, which, don’t get me wrong, not everybody does. I feel that there wasn’t a whole lot of assumptions made. I mean, outside of just, again, a couple of misconfigurations had to be assumed, the data really does speak for itself.

[00:20:20.330] – David Puner
Ramifications for Microsoft, aside from reputational, do we even know what the full extent of the ramifications are or could be?

[00:20:33.290] – Andy Thompson
No, not at all. I mean, we do know that this threat group was looking for information about themselves as far as TTPs, indicators of compromise, and whatnot. That’s great. But we don’t know what information was disclosed relative to proprietary information, communication with senior executives. That sort of stuff was not disclosed in this public announcement. This is called proprietary information. I don’t expect that they will disclose that. As far as I’m concerned, we’ve been given about as much information as they’re going to share with us. That’s why we have the knowledge of the threat actors who were doing reconnaissance on themselves. But beyond that, I don’t think we’re going to get any more information.

[00:21:18.380] – David Puner
Based on what you’ve told me about Cozy Bear, it sounds like they are a professional organization. Do I have that right?

[00:21:27.720] – Andy Thompson
Yeah, this is right. They are absolutely a professional organization. I mean, they have payroll, they have insurance just like we do. They even have an office that they clock into. Threat hunters have actually been able to determine by analyzing code, who is the one that’s behind the keyboard writing the code. They can tell times in which the attacks are going on, typically what time they clock in, when they might go out for lunch. They even take vacations just like you and I do.

[00:21:57.150] – David Puner
Is anybody trying to stop them, or is it more trying to stop what they’re trying to do to others as opposed to actually trying to stop them?

[00:22:06.030] – Andy Thompson
I think everybody’s trying to stop them, from nation states to commercial cybersecurity companies. Even we at CyberArk Labs are actively trying to bring awareness to these types of attacks and prevent them from happening. You do have to understand that nation-state actors are vastly different from the other types of threat actors that we see in the wild. If they’re going to get in, they’re going to really try harder, exponentially harder than you’d typically find.

[00:22:35.280] – David Puner
But there are things that organizations can do to address similar misconfigurations to what Microsoft had going on in this particular case to enhance their cybersecurity posture, I think. I guess that’s more of a question for you.

[00:22:51.670] – Andy Thompson
There are a lot of things that can be done from a best practices perspective that could have provided a barrier to prevent this threat actor. I mean, simple stuff like multifactor, best practices like keeping your prod dev, QA environments off the Internet, making sure that they’re not easily accessible to threat actors. These are just standard best practices that should and ought to be followed.

[00:23:18.520] – Andy Thompson
The other things that could be done would have been from more of an incident threat detection and response, ITDR. This is something where we should be actively be looking for certain indicators of compromise. Logins that don’t have multifactor, whether an OAuth application is idle, making sure that you’re not overprovisioning certain OAuth roles or just roles in general, making sure that you’re monitoring for particularly malicious commands, or even just commands that your standard users aren’t going to be running.

[00:23:52.780] – Andy Thompson
This is a really simple one, but how many people in accounting will be running the IP config command, or how many people in legal are going to be running whoami.exe? These things do log events and can be tracked, and this is a way to detect a threat actor in your organization. Because what really separates Cozy Bear APT29 from all the other threat actors is their willingness to sit and wait. Their dwell time is beyond any other threat actor. They’re just biding their time, slowly waiting for the time to strike. Even if they’re being as slow and sly as possible, they will still create events that could be detected and responded to.

[00:24:36.960] – David Puner
You mentioned some of your suggested ITDR detections and responses, and for the listeners that are interested in checking those out, we’d also direct them to the blog, again, on the CyberArk blog about this particular attack/breach. How can organizations effectively integrate these ITDR recommendations into their cybersecurity strategies?

[00:24:58.660] – Andy Thompson
Well, it all boils down to having a plan, understanding what tools you currently have at your disposal, learning how to integrate those tools, and really having a game plan. Doing tabletop exercises and really understanding the types of attacks that are happening in the wild today and trying to position how you can respond to those attacks.

[00:25:21.060] – Andy Thompson
I think it’s something that CyberArk and CyberArk Labs absolutely can help with because no two organizations are the same. No two organizations have the same priorities, different crown jewels, as they say. It’s not a single answer. It really depends. It’s that quintessential IT thing. But the point here is that there are best practices that should be followed. There are things that we should be monitoring and responding to. If we don’t do that, we’re just opening the front door and the back to these nation-state actors, all the way to the financially motivated criminal groups, and even to the script kitties. Just follow the best practices, try to detect and respond, and I think you’re going to do a pretty good job protecting an organization.

[00:26:07.850] – David Puner
Folks can dig not only into your recommendations, ITDR recommendations on the CyberArk blog, but we’ve also got numerous other posts and material on ITDR there, so it’s a good resource to check out. What do you think the essential takeaways from this attack should be for the global cybersecurity community?

[00:26:26.320] – Andy Thompson
I think that a big takeaway is that no organization is perfect. We’re all humans, and we all make mistakes, and best practices aren’t always going to be followed. But 99% of all the data breaches that we see in the wild today aren’t caused by vulnerabilities, but they’re caused by misconfigurations. For that reason, we can’t rely on one particular product to prevent it. You’ve heard the people say, “Oh, there’s no blinkie box, there’s no silver bullet.” But there almost is with defense in depth.

[00:27:06.430] – David Puner
How can organizations enhance their cyber resilience in the face of persistent and sophisticated attacks?

[00:27:13.240] – Andy Thompson
You can’t necessarily rely on one product to save the day. Maybe you can’t rely on two. But if you slap enough controls on top of yourself-

[00:27:21.880] – David Puner
Layers.

[00:27:23.350] – Andy Thompson
-you’re going to have that impermeable barrier. It’s like Swiss cheese. You have one slice of Swiss cheese, it’s got holes, but you slap more and more on. You might have holes in the different slices, but they’re in different positions, different spots, and different sizes. At the end of the day, if you put enough controls on top of a vulnerability or a misconfiguration or whatnot, with enough defense in depth, that’s really going to adequately protect an organization.

[00:27:51.330] – David Puner
There is some hope, of course.

[00:27:55.720] – Andy Thompson
Again, it’s all about defense in depth, really understanding what your priorities are, what is the most relevant thing to your organization, and wrapping layer of controls around that in order to prevent these nation states, these even financially motivated threat actors. Just preventing threat actors in general is really the key. It all depends on what those crown jewels are, will depend on what controls you put on. But the keyword is controls, plural. It’s not about one single thing that’s going to save your day. It’s not going to be EDR, it’s not going to be a firewall. It’s going to be a combination of all these things.

[00:28:35.490] – Andy Thompson
I do think that what is really relevant here is that these threat actors are primarily going after identities, because identities, that’s really where the privilege is. This is where the data is. If you can abuse a legitimate credential, whether it’s provisioned correctly or not, this is what the threat actors are really, especially these espionage groups, are going after. They’re really looking for this data that can be accessed through identities, whether they be machine identities, whether they be human being identities, whether they’re SaaS applications outside the constraints of your traditional IT world. The key here is that I’m concerned that threat actors are getting into organizations by compromising identities.

[00:29:26.000] – David Puner
There you go. I was going to ask you how identity figures in all this.

[00:29:30.930] – Andy Thompson
Oh, absolutely. I mean, again, it all boils down to identity. Like I was saying earlier, an identity can be anything from a standard user in HR to a doctor in an operating room, all the way to that IT admin that’s running the domain controllers, or in this circumstance, the mail servers. All of them, if you think about it, have some level of privileged access. A doctor may have access to medical records, and that’s relevant, right? We have people in HR that have employment records and things like that.

[00:30:08.890] – Andy Thompson
The thing is the standard previous conception of privileged access used to just be with IT. Threat actors know that’s not the case anymore. This is why they’re targeting senior executives. This is why they’re looking at the legal teams, because it’s not just IT anymore. They’re abusing the privileges of people outside the IT organization to facilitate their espionage and reconnaissance.

[00:30:38.440] – David Puner
Andy, I’ve not yet said this at the end of a podcast interview yet, but here I’m going to do it for the first time. Let’s not do this again soon.

[00:30:48.690] – Andy Thompson
No, let’s not do this again soon. But unfortunately, I feel like it’s inevitable. I don’t want this to happen again, David. Let’s talk about something fun next time. Let’s talk about something cool that’s not going to be about an epic data breach. But you know what? I’m sure I’ll see you soon.

[00:31:05.800] – David Puner
Andy, thanks so much for coming back onto the podcast again.

[00:31:09.500] – Andy Thompson
Thanks again, David. It’s a pleasure and a privilege. Thank you.

[00:31:21.300] – David Puner
Thanks for listening to Trust Issues. If you like this episode, please check out our back catalog for more conversations with cyber defenders and protectors. Don’t miss new episodes. Make sure you’re following us wherever you get your podcast. Let’s see. Oh, yeah. Drop us a line if you feel so inclined. Questions, comments, suggestions, which, come to think of it, are comments. Our email address is [email protected]. See you next time.