EP 23 – Red teaming AI governance: catching model risk early

David Puner:
You are listening to the Security Matters podcast. I’m David Puner, a senior editorial manager at CyberArk, the global leader in identity security.

An organization deploys an AI agent to help customers faster. It’s trained. It works. It starts making decisions. But here’s the problem. The agent starts stepping outside its lane without warning. The agent begins making promises. No policy ever approved discounts. It shouldn’t give benefits. It can’t authorize wildly off-base perks like airline tickets.

No breach triggered this. No adversary pushed it. The danger came from inside the model, from the gap between how it was trained and how it behaves in the real world. When no one ever truly tested it, nobody notices. The model keeps operating. The promises keep stacking up, and the real question is, how far does the damage spread before anyone realizes what the agent’s been doing?

After a customer complains? When the books don’t balance? Or only once the damage is done?

That tension between authority, identity, and limits is at the center of today’s conversation with Pamela K. Isom, former director of AI and technology at the US Department of Energy and current founder and CEO of Isom Advice and Consulting.

And those tensions don’t stay theoretical for long. They show up the moment a system behaves unexpectedly. The only way to catch that early is to deliberately push the system past its comfort zone. As Pamela explains, red teaming reaches far beyond cybersecurity. She says it’s about stress testing the governance around AI systems, ensuring models don’t behave as if they have identities, roles, or authority they were never meant to assume, and making sure there are clear escalation paths in place before something breaks.

Let’s get into it with Pamela Isom.

Pamela K. Isom, founder and CEO of IsAdvice & Consulting, and former director of artificial intelligence and technology at the US Department of Energy. Thank you for coming onto the podcast. Welcome.

Pamela Isom:
Thank you for having me. I’m honored to be here.

David Puner:
Let’s get right into it. From your early work in the private sector to shaping AI strategy at the Department of Energy and pioneering AI governance initiatives that incorporate adversarial testing, and now as the founder and CEO of your own AI governance consulting firm, what are some milestones that have shaped your approach to AI, cybersecurity, and innovation?

Pamela Isom:
Okay. I am going to try to unpack that some.

One of the earliest milestones in my career, as a young programmer in the field, I learned that I’m a risk taker. And I learned that I like to debug systems. Working for municipalities, I liked to debug systems and find out where those gaps are. I really enjoy taking those gaps and figuring out where the issues are and solving those problems with clients, working with municipalities and digging into operational issues.

I got an opportunity then to move to a larger organization, and that’s where I first learned AI. It was expert systems, and it was called OPS5. Not many people were using the tools or this type of software. And the reason why this is important is because it was my first foray into intelligent systems.

So then I took those skills and moved them with me as I went to work for large companies, IBM and a couple others. I can count the companies that I worked for. I was going from being a risk taker and identifying issues and potential risks and turning those risks into opportunities to help clients excel, to leadership and governance when I went into IBM.

That’s when I really started to dig into the importance of trust, the importance of leadership, and the importance of governance. And of course, I was at IBM when Watson first came around. This was a milestone for me because I spent a lot of time turning around troubled projects, understanding the lifecycle, understanding governance, trying to figure out where communication failures were, and turning those troubled projects around.

Again, when I would see gaps, I would turn those gaps into opportunities and innovations. That’s where I got patents and things like that from doing that type of work. I’m an engineer, and then I transitioned from being an engineer into being a solution architect because I like to look at the lifecycle end to end.

I can recall conversations with people saying to me, “You need to stay in your swim lane.” And I was like, “I don’t have a swim lane.” I like looking at things from end to end. But being an engineer first and then an architect, you get to look at things from a bigger picture, even though they still want you to stay in your lane.

I was never that type. I was fortunate enough to look for gaps, look for threats, threat surfaces, and decision failures, and build up client satisfaction and client excellence. That led to executive leadership when I went into government.

Now I’m focused on advancing AI security and innovation in the government. I started with the Patent and Trademark Office, brought that into the Department of Energy, where I headed up the AI and technology office and also led AI work and AI strategy for the CIO’s office before becoming director over AI and technology for the agency.

Back to your question, the milestones were an honor to go into government, but even more, it was an honor to take the skills I’d developed as a risk taker and convert those into DevSecOps approaches so the solutions and innovations agencies are developing are secure while also advancing AI.

Particularly at the Department of Energy, I was driving AI at the energy domain level, heading up use case development. I remember leading a wildfire task force. I was handpicked to lead that effort. What I was driving was how we use AI to mitigate fuel loads that are instigating, facilitating, and exacerbating wildfires.

It was a wonderful exercise working across agencies. I remember working with the Department of Defense. Those leadership roles are what I was able to bring. And a key takeaway for me was that it was an honor to go into government, and a milestone was being able to turn risks and challenges into opportunities, and then turn those opportunities into sustainable innovations.

David Puner:
Thank you for taking us on that ride. I really appreciate it. So, you were with the Department of Energy through September 2022. That strikes me as interesting because that’s right before this enormous wave of generative AI hit everyone. When you were looking at AI back in 2022, did you have any idea that it would be what it is now, or where it is now?

Pamela Isom:
So here’s the thing. Visionaries tend to see down the road. I’m a visionary. I’m a person who felt like AI was really going to take off. I felt that way back when I studied OPS5 earlier in my career. But then it died down.

While I was at Energy, I did feel like AI was really going to take off. I spent a lot of time with the national labs, and they were already utilizing AI to a great extent. I did realize that this was going to happen, and I felt like it would happen. What I didn’t expect was that generative AI would emerge the way it did, where there’s such a dependency today and such reliance on generative AI.

I thought it would be more strategically adopted. When generative AI really started to surface and take off, I felt like organizations started adopting it just to keep up with others. And I still see that today. They’re just grabbing it because that’s the thing to do, to keep up.

David Puner:
And a quick question before we move on to the main event here. Watson—how much was Watson a precursor to the LLMs that we know today?

Pamela Isom:
It’s definitely a precursor. Watson has been around for a while, but back then, when I was there, there was a lot of emphasis on cognitive intelligence and big data.

The emphasis was on big data and managing and manipulating data, which is what AI is all about. So I would say that Watson was one of the leaders and pioneers. Everything is full circle.

David Puner:
Yeah. Really interesting looking back on that now. Let’s talk about one of your focuses, which is red teaming. Red teaming is often seen as a cybersecurity exercise, but you’ve made the case that it’s something business leaders and practitioners should be looking to do. What does that look like in practice, particularly when it comes to AI?

Pamela Isom:
So we know that AI is an amplifier of data. That’s basically what it does. Whatever information it has, it’s going to bring that to light. It’s going to take the patterns, reveal the patterns, and we’re going to make decisions based on that, based on the recommendations it makes.

When it comes to red teaming, what I really emphasize is yes, we have emerging threats because of what AI introduces, like more data poisoning, faster iteration of information, misuse, misrepresentation of data, as well as good use cases.

So what I say is for red teaming, we want to red team the governance. I’ll use a customer example. I have a client that created an AI shopper agent. That agent is there to help understand what’s going wrong within the organization, because they have policies that officials pay attention to for a while, and then after 60 or 90 days, they revert back to old habits.

Let’s say in this example—without revealing too much—student registration and enrollment. With red teaming, they created this AI shopper agent that goes out and imitates being a student. The team doesn’t know about it, and they’re just observing how the staff responds.

Are they enrolling students who are eligible? Are they providing discounts when they shouldn’t? What type of communication are they having with students? Are there escalations that should occur?

That’s an example of red teaming. Red teams go out and test the process. They stress test the process. They’re not trying to find issues with people. They’re trying to ensure the process is working.

This is where cybersecurity red teaming is emerging into something broader. It’s really about leadership and decision quality, and we need more of this. You don’t have to use a shopper agent. You could test governance in other ways.

For example, this happens today. You have an AI agent making promises that are not within policy. How did it decide to promise discounts or commit the organization to airline tickets? It was trained, but it was not tested.

Red teaming stress tests how the model performs. And beyond that, if it does behave this way, is there governance in place, and does that governance work? How long does it take the organization to discover that it committed to providing an airline ticket to a customer?

Should they find out after the customer complains? Or after the ticket is submitted and the financials don’t balance?

David Puner:
So in a case like that, how are you coming up with all the different scenarios that you’re plugging into stress tests? Are the scenarios themselves generated via AI?

Pamela Isom:
So like the shopper agent, that’s AI helping with stress testing. Organizations should work together to decide what red team scenarios should be.

In the examples I used, it wasn’t a cybersecurity person doing it. It was representatives from across the organization. These scenarios need to be created holistically and be cross-representative.

Teams should come together—stakeholders at different levels—to decide how to stress test solutions and to think about the risks if they don’t. What’s the risk of this model if we don’t test it?

Stop leaving it only to cybersecurity teams. Some of this doesn’t sound like a cybersecurity issue, but it can become one. Organizations need to decide what tests make the most sense and where red teaming is most applicable. That’s where experts like me come in.

David Puner:
It’s interesting that earlier you mentioned being a risk taker, and now you’re focused on finding risk and preventing risk. How does that connect?

Pamela Isom:
Because you can be a risk taker. I’m a small business owner, and I definitely took a risk. I said, “I want to do my own thing. I want to step out here. I can do this.” You can be a risk taker and still be a champion for risk mitigation.

Risk mitigation means that you appreciate that clients are risk takers. You support that. You believe people should take risks. But you also have to be stewards of mitigating the adverse impacts of those risks. You have to understand the adverse implications, the consequences, and how to mitigate the impact so it doesn’t have adversarial outcomes for your organization.

If we don’t have risk takers, we go nowhere in society.

David Puner:
Really interesting. So then, back to red teaming. Where do you see the biggest missed opportunities for red teaming in enterprise environments, especially outside traditional security teams?

Pamela Isom:
I think aside from the fact that organizations tend to leave it to cybersecurity experts, they need to look at red teaming as something that can happen by anyone within an organization.

It should be embraced. And we don’t embrace it today. I think that’s one of the missteps. We should see red teaming as a way to protect the organization. It’s a form of assurance, and I don’t think we really see it that way.

I also think we confuse red teaming with auditing. Auditing is about compliance. You have documented procedures, and you’re checking whether you’re meeting those procedures. Red teaming is about understanding. It’s about mitigating risk. It’s about trying to find flaws before someone else finds them.

They’re complementary, but they’re two distinct things. There’s a lot of confusion between auditing and red teaming.

David Puner:
Let’s go back to AI for a moment and talk about AI ethics. Ensuring AI systems are fair, transparent, and accountable. You’ve served as a responsible AI official for the Department of Energy, and you’re a certified AI auditor. What does ethical AI look like in the real world beyond frameworks and principles?

Pamela Isom:
Think about every organization having ethical practices and policies already in place. Start there, and then think about how you ensure AI is not violating those moral policies.

For example, let’s say a model is trained to recognize emotional patterns, such as when someone is excited, anxious, or nervous. Maybe they make faster decisions when they’re nervous.

Now imagine an AI agent that helps support decision-making around a product. A client might express concern that if they don’t decide quickly, they’ll get left behind. The model then starts communicating urgency, saying things like, “We need to decide within the next 24 hours,” or even sooner.

There’s no rule that says it should do that. That’s unethical. It’s a different way of looking at ethics. You wouldn’t tell leaders in your organization to behave that way, but the model has picked up patterns. It has learned that decisions get made faster when it creates urgency.

So the model starts telling someone they need to decide today. That’s unethical behavior. It’s not governed. It shouldn’t do that. I would red team that. I would test how my models perform in situations like this.

David Puner:
Those are great examples. Is there an example where ethical concerns or bias surfaced in an AI project you were involved in, and how did you and your team respond?

Pamela Isom:
Yes. As a small business owner, I’m constantly monitoring systems. And when I was in government, many of the systems we worked with were AI systems.

When an ethical concern surfaced, I wasn’t quick to take it to legal. Not yet. I wanted to examine it first to determine whether it was actually an ethical issue or potentially historical data bias.

We know historical data isn’t always the most applicable data. Traditionally, we use historical data to help models make decisions. So we would look closely at the data sets, discuss them, and then decide what actions were needed to resolve the issue.

I’ll give another related example. Today, we’re working on federated data models. We federate data because not every energy client wants their data exposed to others, even though they trust us with it.

When we create dashboards, especially with aggregated data, we have to anonymize the data and understand the raw data source. Who provided it? We track data lineage as it moves through different tiers and ensure it’s handled properly.

When it reaches the end consumer, it’s not just a security issue. It’s also an ethical issue. We gave our word that we would protect and respect their data. They trusted us with it. If we don’t clean and manage that data properly before it’s aggregated and shared, we’ve violated that trust.

So yes, that’s how we think about ethics in practice.

David Puner:
Yes, it does. Thank you. When organizations think about AI governance, they often focus on the technology, but you’ve said the real vulnerabilities are often in the structures around the systems. What kinds of governance blind spots do you see most often?

Pamela Isom:
I see governance blind spots when people assume that if the AI says to do it, then do it. If the model says the information is correct, then the model said it, so let’s just do it. That’s the biggest one I see.

I also see procurement teams using systems without ensuring that vendor claims match reality. That’s a governance issue. There shouldn’t just be a checkpoint. There should be stress testing to ensure that checkpoint actually works.

I’ve been fortunate to do government contracting. The government trusts me with data. We have a contract, and we have insights. Am I going to just release that to models because I’m trying to figure something out? No.

I see enterprises not getting their data in order for AI consumption. They’re handing documents to models and then trusting them blindly. That’s not what these systems were intended for.

David Puner:
At the Department of Energy, you helped launch the AI Advancement Council and developed an AI risk management playbook. What were some of the most important lessons from that experience, especially for leaders trying to build responsible AI programs today?

Pamela Isom:
The biggest lesson was not to operate in silos. This issue has been around for years.

The challenge was figuring out how we can operate holistically while still respecting autonomy. That’s why I’m very focused on federated models. They respect autonomy but also allow you to operate holistically and pull information together.

I was very proud to pioneer the AI Advancement Council. It brought together stakeholders from across the department. Department heads were one layer, and program offices or operational teams were another layer, and they needed to come together.

They needed to come together vertically and horizontally. Why? Because teams were making AI investment decisions independently, asking Congress for funding for similar efforts, when there were opportunities to reuse and collaborate. That was the main intent, and I believe it was successful.

The playbook gave me an opportunity to take a risk and help organizations understand AI risks and how to mitigate them. Around the same time, the Department of Commerce was developing its risk management framework, and we coordinated with them.

DOE was able to pioneer an AI risk management playbook that included concrete examples. If you encounter this type of risk, here’s what it looks like. Here’s what you can do. Here are options to mitigate it. Again, being a risk taker while driving risk mitigation.

David Puner:
And we keep coming back to that. It really seems like an anchoring mantra for you.

Let’s move over to innovation, national security, and public-private collaboration. You’ve described innovation and national security as deeply connected. How is AI reshaping that relationship?

Pamela Isom:
It’s not simple. Cybersecurity needs AI capabilities to help pinpoint and mitigate risks associated with modern threats, but AI also introduces new risks.

We need AI to analyze information and understand patterns that humans can’t see with the naked eye. That’s how AI supports national security. It helps strengthen our security posture by enabling us to process big data and distinguish between what’s real and what’s fake.

AI isn’t just associated with fake data. It can also help identify invalid data points. It helps us understand what we’re dealing with and provides recommendations to mitigate risks.

David Puner:
What’s the current state of public-private collaboration when it comes to securing AI systems and critical infrastructure? Where are we getting it right, and where do we need to do better?

Pamela Isom:
We’re getting it right by leaning into data science, machine learning, and large language models. We’re integrating cybersecurity and looking for threats that are amplified by AI.

We’re also using AI and emerging technology to help make communities and cities smarter. But we’re still in an immature stage of AI adoption.

Programs like Genesis are promising. They’re just getting started, but they have a common mission. You see agencies and industry aligning around security, national security, grid resilience, and related goals.

We’re moving in the right direction, but we’re not as mature as we could be. We will get there.

We also need to move past the mindset of taking solutions at their word. We can’t do that. We need to get better at explainability, traceability, and data lineage, and managing that lineage. There will always be opportunities to improve in those areas.

David Puner:
And it sounds like at least some of those opportunities are human opportunities.

Pamela Isom:
Yes. If an agent is making recommendations, who’s checking them? Who ensures that a recommendation made last week is still applicable this week?

Where is the human? When does the model escalate to a human?

Models are often trained to try to solve everything themselves. We’ve all had experiences calling customer support and wanting to talk to a human. The system keeps trying to solve the problem on its own.

I recently had an experience with a moving company while relocating. The agent was AI. I was trying to describe my workout equipment, and the system didn’t understand. It generated a huge invoice because it thought all the parts were separate, even though they were one set.

It wouldn’t let me talk to a person. I had to call multiple times before reaching a human. They apologized and corrected the mistake, but I told them it shouldn’t have taken that long, and I shouldn’t have received that invoice.

That was because of how the model was trained. It wasn’t tested outside the happy path. That customer experience taught me again that you have to test beyond expected scenarios.

David Puner:
If you were advising a government agency or a Fortune 500 company today on how to work together more effectively on AI security, where would you start?

Pamela Isom:
If it’s AI security specifically, I would start by identifying threats that are amplified by AI, such as data poisoning and model injection.

Then I would focus on governance. I would start with those three areas: emerging threats, existing threats that are amplified by AI, and governance. Governance is the hidden one that we often overlook.

David Puner:
We’re always looking into our crystal ball to a certain degree, but if you’re looking into your crystal ball into 2026 and beyond, if you are thinking about emerging threats and you were sitting down—or you are sitting down—with a CISO or tech leader planning for 2026 and beyond, what’s the one piece of advice you’d want them to walk away with?

Pamela Isom:
I would say don’t over-delegate authority. The AI needs to understand where that authority is and where the delineations are, and make sure that there are clear limits.

Cybersecurity leaders can do that. They can work with the governance folks. I tell them to help the governance teams understand these are the things we should be watching out for.

And then I would say embrace AI. Use it to help with defenses. Use it from an offensive standpoint, which is where we get back to red teaming. Use it more.

David Puner:
And that is coming from Pamela Isom, who is not only a risk taker, but a risk mitigator.

Pam, before we go, you also host the podcast AI or Not. Where can our listeners find that? How often is it coming out, and what are your plans for 2026 with it?

Pamela Isom:
The podcast is about business leaders from around the globe. They get together and talk about how they’re using AI, whether they are going to use it or not.

It’s a podcast to explore the viability of AI to address mission challenges and leading practices. It airs every other Tuesday, and it’s on all podcast platforms.

David Puner:
Pamela, thank you so much for coming on to Security Matters. I really appreciate you taking the time, and I hope you have a happy New Year and a great 2026. I think they may be the same thing.

Pamela Isom:
Thank you for having me. This has been fun.

David Puner:
All right, there you have it. Thanks for listening to Security Matters. If you liked this episode, please follow us wherever you do your podcast thing so you can catch new episodes as they drop. And if you feel so inclined, please leave us a review. We’d appreciate it very much, and so will the algorithmic winds.

What else? Drop us a line with questions, comments, and if you’re a cybersecurity professional and you have an idea for an episode, drop us a line. Our email address is [email protected]

We hope to see you next time.