I genuinely believe that most AI governance programs begin with good intentions. But that doesn’t instantly equate to a successful program. Often, AI governance fails without the adversarial mindset.
These types of governance programs typically define acceptable use, establish review committees, classify risk, document models, assign owners, and publish principles around fairness, privacy, transparency, and human oversight.
Undoubtedly, those activities matter.
Those programs also tend to assume that people, systems, data, and models will operate within the boundaries the organization designed.
An adversary makes no such assumption. Moreover, the boundaries organizations have designed mean nothing to an adversary.
Attackers will generally search for paths of least resistance that lead to success. This includes ways to manipulate inputs, compromise identities, poison data, exploit integrations, misuse legitimate capabilities, and confuse and manipulate humans.
Employees, contractors, customers, partners, activists, fraudsters, competitors, and nation-state actors may all test the distance between what an AI system was intended to do and what it can be manipulated to do.
Governance that considers only intended behavior is policy.
Governance that anticipates intentional manipulation becomes resilience.
AI governance without an adversarial mindset documents how a system should behave. It does not prepare the organization for how the system can be made to behave.
The Adversary Has a Vote
Executives often discuss AI risk as though the organization controls all relevant variables.
Leaders choose the model. Engineers establish the architecture. Data teams manage information. Security implements controls. Legal writes policy. Users receive training.
Then the system enters the real world.
Customers provide unexpected inputs. Employees find shortcuts. Vendors go out of business. Models drift. Credentials become exposed. Attackers study architectures and systems. Data sources become contaminated.
Ultimately, the adversary gets a vote in how the system operates.
This principle has shaped cybersecurity for decades. A secure architecture cannot assume that users will follow instructions, data will remain trustworthy, or that controls will continuously operate exactly as designed.
AI governance must adopt a similar reality.
MITRE ATLAS (https://atlas.mitre.org/) documents tactics and techniques used against predictive, generative, and agentic AI systems. NIST has developed a taxonomy for adversarial machine learning (https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf). The NCSC’s secure AI guidance (https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development/guidelines/secure-design) emphasizes threat modeling across design, development, deployment, and operation.
Those movements create a message that is consistent: AI risk cannot be governed solely through compliance reviews and intended-use documentation.
AI Governance Must Cover More Than Model Failure
Executives commonly focus on whether an AI system will produce an incorrect answer.
Yet that is only one failure mode.
Interestingly, an AI system may produce an accurate answer for the wrong person. It may also follow a valid command issued through a compromised identity or expose sensitive information while correctly completing a task. Sound recommendations may be generated based on poisoned data.
The model may operate exactly as designed while the larger system fails.
This distinction matters because AI is not merely a model.
It is an architecture of identities, data, software, infrastructure, integrations, humans, workflows, and delegated authority.
An adversary does not need to attack the most sophisticated component. The adversary will attack the component that produces the greatest advantage or outcome for the least effort.
That may very well be the model itself.
But, it may also be an exposed API key, overprivileged service account, manipulated document, compromised developer, insecure plugin, careless employee, or trusted third-party data source.
An Adversarial Mindset Is Not Merely Pessimism
Some leaders resist adversarial thinking because it can sound negative or obstructive. Others claim that doing so empowers and/or validates the adversary.
Those notions misunderstand the purpose of the adversarial mindset.
An adversarial mindset does not assume that every person is malicious or that every AI initiative will fail. It assumes that valuable systems will attract manipulation and that unintended behavior becomes more likely as complexity, scope, and authority increase.
It asks the organization to examine its assumptions before an attacker does.
This mindset challenges statements and/or beliefs such as:
- Only employees can access the system.
- The model does not have access to sensitive data.
- A human reviews every important decision.
- The agent can only use approved tools.
- The training data comes from trusted sources.
- We have a kill switch, and it works.
Each statement may be technically accurate while concealing dangerous assumptions. As such, further probing may look like:
- Which employees?
- Through which identities?
- Does the human reviewer understand enough to challenge the model or system?
- Can approved tools be combined to create an unapproved outcome?
- Who determines whether a source remains trustworthy?
- How long does shutdown actually take?
The adversarial mindset turns reassuring claims into testable questions.
Seven Questions Leaders Should Ask
1. How Could Someone Intentionally Misuse This System?
Governance reviews often begin with the approved business use case.
Adversarial governance begins with the abuse case.
Leaders should scrutinize the angles. Ask how an employee, customer, contractor, criminal, or competitor could use some legitimate capability for a different objective.
A customer-service assistant might help employees retrieve account information. Could it also help a malicious insider assemble customer profiles?
A fraud model might identify suspicious payments. Could someone probe its thresholds and learn how to avoid detection?
A security agent might isolate compromised systems. Could an attacker manipulate it into disrupting legitimate operations?
Security teams should create abuse stories alongside legitimate user stories. Every material use case should identify who may benefit from subverting it and which capabilities they would seek out in pursuit of that benefit.
2. What Happens When the Data Becomes Hostile?
Organizations tend to view data as an input.
Adversaries may use it as code, instructions, a weapon, or a persistence mechanism.
Manipulated training data can alter future model behavior. Poisoned reference material can corrupt retrieval-augmented systems. Malicious instructions embedded in emails, websites, documents, or images can influence agents that consume external content.
The source may appear trusted while the content is not.
Leaders should ask:
- Which data sources can influence the system?
- Who can change those sources?
- How does the organization establish provenance?
- Can the system separate data from instructions?
- What happens when sources conflict?
- How quickly can poisoned information be detected and removed?
Data governance must assess not only quality and privacy but also hostility.
3. Which Identity Creates the Greatest Blast Radius?
Eventually, many attacks turn out to have some intersection with identity.
A compromised developer can change code. A stolen service credential can invoke functionality. An overprivileged agent can call dangerous tools. An administrator can modify thresholds or do things like suppress logs.
Leaders should identify the human and non-human identities capable of influencing high-consequence AI systems.
They should understand:
- Who can alter data, especially data used for training models.
- Who can change policies or guardrails.
- Who can deploy or replace models.
- Which agents can invoke production tools.
- Who can disable monitoring.
- Which identities can approve their own changes.
The most dangerous identity may not possess the most obvious administrative title. It may be an automation account that quietly connects elements such as models, data, and production systems.
4. Can the Human in the Loop Be Manipulated?
Organizations frequently rely on “human in the loop” as their final safeguard.
Adversarial thinking asks whether the human loop actually works.
People may over-trust AI recommendations, especially when a system appears confident or technically sophisticated. Reviewers may approve outputs automatically because of enormous volumes of data to review, time pressure, weak interfaces, inadequate context, or fear of challenging a system the organization has heavily promoted.
An attacker may also target the reviewer, a human, directly. After all, whether purposely or not, humans are the source of many unfortunate cyber events.
Manipulated systems could present targeted evidence, conceal uncertainty, overwhelm the reviewer with volume, or frame the decision in a way that encourages the desired behavior (of the nefarious actor).
Meaningful human oversight requires authority, context, time, training, and psychological permission to disagree.
A person who can technically click “reject” but is organizationally discouraged from doing so is not an effective control.
5. What Happens When Trusted Components Become Untrustworthy?
AI systems depend on external data, models, open-source libraries, cloud platforms, plugins, APIs, and vendors.
Each dependency extends the trust boundary.
A provider may change a model. An integration may gain new capabilities, or lose existing ones. A library may become compromised. A data source may introduce manipulated content. A vendor may retain more information than expected.
Third-party risk assessments often occur before deployment and then fade into GRC oblivion.
Adversarial governance treats trust as temporary.
Leaders should know which external components can influence decisions or actions, what changes providers are making, how the organization detects those changes, and whether it can continue operating safely when the dependency becomes unavailable or untrustworthy. These are elements that make up a resilient ecosystem.
6. How Would We Detect a Quiet Failure?
Not every AI incident will produce blinking lights, loud alarms, an obvious outage or some catastrophic result.
Some of the most damaging failures will appear gradually.
A model may become slightly less accurate for a specific population. An agent may begin retrieving more data than it needs. A recommendation system may slowly favor manipulated content. A fraud model may become less sensitive to a criminal technique.
The system continues to operate, and traditional availability metrics remain healthy.
Leaders need indicators that reveal changes in context, behavior, authority, data access, confidence, exception rates, and human impact.
They should also monitor near misses. An action stopped by a human or compensating control still reveals a weakness in the system.
Simply put, governance cannot measure only uptime and adoption. It must measure whether the system remains within its intended behavioral boundaries.
7. Can We Contain the System Before We Understand the Incident?
Executives often assume that teams can shut down an AI system when something goes wrong.
That assumption should be tested.
A system may be embedded in business workflows. Multiple applications may depend on this integration. Agents may retain active sessions or credentials. Third-party components may continue processing information. Teams may hesitate because shutting some system down could create operational consequences.
Incident response requires the ability to reduce authority quickly, even before investigators understand the complete failure.
Organizations should be able to:
- Revoke agent and service-account access.
- Disable specific tools or integrations.
- Quarantine suspicious data sources.
- Roll back models and configurations.
- Move automated decisions into manual review.
- Preserve evidence for investigation.
- Continue critical operations in a degraded mode.
The ability to stop an AI system safely needs to be a design requirement, not an emergency improvisation.
Turn Governance Into an Adversarial Operating Model
An adversarial mindset must produce more than provocative questions. When pursued properly this mindset should guide and mold entire security programs.
Organizations should embed this mindset into their operating model.
That includes:
- Threat modeling – examine the complete AI architecture, not only the model.
- Abuse-case development – document how legitimate capabilities could support illegitimate objectives.
- Red teaming – without boundaries (attackers have none) test models, identities, integrations, users, and workflows.
- Control validation – under realistic conditions, prove that guardrails work rather than accepting that they exist.
- Behavioral monitoring – actively detect changes in data access, authority, tool use, and outcomes.
- Incident exercises – regularly rehearse containment, rollback, investigation, communication, and recovery. This can follow the standard tabletop model even though many of those exercises introduce boundaries and constraints that take them out of the realm of realistic conditions.
- Continuous reassessment – review risk whenever the system’s data, model, tools, users, or authority change.
Governance should operate as a feedback loop:
Assume → Challenge → Test → Observe → Adapt
Typically, a policy changes only when someone updates the document. An adversarial governance program needs to change when evidence reveals that an assumption no longer holds.
Policy Describes the Organization You Hope Exists
AI governance policies describe approved behavior. They define responsibilities, expectations, controls, and boundaries.
These definitions are necessary.
Sadly, they are not the same as actual readiness.
The real organization includes shortcuts, legacy access, human bias, compromised credentials, conflicting incentives, third-party dependencies, weak integrations, and determined adversaries.
An adversarial mindset closes the distance between the organization described by policy and the one that actually operates under pressure.
Leaders must ask more than whether AI is accurate, compliant, or useful.
They must ask how someone could manipulate it, misuse it, impersonate a trusted identity, corrupt its data, exploit its authority, or influence the humans responsible for oversight.
AI governance without an adversarial mindset is just policy.
Policy defines the rules.
The adversarial mindset determines whether those rules survive contact with reality.
