There is an important difference between an AI system passing its own organisation’s tests and surviving independent challenge.
As models become increasingly capable of interacting with software, networks, data and external tools, that difference becomes commercially significant.
The central assurance question is changing from:
Does the model work?
to:
What happens when somebody deliberately tries to make it fail?
Internal testing has structural limitations
Internal teams understand their systems better than anybody.
That is valuable.
But familiarity can also create blind spots.
Developers know intended behaviour.
Independent evaluators are incentivised to search for unintended behaviour.
Both are necessary.
NIST’s generative AI risk guidance includes independent audit and AI red-teaming. The principle is familiar from other high-stakes industries.
Financial statements are independently audited.
Cybersecurity systems undergo penetration testing.
Safety-critical products face certification.
AI systems receiving meaningful autonomy may increasingly require equivalent challenge.
Agents increase the stakes
A chatbot producing an incorrect sentence creates one category of risk.
An agent capable of accessing systems, executing code, sending messages or modifying records creates another.
The potential failure is no longer contained inside the conversation.
It can propagate into the environment.
Testing therefore needs to examine more than answer quality.
Evaluators should consider:
permissions;
tool use;
privilege escalation;
unexpected action chains;
resistance to manipulation;
failure recovery;
human escalation;
auditability.
Governance must be able to stop deployment
A governance framework that can only produce reports is incomplete.
If assurance identifies a material weakness, somebody must possess the authority to stop or restrict deployment.
That principle is uncomfortable because it can slow commercial activity.
But a control that cannot change the outcome is not really a control.
Independent challenge therefore needs both technical credibility and organisational authority.
Evidence becomes the product
As organisations purchase AI systems, procurement teams will increasingly ask vendors for evidence.
What testing has occurred?
Who performed it?
Which failure scenarios were examined?
What remains unresolved?
Which permissions are enabled?
How is activity monitored after deployment?
That evidence can become a commercial differentiator.
The safest system will not necessarily be the least capable.
It may be the system whose capability is most clearly understood and controlled.
SOS perspective
Through SOS AI assurance and governance, we view independent challenge as a core element of responsible high-stakes AI.
The objective is not to prevent useful autonomy.
It is to establish the evidence required to grant autonomy responsibly.
The strongest governance question may therefore be remarkably simple:
What happened when someone independent tried to break it?
Sources reviewed
Related 2026 analysis: Explore independent AI assurance in procurement.
Turn this SOS analysis into a controlled commercial decision.
Discuss ai assurance with SOS