There is a moment coming in AI that I think many organisations are still underestimating.
It is not the moment when AI becomes smarter than us.
It is the moment when AI stops merely giving us answers and starts taking actions on our behalf.
That shift changes everything.
A chatbot that gives a bad answer creates a quality problem.
An AI agent with access to your systems, credentials, APIs, code repositories, databases or production environment can create an operational problem.
And that is why I think the most important question in AI right now is no longer:
How intelligent is the model?
It is:
How much authority are we prepared to give it?
For the last couple of years, most conversations about AI have focused on capability.
Can it write code?
Can it analyse documents?
Can it answer customer questions?
Can it automate business processes?
Those are still important questions.
But I think we are now entering a much more serious phase.
The question is no longer only what AI can say.
It is what AI can do.
And from an engineering, security and governance perspective, that difference changes almost everything.
A chatbot makes mistakes. An agent can take action.
Recent cybersecurity evaluations involving some of the world’s most capable AI systems have shown agents behaving in ways that were not always anticipated by the people running the tests.
They have searched for vulnerabilities, used tools, interacted with external systems and, in some cases, taken unexpected actions while attempting to complete an objective.
Before this turns into a Hollywood story, I want to make one thing clear.
This is not evidence that machines have suddenly become conscious or decided to take over the world.
That is not the interesting part.
The interesting part is much more practical.
We are building software that can receive a goal, plan several steps ahead, use tools, react when something blocks it and find another route to continue.
Once software can do that, we need to stop thinking about it as just another application feature.
We are giving it agency.
And agency without boundaries is where things get interesting.
The difference between a wrong answer and a wrong action
If I ask a chatbot a question and it gives me the wrong answer, I have a quality issue.
Maybe the model hallucinated.
Maybe the prompt was weak.
Maybe the grounding was poor.
Maybe the response should have been validated before being shown.
Those problems are serious, but manageable.
Now imagine something different.
An AI agent has access to:
your browser
APIs
source code
credentials
databases
a terminal
a ticketing system
production tooling
Now the risk changes.
A wrong answer is one thing.
A wrong action is something else entirely.
That is the distinction I think engineering leaders need to pay attention to.
The human saying “no” still matters
One of the most interesting recent AI security evaluations involved an agent taking actions that researchers had not explicitly intended.
At one point, a human maintainer still had to approve what the system was trying to do.
The human refused.
Nothing happened.
To me, that is not a side detail.
That is the story.
The final control was still human.
And the human said no.
That is exactly where the conversation around AI governance needs to move.
“Human in the loop” cannot just be an architecture diagram
Almost every enterprise AI presentation now includes the phrase:
Human in the loop.
It sounds good.
It makes everyone feel safer.
But I often wonder whether we have really defined what it means.
What exactly is the human approving?
At what point in the workflow?
Can the AI continue if nobody responds?
What happens when confidence is low?
What happens when the risk level is high?
Can someone override the AI?
Is the override logged?
Can we later reconstruct why the AI made a recommendation and why the human approved or rejected it?
That is where governance becomes real.
If AI is only drafting something, the risk is relatively contained.
If AI can send the message, update the customer record, execute code, modify infrastructure, approve a transaction or change production data, then the control model has to become much stronger.
The architecture must change with the amount of authority we give the system.
That is something I think many organisations may underestimate while rushing towards automation.
The biggest AI risk may not be intelligence
We spend a lot of time discussing model capability.
How strong is the reasoning?
How good is the coding?
How large is the context window?
How autonomous can the agent become?
Those things matter.
But from an enterprise perspective, another question may be even more important:
What permissions have we given it?
An incredibly capable AI model with no access to anything can still generate bad information.
But its ability to create operational damage is limited.
A less sophisticated system with broad credentials, unrestricted network access and permission to execute actions could potentially create much more damage.
For me, the risk equation is increasingly:
Capability + Autonomy + Access + Time
That combination matters more than capability alone.
We already know how to solve part of this problem
The interesting thing is that these governance principles are not completely new.
Cybersecurity has been using them for years.
We do not give every employee production access.
We do not give every system administrator privileges.
We use least privilege.
We separate responsibilities.
We record sensitive actions.
We require approvals for high-risk changes.
Why should AI agents be treated differently?
If anything, I would argue that we should initially be more cautious.
Start with one question: what can this agent do without a human?
If I were designing an enterprise AI agent today, I would not start by asking:
What tools can we connect to it?
I would start with:
What is this agent allowed to do without a human?
That question should influence the whole architecture.
Low-risk activity
An AI drafts a customer response.
A human reviews it and sends it.
Fine.
Medium-risk activity
An AI classifies an issue and recommends the next action.
The recommendation may proceed only if confidence is high enough and the user confirms it.
Better.
High-risk activity
An AI proposes a database change, production deployment, financial transaction or cybersecurity action.
At that point, I do not want the AI acting autonomously.
I want it to:
prepare the action
explain its reasoning
show the risk
present supporting evidence
wait for an authorised human
That is not preventing innovation.
That is engineering discipline.
Testing has to change too
This is where I think the testing profession has a very interesting future.
Traditional testing often asks:
Did the system produce the correct result?
With agentic AI, that question is no longer enough.
We also need to test:
What happens when the agent cannot achieve its goal normally?
Does it stop?
Does it ask for help?
Does it try another approach?
Does it attempt to work around a restriction?
Does it request additional access?
What happens when instructions conflict?
What happens when confidence drops?
What happens when an external tool gives an unexpected response?
What happens when the user intentionally provides malicious input?
What happens when the agent has technical access to information that it should not actually use?
That is a very different testing problem.
We are no longer testing only outputs.
We are testing behaviour.
And behaviour becomes much harder to predict when the system is capable of planning dynamically.
QA may become more important, not less
There is a popular narrative that AI will replace testers.
I think that is too simplistic.
AI will absolutely automate more test design, test execution, analysis and reporting.
That is already happening.
But autonomous systems introduce an entirely different category of quality risk.
Someone still needs to define:
acceptable behaviour
unacceptable behaviour
risk boundaries
approval thresholds
safe fallback behaviour
escalation paths
audit requirements
recovery mechanisms
That work sits somewhere between testing, architecture, security, governance and product risk.
The role will change.
The responsibility will not disappear.
We should also avoid sensationalising the current incidents
Some headlines make recent AI security incidents sound as though models simply escaped from their environments and started attacking organisations.
That is an oversimplification.
In many cases, these were deliberately difficult cybersecurity evaluations.
Some safeguards were disabled.
Some systems were deliberately given access to tools and networks so researchers could understand the boundaries of their capability.
That context matters.
But dismissing the incidents because they happened during testing would also be a mistake.
The entire purpose of testing is to discover uncomfortable behaviour before it appears in production.
If testing exposes something we did not expect, the right response is not panic.
It is:
Understand it. Control it. Test it again. Improve the architecture.
That is what mature engineering looks like.
This is not just a Silicon Valley problem
It is easy to think that frontier AI security is something happening far away in Silicon Valley, London or Washington.
But organisations everywhere are starting to connect AI into normal business systems.
CRM.
Customer support.
Software engineering.
Finance.
HR.
Security.
Operations.
Service management.
The important question for most organisations is not whether they are building frontier AI models themselves.
They are not.
The question is what happens when increasingly capable AI agents are connected to the systems that actually run their business.
Once agents begin crossing those boundaries, the blast radius of a poorly designed workflow becomes much larger.
That is why governance cannot be something we bolt on after the demonstration works.
It needs to be part of the original design.
My view: AI should earn autonomy
I am very positive about AI.
I build with it.
I use it regularly.
I believe it will fundamentally improve how we design, test and operate software.
But I do not believe the goal should be maximum autonomy from day one.
My view is simple:
AI should earn autonomy.
Start with recommendation.
Then drafting.
Then supervised action.
Then limited automation.
Increase authority only when the behaviour is understood, tested, monitored and auditable.
That is how mature organisations manage people and systems already.
You do not give someone unrestricted production access on their first day.
Why would we do it with an AI agent?
Trust may become more valuable than raw capability
Every major AI company is competing on capability.
Better reasoning.
Better coding.
Faster models.
Longer context.
More autonomous agents.
All of that is exciting.
But enterprises are going to ask another set of questions:
Can I understand what this AI did?
Can I stop it?
Can I override it?
Can I restrict what it can access?
Can I audit its actions?
Can I identify who approved a high-risk decision?
Can I prove what happened after something goes wrong?
Those questions will matter just as much as model intelligence.
That is where AI engineering meets governance.
And I think it will become one of the defining engineering challenges of the next few years.
The most impressive AI system may not be the one capable of doing everything without us.
It may be the one that understands exactly when it should stop and ask a human.
My takeaway
AI agents are moving from generating information to taking action.
That is a major technological step forward.
It is also a major increase in engineering responsibility.
As we give AI more capability, we need to invest equally in permissions, testing, monitoring, approval, security and accountability.
The future of AI should not be about removing humans from every workflow.
It should be about putting humans at the points where judgement and accountability actually matter.
AI can accelerate the decision.
Accountability should remain human.
This article reflects my personal views, professional experience and interpretation of the developments discussed. AI tools were used during the research and editing process, but the arguments, conclusions and responsibility for the published content are my own.