AI agents change the engineering question from “What can the model generate?” to something much more consequential:
“What are we prepared to let the system do without asking a human first?”
An agent that only produces a recommendation is one kind of system. An agent that can update a CRM record, send an email, deploy code, approve a refund or change a production configuration is something else entirely.
The temptation is to frame this as a choice between human-in-the-loop and fully autonomous AI. In practice, there is a much more useful design space in between.
Autonomy should be treated as an engineering variable: something you deliberately increase or decrease according to the consequences of an action.
| Observe | Suggest | Prepare | Bounded execute | Adaptive autonomy |
Increase authority only as evidence, controls and reversibility improve.
Autonomy is not binary
A useful starting point is to separate what an agent can observe, decide and do.
- Observe: gather information and understand the state of a system.
- Suggest: produce a recommendation for a human to review.
- Prepare: create the artefacts needed for an action, but stop before execution.
- Execute: perform a bounded action automatically.
- Escalate: recognise uncertainty or risk and hand control back to a person.
The same agent can operate at different levels for different actions. It might automatically classify a support ticket, prepare a customer response, update an internal field and still require explicit approval before sending the message.
That is usually a more useful architecture than asking whether the agent is “autonomous”.
The action matters more than the model
When deciding how much autonomy to grant, the important question is not simply how capable the model appears to be. It is what happens when it is wrong.
Consider five properties of an action:
| Property | Question |
|---|---|
| Reversibility | Can the action be safely undone? |
| Blast radius | How many customers, systems or records could it affect? |
| Authorization | Is the agent actually permitted to make this decision? |
| Ambiguity | How much interpretation is required? |
| Auditability | Can we reconstruct what happened and why? |
These characteristics give you a much better basis for autonomy decisions than a generic statement such as “the model is 95% accurate”.
An inaccurate draft email may be annoying. An inaccurate payment instruction or production change may be materially different.
A practical autonomy ladder
One way to design an agent is to establish explicit autonomy levels.
Level 1 — Recommend
The agent analyses the situation and proposes an action. A human makes the decision.
This is appropriate when the decision is high-impact, difficult to reverse or still poorly understood.
Level 2 — Prepare
The agent does the work needed to make the action easy for a human to approve.
For example, it might prepare a pull request, draft a customer response, assemble a procurement request or identify the records that need changing.
Level 3 — Execute within a boundary
The agent can act automatically, but only inside explicitly defined limits.
Those limits might include which systems it can access, which records it can change, the maximum financial value, allowed API operations or the environments in which it can operate.
Level 4 — Execute and recover
The agent can execute a wider range of actions and is also given mechanisms to detect failures, retry safely and roll back changes.
This is a substantially different engineering problem because recovery becomes part of the agent design.
Level 5 — Adaptive autonomy
The system changes how much autonomy it uses according to context, confidence, policy and risk.
For example, an agent might automatically process routine cases but escalate unusual combinations of circumstances to a human.
The important point is that higher autonomy should be earned by evidence, not granted because an agent looks impressive in a demonstration.
Guardrails are part of the architecture
If an agent can take actions, prompting alone is not a sufficient control mechanism.
Useful technical controls include:
- Allow lists: explicitly define which tools and operations are available.
- Scoped credentials: give the agent the minimum permissions required for its task.
- Spending and rate limits: constrain financial exposure and runaway activity.
- Dry-run modes: allow the agent to demonstrate an action without executing it.
- Approval gates: require human confirmation for defined classes of action.
- Idempotency: make retries safe rather than allowing duplicate side effects.
- Audit logs: record inputs, decisions, tool calls and outcomes.
- Rollback: provide a tested mechanism for reversing changes where possible.
- Time limits: prevent an agent from continuing indefinitely.
These controls should sit outside the model wherever possible. A model can be instructed not to issue a particular command, but an authorization layer can enforce that restriction.
Human approval is not automatically a safety mechanism
Adding a human approval button sounds reassuring. It can also create a false sense of security.
If an agent produces hundreds of actions per hour and a human is expected to approve every one, the human may become a rubber stamp. If the information needed to make a decision is buried inside a long explanation, the approval step may add very little real control.
A good approval workflow therefore needs to answer:
- What exactly is the human approving?
- What information do they need to make the decision?
- What actions are automatically blocked?
- What happens if the human does not respond?
- Can the human meaningfully inspect the proposed change?
The goal is not simply to put a person somewhere in the workflow. It is to put human judgement where it provides useful control.
Run an autonomy experiment
If you are considering deploying an agent, I would resist the urge to start with “How autonomous can we make it?”
Instead, choose one narrow workflow and run a controlled experiment.
Week 1: Map the workflow
Document the current process from input to outcome.
For every action, record:
- who currently performs it;
- what information is required;
- what systems are touched;
- what can go wrong;
- how reversible the action is;
- what evidence is required for audit.
Week 2: Run in shadow mode
Let the agent observe real cases and make decisions without taking side effects.
Compare its proposed actions with what actually happened.
This gives you a baseline without giving the agent production authority.
Weeks 3–4: Introduce bounded execution
Allow the agent to execute a carefully selected class of low-risk actions.
Keep higher-risk actions behind an approval gate.
Measure what happens rather than relying on anecdotal impressions.
Weeks 5–6: Test the boundaries
Deliberately introduce ambiguity, missing information, unusual requests, permission failures and tool failures.
The interesting question is not only whether the agent completes normal cases. It is whether it behaves safely when the world stops looking like the examples it was given.
What should you measure?
Agent evaluations need to cover both the quality of decisions and the consequences of actions.
| Metric | What it tells you |
|---|---|
| Task completion | How often the intended outcome is achieved. |
| Intervention rate | How often a human needs to step in. |
| Unsafe action rate | How often the agent attempts or performs a prohibited action. |
| Rollback rate | How often executed actions need to be reversed. |
| False escalation | How often the agent asks for help when it could safely proceed. |
| Latency | Whether autonomy actually improves the end-to-end workflow. |
| Cost | Whether the additional model and tool usage is economically justified. |
One metric deserves particular attention: harmful action rate.
A system that completes 99% of tasks but occasionally takes a highly damaging action may require a very different architecture from one that simply gets 95% of routine classifications correct.
The autonomy boundary should be explicit
One of the most useful artefacts you can create is an agent action policy.
For every available tool or operation, classify it as something like:
- Allowed: the agent can execute automatically.
- Conditional: execution is permitted only when defined constraints are satisfied.
- Approval required: a human must explicitly authorise it.
- Prohibited: the agent cannot perform it.
This turns a vague conversation about “AI safety” into an engineering contract.
It also makes reviews easier. When a new capability is proposed, you can ask where it belongs in the policy rather than debating autonomy in the abstract.
Start with reversibility
If I were designing an agent from scratch, one of the first questions I would ask about every proposed action is:
“If the agent gets this wrong, how quickly and reliably can we put the world back?”
That question naturally pushes reversible, observable and low-blast-radius operations towards greater autonomy.
It also highlights where stronger controls are needed.
A CRM agent changing an internal classification is very different from an agent changing a customer’s bank details. A coding agent opening a pull request is different from one deploying directly to production. A support agent drafting a refund is different from one authorising the payment.
The real engineering challenge
The interesting question about agents is not whether they can use tools.
They can.
The interesting question is whether we can construct a system in which the agent’s authority is proportional to the evidence that it can be trusted with that authority.
That means treating autonomy as a controlled engineering variable, measuring behaviour in realistic conditions and increasing the boundary gradually.
Start with observation.
Then recommendation.
Then preparation.
Then bounded execution.
And only then consider giving the system more freedom.
Autonomy should not be a feature you switch on. It should be a capability you progressively earn.