An AI assistant that gives a poor answer can mislead someone. An AI agent that can take actions can also change data, send messages, trigger workflows, spend money or affect customers.
That changes the engineering problem. The question is no longer only “Can the model complete the task?” It is “What is it allowed to do, how will we detect trouble, and how quickly can we stop it?”
A stronger system prompt may help shape behaviour, but it is not a security boundary. Controls need to exist outside the model.
Separate capability from permission
An agent should not receive broad access simply because it might need it. Give it the minimum tools and data required for the task. If it only needs to summarise records, it should not have permission to edit or delete them.
Think in tiers:
- Tier 1 — Read and recommend: search, summarise, classify and draft. No external side effects.
- Tier 2 — Reversible actions: create a draft, update a low-risk field or open a ticket, with clear logs and an undo path.
- Tier 3 — Consequential actions: send external communications, change production systems, approve payments or modify access. Require explicit approval or narrowly defined policies.
- Tier 4 — High-impact or irreversible actions: destructive operations, financial transfers, sensitive data disclosure or security changes. Keep these outside autonomous execution unless a separately justified control system authorises them.
The exact tiers depend on the business. The principle is stable: autonomy should be earned through evidence and constrained by impact.
Five controls to build outside the prompt
- Tool-level authorisation. Enforce permissions in the API or service, not just in natural-language instructions. Check identity, scope and resource on every action.
- Bounded execution. Limit the number of steps, runtime, spend, records affected and destinations an agent can reach. Require confirmation when it crosses a defined boundary.
- Approval gates. Pause before consequential actions. Show the operator what will happen, which data will change and why the agent proposes it.
- Audit trails. Record the request, relevant context, tool calls, decisions, approvals and outcomes. Logs should support investigation without unnecessarily retaining sensitive data.
- A real stop mechanism. Make it possible to revoke credentials, disable tool execution, cancel queued work and prevent retries. Test this path before production.
Plan for prompt injection and unexpected behaviour
Agents often read untrusted content: emails, webpages, documents, tickets or repository files. That content may contain instructions that conflict with the task. Treat retrieved text as data, not as authority to grant new permissions.
Keep secrets out of model-visible context where possible. Separate the agent’s planning from the service that enforces actions. Validate arguments and destinations, and do not let a model decide whether its own actions satisfy every security policy.
Before launch, test realistic failure scenarios: a malicious instruction inside a document, a request to export unrelated records, repeated tool calls, an unexpected large purchase, a loop that consumes tokens, and a downstream service returning misleading data.
Define the operating envelope
For each agent, write down what it may do, what it must never do, when it needs approval, which systems it can reach, what limits apply and who owns incidents. Set measurable thresholds for latency, failure rate, spend and policy violations.
Start with a narrow workflow. Run it in read-only or recommendation mode first, compare its decisions with human decisions, then introduce low-risk actions. Expand permissions only after the system has passed defined evaluations and the recovery process has been exercised.
Make the stop button part of the design
A kill switch that nobody has tested is a hope, not a control. Rehearse revoking access during an active run. Confirm that pending actions are cancelled, retries do not restart the job, downstream credentials are disabled and an operator receives an alert. Document who can invoke the stop and how the service is safely restarted.
Track both productivity and control performance: successful task completion, human override rate, policy violations, unintended side effects, time to detect a problem and time to contain it. A high completion rate is not enough if failures are difficult to detect or reverse.
Autonomy is a release decision
Give agents more freedom only when evidence supports it. Begin with low-impact actions, evaluate against realistic adversarial and ordinary cases, keep permissions narrow and make rollback practical. The goal is not maximum autonomy. It is useful autonomy within a boundary the organisation understands and can enforce.
A question for engineering and security leaders: if one of your agents started behaving unexpectedly right now, could you stop its actions—not just ask it to stop?