The short version

"How autonomous should the agent be" is the wrong question, because it applies one dial to actions that have nothing in common. The useful axis is reversibility. An agent can act freely where the action can be taken back, act with an audit trail and an undo path where recovery costs something, and must never commit alone where the consequence has left your control. Nearly every agent incident we have seen or read about reduces to the same mistake: an action filed one band lower than it belonged, usually because reversing it looked easy from inside the system.

Most discussion of agent safety is a conversation about trust, conducted in adjectives. Is the model good enough yet, is the use case low-risk, are we comfortable. None of it produces a design, because none of it is a property of the action being taken.

There is a better question, and it has the advantage of being answerable by someone who knows nothing about machine learning: if this goes wrong, can we take it back?

Reversibility is the real axis

Sort every action your agent can take into three bands.

Reversible. Reading. Searching. Drafting. Summarising. Producing a recommendation nobody has acted on. If this goes wrong the cost is a wasted moment and the remedy is to try again. Let the agent run. Gating this band is where teams spend their approval budget and get nothing for it.

Recoverable with effort. Creating a ticket, writing a file, updating a record, scheduling something. These can be undone, but undoing takes work and someone has to notice first. The agent can still act, on two conditions: every action is logged with the reasoning that produced it, and an undo path exists and has actually been tested. "We could reverse it manually" is not an undo path until someone has.

Irreversible. Money moves. A message is sent. Something is deleted with no restore. A third-party system accepts a request. Here the agent proposes and a human commits — not because the model is untrustworthy, but because the cost of being wrong is unbounded and no amount of model improvement bounds it.

SORT BY WHAT YOU CAN TAKE BACK BAND EXAMPLES WHO COMMITS REVERSIBLE cost of error: a moment read, search, draft recommend, summarise the agent no gate earns its keep RECOVERABLE cost of error: someone's day raise a ticket, write a file update a record the agent, logged with a tested undo IRREVERSIBLE the consequence has left your control money moves, a message sent a deletion, a third party accepting a request a human, always the agent proposes and shows its evidence A BETTER MODEL MOVES NOTHING BETWEEN THESE BANDS BECAUSE REVERSIBILITY IS A PROPERTY OF THE ACTION, NOT THE AGENT
Incidents cluster where an action was filed one band too low.

Three things less reversible than they look

The classification is easy in principle and goes wrong in the same three places.

Sending anything to a person. Deleting a message from your own system is trivial, which makes "send" feel recoverable. It is not. The copy that matters arrived somewhere you do not control, and was read by someone whose opinion you cannot un-form. A retracted email is a second event, not an undo.

Writing to a record other systems read. Updating a field looks recoverable because the previous value can be restored. But between the write and the restore, downstream systems acted on it — a report ran, a rule fired, a notification went out. You can revert the field. You cannot revert what it caused.

Anything a third party accepted. Once an external API returns success, the action belongs to them and their reversal process, on their timetable and terms. This is worth labelling irreversible even where a cancellation endpoint exists, because now you are depending on a second call succeeding during exactly the incident that made you need it.

Propose, don't commit

The pattern that falls out of all this is simple. The agent produces a fully specified proposed action, along with the evidence that led to it. Something else commits — a person, or a deterministic rule that checks the proposal against constraints the agent cannot see or modify.

That split plays to both components' strengths. Language models are genuinely good at reading messy context, pulling out what matters and drafting the next step. They are not the right thing to hold a limit in, because a limit must behave identically every single time, and that is what code is for. This is the same argument as in banking architecture — the client may display a limit, but the server enforces it — with the agent playing the part of the client.

Two details make the difference between the pattern working and merely existing.

The proposal must be complete. "Refund this customer" is not a proposal; "refund £42.60 to order 11823 against the original card, reason: duplicate charge" is. A proposal a human has to research before approving has moved the work rather than reduced it.

And the validation rule must be independent. If the agent generates both the action and the check that approves it, there is no check.

When approval becomes theatre

Worth stating because it is the most common way a well-designed gate fails: an approval step only works if the person can genuinely evaluate the thing in the time they have.

An agent proposing four decisions an hour gets real review. The same agent proposing four hundred gets a rubber stamp — and a rubber stamp is worse than no gate at all, because it generates an audit trail that says a human reviewed each one. You have added risk and documented its absence.

So the volume of proposals is a design constraint, not an output. If the queue is unreadable, narrow what the agent is allowed to propose, batch similar items so one decision covers many, or move the clear-cut cases to a deterministic rule and send only the ambiguous remainder to a person. That last one is the same deferral design that document AI systems need, for the same reason: the value is in routing the hard cases to a human, not in pretending there are none.

Failure modes to design for

Silent failure. A model will nearly always produce something rather than refuse, so a failure arrives as a plausible wrong answer and your monitoring records a successful call. Two defences: validate the proposal against constraints outside the agent, and track how often it declines to act. A refusal rate of zero means it is guessing when it should be stopping.

Partial completion. Multi-step agents fail halfway through, leaving a state no human designed. Decide in advance whether a half-done sequence rolls back or holds for review, and make it visible either way. The worst outcome is an agent that quietly abandons step three and reports success.

The duplicate retry. A control loop that cannot tell whether a step succeeded will try it again. Agents hit this far more than ordinary code, because "did that work?" is itself a judgement they are making. Every committing action needs an idempotency key, generated at proposal time rather than at execution, so a retry is recognisably the same action and not a second one.

What to settle before building

  1. List every action the agent can take and put each in a band. The argument this provokes is the valuable part.
  2. For each recoverable action, name the undo path and have someone run it once.
  3. Write the validation rules outside the agent, in ordinary code, against constraints it cannot read.
  4. Estimate the approval volume before you build the queue, and design for what a person can actually read.
  5. Give every committing action an idempotency key.
  6. Decide what a half-finished sequence does, and make it observable.

None of this is about the model, which is the point. These are the parts that stay true as models improve, and they are the parts that decide whether an agent is still running in production a year after the demo.

Questions we get asked

How autonomous should an AI agent be?

Autonomy is the wrong question because it is a single dial applied to actions that differ enormously. Classify each action by whether it can be taken back instead. An agent can be fully autonomous over reversible actions, autonomous with an audit trail and an undo path over recoverable ones, and should never commit an irreversible action on its own.

What counts as an irreversible action for an agent?

Anything that moves money, anything another person has now seen, anything deleted without a restore path, and anything a third-party system has accepted. The common thread is that the consequence leaves your control. A sent email is irreversible even though deleting it from your outbox is trivial, because the copy that matters is no longer yours.

Is a human approval step enough to make an agent safe?

Only if the human can actually evaluate what they are approving in the time available. An approval queue that arrives faster than it can be read becomes a rubber stamp, and a rubber stamp is worse than no gate because it manufactures an audit trail suggesting review happened. If approvals cannot be meaningful, reduce what the agent proposes rather than adding a button.

What is the propose-and-commit pattern?

The agent produces a fully specified proposed action together with the evidence behind it, and something else performs the commit — a person, or a deterministic rule that validates the proposal against constraints the agent cannot modify. It keeps the language model where it is strong, which is drafting and reasoning over context, and keeps the final write under a mechanism that behaves the same way every time.

Why do agents fail silently in production?

Because a language model will usually produce something rather than refuse, so a failure arrives as a plausible wrong answer rather than an exception. Conventional monitoring sees a successful call. The defences are checking the proposed action against constraints the agent cannot see or alter, and measuring the rate at which it declines to act, which should never be zero.

How do you stop an agent retrying an action twice?

The same way any distributed system does: make the action idempotent, with a key generated at proposal time rather than at execution. Agents retry more than conventional code because a control loop that cannot tell whether a step succeeded will often try again. Without an idempotency key the second attempt is a second action.

Related service AI development Agents with boundaries, built for production

If you are scoping an agent and want the action list banded before anyone writes a prompt, that is the hour that decides whether the thing survives its first bad week. It is where we start on AI projects, and it is cheaper than every alternative.