Boundaries before scale: a practical guide to human oversight

A person clicking "approve" at the end of an AI workflow may be doing little more than taking responsibility for a decision they cannot inspect. If the agent has already sent a message, changed a record or fetched data it was not entitled to see, the final sign-off arrives too late.

Oversight has to be designed into the route the work takes. NIST's voluntary AI Risk Management Framework is a useful reference for managing risk throughout an AI system's life, while OWASP's Agent Control Standard calls for agents whose access and actions can be inspected, traced and controlled at runtime.^[1][2] Neither document is a certificate that your particular deployment is safe. This guide is a way to turn those ideas into one testable business workflow.

Pick the decision before you pick the approver

Take an accounts-payable assistant that reads an invoice and prepares a payment. The invoice amount may be small, but changing a supplier's bank account is a different kind of decision. The finance owner should specify which data the assistant may read, what it may draft, which system changes require a named person's approval and which actions it may never attempt. These are proposed design choices, not a claim about a client implementation.

Write the boundary as a verb and an object: "read approved invoices", "draft a payment", "do not alter payee details". Name the business owner and the permission's expiry. A general instruction to "keep a human in the loop" leaves the consequential part of the job undefined.

Give the reviewer something to inspect

For each proposed write, show the reviewer the source record, the changed fields, the destination and what the agent could not verify. A reviewer should be able to compare the proposed payment with the invoice and supplier master record without trusting the agent's summary. Keep the action log in the surrounding system, not only in the agent's own narrative. OWASP's standard describes visibility into what an agent is, what it can access, what it did and why, plus policy hooks that can enforce controls at runtime.^[2]

A person who sees only a confident sentence and an "approve" button is not a control. A person who sees the payee, amount, source and precise system action can make an actual decision.

Stop the work at the right moment

The cleanest approval point is before the irreversible action. The assistant can prepare a payment; a finance approver authorises its release. It can draft a supplier email; a person sees the recipient and content before it sends. If an invoice points to a new bank account, the workflow should pause and follow an independently defined verification process, rather than asking the same agent whether the change looks legitimate.

Stopping is more than a UI button. Test whether the organisation can revoke access while a task is running, prevent further writes and see which actions already completed. Ask what happens on retries: if a tool times out after creating a payment, does the agent check the payment record before attempting it again? These are acceptance tests for the system around the model.

Rehearse an exception, not a tidy demo

Run one representative job with a deliberately ambiguous invoice. Introduce a missing field or a changed payee. Check whether the assistant stops, records the reason, routes the exception to the right owner and resumes only after a fresh decision. Repeat the exercise after changing its permissions mid-task. The test should expose both the work completed and the work refused.

NIST describes its framework as voluntary guidance for incorporating trustworthiness into AI design, development, use and evaluation. It does not prescribe one approval screen for every workflow.^[1] The oversight should be proportionate to the action: drafting a note and releasing money should not cross the same gate.

Before scaling an agent, try to replay a difficult run. Can the owner identify each write, who authorised it and where the agent stopped? If the answer depends on a polished final summary rather than an action record, the oversight is still a design sketch.

Sources

1. NIST AI Risk Management Framework
2. OWASP Agent Control Standard (ACS)