Human-in-the-loop design for an AI agent decides which actions need a person’s approval, how uncertain results reach one, what happens when nobody responds, and what is logged. Classify actions by reversibility, set confidence thresholds by weighing review cost against error cost, make escalation fail closed, and log every step so any decision can be replayed.
An AI agent is useful because it acts: it looks things up, drafts replies, fills forms and prepares changes. That is also why it needs oversight. A chatbot that answers wrongly produces a bad answer; an agent that acts wrongly produces a changed record, a sent email or a payment. Human-in-the-loop (HITL) design is the set of decisions that keep those actions inside limits you chose. This guide covers the four review patterns, a policy that matches the oversight to the action, how to set confidence thresholds with a worked cost example, how to design escalation and the review screen, what to log, and how to move from suggestion to autonomy in stages. It supports our AI agent development work and the broader AI automation for business approach.
Why agents need more oversight than other AI
In a fixed workflow with an AI step, the path is yours: the model reads or classifies, and rules decide what happens next. An agent chooses its own tools and sequence, so its errors can compound across steps; a wrong lookup in step two shapes everything after. It also has tools that change things, and it reads content, such as emails and documents, that may contain instructions meant to steer it. The OWASP Top 10 for Large Language Model Applications lists prompt injection and excessive agency among its key risks. The practical answer is not to avoid agents but to bound them: limited tools, gates on writes, and a record of everything.
Four review patterns
| Pattern | How it works | Suits | Cost |
|---|
| Approve before acting | The agent prepares an action; a person approves, edits or rejects before anything happens | Writes, sends, payments and anything irreversible | Highest review effort; lowest risk |
| Review after acting, with an undo window | The agent acts; a person can review and reverse within a set period | Low-risk, reversible actions such as tagging or drafting internal notes | Medium effort; needs a real undo |
| Exception only | The agent acts on its own above a confidence threshold and sends the rest to a person | High-volume, well-measured tasks | Low effort once thresholds are calibrated |
| Sampled audit | Everything runs, and a random sample is reviewed to detect drift | Mature, stable tasks | Lowest effort; detects problems late |
Most real agents use more than one pattern: approval before any write, exception routing for uncertain reads, and a sampled audit of routine outputs to catch silent drift.
Match oversight to the action
The simplest policy classifies each tool the agent can use by what it can do, and assigns oversight to the class.
| Action class | Examples | Default oversight |
|---|
| Read only | Look up an order, search a knowledge base, read a ledger | Automatic; log the access |
| Draft, internal | Prepare a note, a summary or a draft reply for a person | Automatic; a person uses or discards |
| Reversible write | Change a status, add a tag, create a task | Automatic with an undo window, or approval if the record is sensitive |
| External or hard to reverse | Send an email to a customer, update a customer-facing record, create a supplier | Approval before acting |
| Financial or legal | Release a payment, approve a credit, sign or submit a filing | Approval by a named role, and for larger values two approvers |
| Destructive | Delete records, close accounts | Not given to the agent |
Write the table as configuration, not as text in a prompt. A model can be argued out of a rule in a prompt; it cannot call a tool the system never exposes, and it cannot skip an approval step that the workflow enforces.
Confidence thresholds: what they are and how to set them
A confidence value tells the system how sure the AI is about an extraction, classification or proposed action. Treat it carefully.
A model’s own stated confidence is not automatically reliable. Better signals include a retrieval score, agreement across repeated runs, whether the output passes validation (a total that matches line items, an account code that exists), and a classifier calibrated on your own data. Whichever you use, check it against your evaluation set: among items scored 0.9 and above, how many were actually right? The test-set method is in how to evaluate an AI pilot.
Choose the threshold by cost, not by feel. Two costs compete: the cost of a person reviewing an item, and the cost of an error that slips through on an automatic item. Raise the threshold and more items go to review (higher review cost) while fewer errors pass (lower error cost). The best setting minimises the total.
A worked example (illustrative numbers)
Suppose 1,000 items, a review cost of 1.50 per item, and a cost of 25 for each error that goes through uncaught. The threshold results below are invented to show the calculation. In practice you read them off your own evaluation set.
| Threshold | Share handled automatically | Error rate among automatic items | Review cost | Error cost | Total per 1,000 items |
|---|
| 0.95 | 55% (550 items) | 0.5% (2.75 errors) | 450 × 1.50 = 675.00 | 2.75 × 25 = 68.75 | 743.75 |
| 0.85 | 78% (780 items) | 2% (15.6 errors) | 220 × 1.50 = 330.00 | 15.6 × 25 = 390.00 | 720.00 |
| 0.70 | 90% (900 items) | 5% (45 errors) | 100 × 1.50 = 150.00 | 45 × 25 = 1,125.00 | 1,275.00 |
The middle threshold is cheapest overall, but only just. Three cautions follow. First, the 0.95 and 0.85 totals are close, so a different error cost could flip the answer; test that. Second, the average hides the worst case: if one error type is far more costly, give that type its own, stricter threshold, or send it to review always. Third, thresholds drift as inputs change, so re-run the calibration after any model, prompt or document change, and track the override rate in production.
Escalation: who is asked, how fast, and what happens if nobody answers
Escalation is an operational design problem as much as an AI one.
- Route by reason, not just by queue. Low confidence goes to the process team; a policy exception goes to the policy owner; a suspected injection or data problem goes to security.
- Set a time limit for each route. Reviewers need a service level, and the system needs a timer.
- Fail closed. If no one responds, the item waits, escalates to a manager or is declined. It must never proceed because silence looks like consent.
- Cover absence. Named delegates and rota rules apply to review queues as they do to approvals; the patterns are in the approval workflow design guide.
- Have a kill switch. Someone with authority must be able to pause the agent, one tool or one route immediately, without a deployment.
- Limit blast radius. Cap actions per hour, values per action and records touched per run, so that a fault is bounded before a person notices.
Design the review screen for the reviewer
Review quality depends on the screen. A reviewer who has to open five tabs will approve everything.
Show, on one view: the request or document, the sources the agent used (with links), the action it proposes, what will change if approved (as a before/after), the confidence or validation results, and anything the agent was unsure of. Offer three actions: approve, edit then approve, and reject with a reason. Capture the edit and the reason, because each correction is a new test case. Feeding them back into the evaluation set is how the system improves safely.
Watch for rubber-stamping. If approval rates sit at 100% and decisions take two seconds, the gate is decoration. Counter it by mixing in known test items, sampling decisions for a second review, tracking time per decision, and keeping the review queue small enough to be real.
What to log
A log is useful only if it can reconstruct a decision. For each run, record:
- the trigger and the identity of the requester;
- the inputs the agent saw, or references to them where the content is sensitive;
- the sources it retrieved, and the versions of the prompt, the model and the tool definitions;
- each tool call with its arguments and result;
- the proposed action and the confidence or validation outcome;
- the reviewer, the decision, any edit and the reason;
- the final outcome and the time each step took.
Store logs apart from the application, protect them from editing, and apply retention and access rules, because logs can contain personal or confidential data. Redact what the log does not need. With logs of this kind you can answer "why did it do that?" by replaying the run, and you can compute the metrics that tell you whether oversight is working.
Permissions and guardrails that sit under the human
Human review is the last gate, not the only one.
- Least privilege. Give the agent only the tools and the data scope the task needs, using a service account with its own limited rights.
- Treat content as data. Emails, documents and web pages the agent reads are untrusted input. They should never be able to change its instructions or widen its permissions.
- Validate outputs before acting. Check structured output against a schema and against business rules.
- Allow-list destinations. If it can send email or call an API, limit who and what it can reach.
- Keep secrets out of prompts. Credentials belong in the tool layer.
Moving from suggestion to autonomy in stages
Autonomy should be earned task by task, not granted for the whole agent.
| Stage | What the agent does | Promotion test |
|---|
| Shadow | Runs alongside people; its proposals are logged but not shown | Agreement with human outcomes on the evaluation set meets the threshold agreed in advance |
| Suggest | Proposals shown to staff, who act themselves | Staff accept most suggestions and corrections are falling |
| Approve | Agent prepares and executes after a person approves | Approval rate and edit rate are stable; no serious misses |
| Exception only | Acts alone above threshold; sends the rest to review | Error rate on automatic items is within the agreed limit for a set period |
| Sampled audit | Runs with periodic review of a sample | Drift monitoring is in place and reviewed |
A task can sit at different stages: drafting a reply might be at exception-only while sending it remains at approve. Each promotion is a recorded decision by the process owner, not a quiet change in a configuration file. The same staged approach applies to document processing, as described in IDP rollout stages for a large enterprise.
Metrics that show oversight is working
- Share of items escalated to review, by reason.
- Override and edit rate of reviewers.
- Time to decision against the service level.
- Errors found in a sample of automatic items.
- Errors found after the fact, such as customer complaints or reversals.
- Cost per item including review time.
Track them weekly in the first months. A falling edit rate with stable sampled-error rates is the evidence for promoting a task; a rising edit rate is the early warning that inputs or models have changed.
Timeline Digital is a custom software development company founded in 2013 and headquartered in Islamabad, Pakistan. This guide was written by Usama Asif. Our agent builds treat approval gates, thresholds, escalation routes and logs as part of the requirements agreed in the Plan step, and the free pilot lets your team see the review experience on your own data before the full build. To discuss a specific agent, use the contact page.