Lesson 5.4
Human-in-the-Loop and Oversight Design
How to decide where humans belong in a Claude system, design approval and review mechanisms that people actually use well, and avoid the failure modes of rubber-stamping and oversight that nobody can staff.
Human oversight is the control every stakeholder asks for and the one most often designed badly. A confirmation dialog on every action produces fatigue and rubber-stamping; no confirmation at all produces incidents. The professional architect decides oversight per action based on reversibility and impact, designs the human's experience so that judgement is actually exercised, and uses the human's decisions as training data for the system.
Three postures
| Posture | Human role | Fits |
|---|---|---|
| Human-in-the-loop | Approves each consequential action or output before it takes effect | Irreversible, high-impact, regulated decisions; early-stage deployments |
| Human-on-the-loop | Monitors, samples and intervenes; system acts autonomously within bounds | Reversible actions with monitoring; mature systems with good evals |
| Human-out-of-the-loop | Audits after the fact | Low-impact, high-volume, easily reversed actions |
Most systems mix postures per action type. The decision is made action by action, not system by system.
Deciding per action
Score each action the system can take on reversibility, impact and confidence:
- Reversible, low impact (draft a reply, create a ticket in draft, run a read query): autonomous, audited.
- Reversible, higher impact (update a record, post an internal comment): autonomous with monitoring and easy undo, or batched for review.
- Irreversible or external (send to a customer, pay, delete, deploy, share outside the organisation): approval before action.
- Low confidence on anything: route to a human regardless of impact.
Confidence comes from the model's own signal where you have calibrated it (a structured confidence field, a grader score) and from structural signals (the action is unusual for this route, the amount is above a threshold, the recipient is new). Regulation may mandate human review for certain decisions irrespective of your scoring.
Mechanisms
- Approval gates on tools. In the Agent SDK a permission callback or pre-tool-use hook pauses the run and asks; in Managed Agents permission policies can require confirmation for named tools or let the platform evaluate risk and pause when uncertain; in your own loop the dispatcher holds the call until approved. The proposed action, its arguments and the model's stated reasoning go to the reviewer.
- Review queues. For batch and asynchronous work, outputs land in a queue with the evidence attached; reviewers approve, edit or reject. Edits are gold data.
- Escalation paths. The model, or a hook, can route a case to a human when it detects ambiguity, policy edge cases or user distress. Define the path and the SLA.
- Kill switches. Operators can stop a session, a route or the whole system without a deployment.
Designing against automation bias
Humans asked to approve many correct suggestions start approving without reading. Counter it:
- Make confirmations rare. Only consequential actions prompt; routine ones do not.
- Show evidence, not conclusions. Present the source passages, the tool results and the proposed change side by side, so that checking is easier than trusting.
- Add friction proportional to risk. Typing a recipient's name or an amount to confirm a payment is reasonable; a single click is not.
- Seed audits. Inject known-wrong items into review queues occasionally and measure whether reviewers catch them.
- Rotate and limit. Reviewers' accuracy drops with volume; cap queue size per person per shift.
- Measure override rates. A reviewer who never rejects is not reviewing.
Explainability for reviewers
Reviewers need to understand why the system proposes an action. Have the model produce a short, structured justification with citations to evidence as part of the output contract. Do not present thinking blocks as the explanation; they are not a reliable account of the reasoning and are often omitted or summarised. Log the justification with the decision so that audits can reconstruct the case.
Staffing and SLAs
Oversight that requires humans needs humans. Estimate volumes, approval latency targets and reviewer capacity before committing to in-the-loop designs. Define what happens when no reviewer is available: hold the action, fall back to a safe default, or escalate. A system whose approvals time out into automatic approval has no oversight.
Closing the loop
Every human decision is a label. Record approvals, rejections and edits with the case, feed them into the evaluation dataset, and use them to calibrate confidence routing and model graders. Over time this is how a route earns a move from in-the-loop to on-the-loop: the data shows the human rarely changes the outcome, and the regression suite proves the behaviour is stable.
Exam focus
- Oversight posture is decided per action by reversibility, impact and confidence, not once for the whole system.
- Irreversible or external actions require approval before execution; low confidence routes to a human regardless of impact.
- Mechanisms: tool-level approval gates (SDK callbacks and hooks, Managed Agents permission policies), review queues, escalation paths, kill switches.
- Design against automation bias with rare prompts, evidence-first displays, risk-proportional friction, seeded audits and override-rate monitoring.
- Reviewer explanations come from structured justifications with evidence, not from thinking blocks.
- Human decisions are labels; use them to calibrate routing and to justify relaxing oversight over time.
Check yourself
5 questions
1.A finance-operations agent can look up invoices, draft payment proposals, and execute payments. The team wants human oversight without slowing routine work to a crawl. Which design is MOST appropriate?
2.Six months after launching an agent with a human approval step, an audit finds that reviewers approve 99.6% of requests within three seconds, including several that were later found to be erroneous. Which conclusion and corrective action is MOST appropriate?
3.An architect must implement an approval gate for an agent's high-risk tool calls. Approvers may take minutes or hours to respond. Which implementation is MOST appropriate?
4.A regulator asks a lender to demonstrate, for a given automated credit decision, who was accountable, what information the system used, and whether a human reviewed it. Which design feature MOST directly supports this?
5.A fleet of autonomous agents processes customer requests around the clock. During an incident caused by a bad prompt deployment, agents performed thousands of incorrect actions before the on-call engineer could redeploy. Which control should the architect add?