The Reviewer Who Stopped Reviewing

RBI's draft model risk guidance, released for consultation in June 2026, asks lenders to establish robust human oversight for AI models, including those used in automated decision-making (paragraph 60). Most lenders will meet that expectation the obvious way: a credit officer or operations analyst sees the model's recommendation and approves it before it takes effect.

That step is easy to build and easy to show an examiner. It is also the control most likely to stop working without anyone noticing. An earlier piece, The Handoff Problem, looked at chains of agents where the person moves from between the steps to watching the loop. This piece is about the person who stays in the loop, and why staying there is not enough.

The draft is specific. Oversight arrangements should include human-in-the-loop or human-on-the-loop mechanisms, override and kill-switch arrangements, and periodic human review of model-driven decisions (paragraph 60). They should account for automation bias, over-reliance on model outputs and decision fatigue (paragraph 61). The people doing the oversight should understand how the model works and be able to challenge, override or escalate (paragraph 62). And the arrangements themselves, including decisions, interventions, overrides, incidents and near misses, should be reviewed periodically and strengthened (paragraph 63).

On 19 October 2026 the UK Financial Conduct Authority published what this looks like when it goes wrong. Its good and poor practice paper, drawn from supervising firms rather than from new rules, describes a firm where the reviewer role was unclear and "became a tick-box exercise over time". Reviewers accepted AI recommendations with less scrutiny and challenged less often, even where there were signs of error. Escalation criteria were unclear, so some outputs went through despite low confidence. The paper also names the opposite failure: firms leaning so heavily on human review that it hid weaknesses in the model itself.

Nothing in the FCA's example was missing on paper. There was a reviewer, a review step and a workflow that routed outputs through it. The control existed and had stopped doing anything.

Four forces work on a reviewer, and none of them requires anyone to be careless.

The model is usually right. If it agrees with a competent officer on the large majority of files, agreeing becomes the efficient habit, and each disagreement costs time the officer is measured on.

The reviewer sees the answer first. A recommendation shown before the reviewer has formed a view becomes the starting point, and the review turns into a search for reasons to disagree rather than a judgment of the file.

Nobody wrote down when to stop. Without defined triggers, a borderline output looks much like a clean one, and the default is to let it through.

Nobody measures the reviewer. Model performance is monitored. Whether the person approving the model's outputs still exercises judgment usually is not, so the decay stays invisible until an incident.

Paragraph 61 names these risks. It does not say how to design against them, and an oversight policy that restates paragraph 61 will not change any of the four.

The fix sits in how the work is designed, not in the model. A review step drawn as a box on a process map has no decision rights, no workload, no standard and no measure. A review job has all four: what the reviewer may do (approve, override, hold, escalate), how many cases a day the role can carry while still judging them, what a good decision looks like, and how anyone will know if the role has stopped working.

This is operational redesign, and it usually falls between functions. The business owns throughput, risk owns the policy, and technology owns the workflow. The reviewer's job is designed by nobody unless someone is given that work.

Route by risk, not by default. Decide where a person adds judgment and where they add only delay. Low-exposure, high-confidence cases can proceed with sampled review; higher exposure, low model confidence, missing data and policy exceptions go to a person. Reviewing everything is the surest way to make every review a formality. The FCA lists risk-based placement of human review as good practice.

Blind review where it counts. While a model is being established, and on a continuing sample afterwards, the reviewer decides before seeing the model's recommendation. The FCA describes a firm that did this for affordability assessments, which let it compare human and machine judgments without the reviewer simply deferring to the system. Agreement measured this way is evidence. Agreement measured after the reviewer has seen the answer is not.

Written escalation triggers. Confidence below a set level, a document the model could not read, a borrower flagged as vulnerable, an amount above a limit: each one stops the output and sends it to a named person with a defined next step. The triggers belong to the risk function, and changing them is a controlled change.

Make disagreement cheap. An override should take the same effort as an approval, with a reason code and no extra sign-off. If disagreeing with the model needs a manager and agreeing needs a click, the workflow has priced challenge out of the job.

Measure the reviewer as well as the model. Track override rates, holds, escalations, time per case and agreement on blind samples, by reviewer and by segment. An override rate drifting towards zero while the model has not changed is the signal the FCA's example describes. Paragraph 63 asks for periodic review of decisions, interventions and overrides; that review is only possible if the workflow records them in the first place.

Blind samples, written triggers and realistic caseloads cost reviewer time. A value case that counts the reviewer as free, or assumes each file gets twenty seconds, is assuming the review does not happen. The honest version sizes the review the workflow needs and still clears the bar. If it does not, the workflow was not worth automating at that level of risk, and it is better to learn that before deployment than after an incident.

The design of the review job belongs to the first line, which runs the workflow and owns the outcome. The second line can test it directly: seed the queue with cases where the model is known to be wrong and count how many are caught. That test answers the question an examiner reading paragraph 62 will ask, which is whether the people overseeing the model can actually challenge it.

Pull last month's log for any AI-assisted decision with a human approver. If the override rate is close to zero and nobody can say whether that is because the model is right or because the reviewer has stopped looking, the human in the loop is a signature, not a control.