Scovai Scovai
AI & Operations 2026-11-12 1 min read

A Stricter Reviewer Joined in May and Your Agent's Quality Trend Went Flat โ€” the Agent Was Improving the Whole Time

DSL

Dr. Sarah Liu

A Stricter Reviewer Joined in May and Your Agent's Quality Trend Went Flat โ€” the Agent Was Improving the Whole Time

Two reviewers graded the same batches of the same deployed AI agent and differed by 0.79 standard deviations on the composite quality index. The largest quality movement any single logged deploy produced over the same six months was 0.20 standard deviations (Zhang & Esposito, 2026). The noise in that AI agent evaluation was roughly four times the size of the signal it was built to detect. When a stricter reviewer joined the rotation in May, the firm's raw quality trend went flat โ€” and the agent had been improving the entire time.

That is not a story about sloppy reviewers. It is a story about what happens when you run a score stream without ever checking whether the scores are comparable to each other.

The study: 2,611 graded interviews, 14 batches, one deployed agent

Liu Zhang and Mark Esposito studied a single firm running a voice-and-video AI agent that conducts live job interviews. From March through August 2026 the firm's own evaluation pipeline scored 2,611 production recordings in 14 batches against a nine-family rubric, with human reviewers deliberately overlapping inside each batch. That overlap is the whole method: it is the only thing that lets you separate how severe a reviewer is from how good the agent actually got (Zhang & Esposito, 2026).

The design matters because it mirrors what mid-market operations teams already do. Two or three people grade agent output on a rubric, the scores go on a dashboard, and somebody reads the line. The only difference is that most teams do not overlap their reviewers โ€” so they have no way to tell the two effects apart.

Here is what the overlap revealed.

Reviewer B โ€” a trainer who joined in May โ€” scored the same batches 0.79 standard deviations (SE 0.05) below reviewer A. Across the nine batches both of them scored, the gap was negative in every single batch, ranging from โˆ’0.90 to โˆ’0.53 SD, with an inverse-variance pooled gap of โˆ’0.77 (SE 0.03). A heterogeneity test rejects exact constancy (Q = 19.0, p = 0.01), so the offset is an approximation rather than a clean constant โ€” but the direction never once reversed.

Strip the reviewer effect out and the picture inverts. The latent quality index rose 0.53 standard deviations, from 0.05 SD on 29 March to 0.58 SD on 10 August. The improvement was real. The flat line was an artefact of who was holding the pen. The latent critical-failure rate tells the same story in a different shape: 2.8% in March, peaking at 6.3% around 9 May, then falling steadily to 2.5% by 10 August (Zhang & Esposito, 2026).

The reviewer gap was larger than anything the agent did

Put the three variance components next to each other, because the ordering is the finding:

  • Reviewer offset: 0.79 SD
  • Interview-to-interview noise: 0.89 SD (case-level)
  • Real batch-to-batch movement: 0.20 SD

All three are from the same dataset (Zhang & Esposito, 2026). Genuine movement in agent quality is the smallest term in the system. A staffing change on the review side was, on its own, large enough to erase six months of real improvement from the raw trend. A team watching that dashboard in June would have concluded the agent had plateaued โ€” and might reasonably have rolled back a deploy that was working.

This is the part worth sitting with. Most agent governance debates are about how much oversight to buy. This result says the prior question is whether the oversight you already bought produces numbers that can be compared to each other across time. If it does not, more of it adds volume, not information.

Stricter on the rubric, more lenient on the deal-breaker

The reversal inside the data is the detail that should make anyone running agent QA uncomfortable. The same reviewer B who graded 0.79 SD harder on the composite index flagged critical failures at 2.0% against reviewer A's 5.0% โ€” a within-batch difference of โˆ’2.2 percentage points in the opposite direction (Zhang & Esposito, 2026).

Severity is not one dial. A reviewer can be tough on graded items and soft on the binary that actually stops a deploy. Which means calibrating your reviewers on the rubric score tells you nothing about whether they agree on what counts as unacceptable. Those are two separate calibration problems, and almost no QA process treats them that way.

Your evaluation has a shelf life, and the honest answer is "between one and twenty-one weeks"

The second half of the paper puts a clock on the evidence. Fitting the rate of unaccounted movement between logged deploys, forecast uncertainty on the index reaches 0.18 SD after four weeks, 0.25 after eight, and 0.30 after twelve. It reaches the size of the largest observed deploy contrast โ€” that 0.20 SD change on 12 April โ€” after about five weeks at the point estimate, six at the pooled rate.

And then the authors do the thing that makes this paper usable: they report the interval. Across the 95% range for the drift rate, that horizon runs between 1 and 21 weeks. Fourteen batches identify a variance loosely, and the estimate is driven by a small number of step-like changes.

Do not quote the five weeks without the interval. The width is the finding. It says that nobody โ€” including the authors โ€” currently knows how fast an agent evaluation goes stale, and that a quarterly sign-off is a bet on the optimistic end of a range that extends to one week.

One independent signal moved in the same direction as the adjusted index: interview-related support tickets fell by about 10% per month relative to technical-issue tickets. That is corroboration from an outcome the evaluation pipeline does not control, which is worth more than another rubric.

The counter-argument, stated properly

This is one deployment. One firm, one agent, fourteen batches, a preliminary working paper that has not been peer reviewed. The authors are explicit: "These data do not identify a causal effect of evaluation on performance." Nothing here shows that running an evaluation program made the agent better.

So the finding is narrower than the headline and more durable than a single case. The claim is not "evaluation improves agents." The claim is that a score stream with uncontrolled rater severity can invert the sign of a real trend โ€” and that is a measurement property, not a property of this firm.

The reflex response is to add reviewers. Theory says be careful there too: stronger auditing deters some misconduct while raising concealment among agents that still violate, so the catch rate can fall as audit volume rises (Gans & Holden, NBER, 2026). Oversight design beats oversight volume in both literatures. Where the human sits is a throughput variable as much as a safety one โ€” scheduled mid-task confirmation points cut task completion time by 13.54% against confirming only at the end, and 81% of participants preferred them (Zhou et al., CHI 2026).

And this is not a transitional problem you can wait out. In a survey of 300 technology executives and practitioners ranking 101 agentic tasks, 59% already plan permanent human oversight, with accountability (48%) and hallucinations (47%) as the stated concerns (MIT Technology Review Insights, 2026). If the human review layer is permanent, the score stream it produces is permanent infrastructure โ€” and nobody treats it that way.

What to fix in your AI agent evaluation before the next deploy

Four changes, in order of cost. The first two are nearly free and come straight out of the paper's method.

1. Overlap reviewers inside every batch. Have at least two people grade a shared subset of the same items in the same period. That one habit is what makes severity separable from real movement; without it, every trend line you draw confounds the two. It costs duplicate grading on a sample, not a second headcount.

2. Calibrate the critical-failure flag separately from the rubric score. They diverged by 2.2 percentage points in the opposite direction here. Agreeing on a composite number is not agreeing on what stops a release.

3. Re-evaluate on deploys, not on the calendar. Evidence currency decays at a rate you have not measured. Tie the evaluation to the change log โ€” every logged deploy gets a fresh read โ€” rather than to a quarterly review date that may already be four weeks past the useful horizon.

4. Corroborate with one operational outcome the evaluation does not control. Tickets, escalations, rework rate, handle time. If the rubric moves and the outcome does not, trust the outcome.

The reason this is not an abstract measurement concern: in any governance model where an agent earns autonomy from its own track record, the score stream is a control surface. If autonomy expands per action type off a calculated ledger of clean approvals, or a seat gets closed out on a periodic supervisor scorecard, then rater drift does not just mislead a dashboard โ€” it decides how much latitude a digital worker gets and whether a human stays in the loop. Where that chain is written down explicitly โ€” confirm-first by default, autonomy granted per action type, a named supervisor above every seat, as in Scovai's Digital Employee model โ€” the dependency is at least visible. Where it is not, the comparability of the underlying scores has quietly stopped being an analytics question and become an accountability one.

The decision for this quarter

You do not need a new tool, a new vendor, or a bigger review budget. You need to know whether the last four scores in your AI agent evaluation were produced by the same standard โ€” and right now, almost certainly, you cannot tell. Not because the process is bad, but because nothing in it was ever designed to answer that question.

So pick the smallest version that answers it. This quarter, have a second reviewer independently grade 10% of the items your primary reviewer already graded, in the same batch, blind. Compare the two distributions, then compare that gap against the movement you have been reporting upward. If the gap approaches the size of the signal, your agent quality trend has been measuring your staffing rota.

The cost is a few hours of duplicate grading. The alternative is continuing to make autonomy, rollback and headcount decisions on a number whose units change when someone goes on leave.

Measure the reviewer before you trust the review.

Ready to go beyond the CV?

Scovai's AI-powered Talent Passport reveals what resumes can't: personality, potential, and true job fit.