An Appeal Process Is Not Error Insurance: Job Seekers Value a Human Reviewer Above Every Procedural Safeguard You Can Bolt On

Put 1,919 US job seekers in front of two AI screening systems and force a choice, and the feature that moves them most is not the error rate. It is whether a human reviewer makes the call. In a preregistered conjoint experiment posted in September, human involvement raised the probability that a screening system was chosen by 0.272 — statistically indistinguishable from the 0.285 gained by cutting wrongful rejections from 30% to 10%, with the difference's confidence interval straddling zero (Wang, Sturgis & de Kadt, 2026).

Read the trade twice. Two-thirds of your false rejections, or a name on the decision. Applicants price them about the same.

That result is interesting. The next one is operationally expensive, because most mid-market reject paths are built on its opposite.

What 1,919 Job Seekers Priced When the Trade Was Forced

The design is why this is worth a quarter of your attention. Respondents each made eight paired-profile choices between AI screening systems — 30,704 profiles in total — with six attributes randomized independently of one another: decision authority, error rate, explanation, opt-out, appeal, and independent bias audit.

Independent randomization is the methodological point. Prior work usually moves procedure and performance together, so it cannot tell whether applicants want an appeal for its own sake or only for the errors it catches. This design separates them, and it was preregistered before the data came in.

The sample is a working one, not a student pool: mean age 36.6, 35.0% unemployed and actively job-seeking, and 74.9% already evaluated by AI at some point in their search.

Here is the price list, in change-in-probability-of-being-chosen (Wang, Sturgis & de Kadt, 2026):

Pooled human involvement beat appeal, the opt-out and the bias audit, each at p < 0.001. Human involvement, appeal and the opt-out each beat the bias audit, again at p < 0.001.

A human on the decision is worth roughly four times an independent bias audit.

The Appeal Did Not Get More Valuable as the System Got Worse

This is the finding that inverts the standard remediation order.

If procedural features are error insurance — if an appeal is valuable because it catches wrongful rejections — then its value must rise as wrongful rejections become more common. The experiment tested exactly that, with a preregistered equivalence bound of ±0.05.

The appeal's value did not rise. Neither did the opt-out's, the bias audit's, or the explanation's. Appeal and the opt-out passed a stricter ±0.03 bound; the bias audit and explanation passed the registered ±0.05. The error-correction prediction was supported for none of them.

Appeal is the sharpest case, because its corrective function is the most direct of the four. Its interaction with the error rate was non-significant with a 90% confidence interval inside ±0.03. The attribute told respondents re-review existed. It said nothing about how well it worked — and that was enough, at every error rate tested.

Human involvement was the one attribute that fell outside the bound, and the authors are careful about it: its advantage was 0.285 at a 10% error rate, 0.289 at 20%, and 0.239 at 30% — non-monotone, scale-sensitive, and not significant under logit (p = 0.21). They explicitly decline to read that as human review losing value as errors mount. Neither should you.

The usable conclusion is narrower and firmer: performance work and procedure work answer different demands, and neither one buys the other. An employer who installs a more accurate model while removing the human from the decision gains on one dimension and loses on the other.

A Bias Audit Is Not Something a Candidate Can Invoke

Look again at the gap between appeal (+0.156) and bias audit (+0.068). The paper's reading of it is the most portable sentence in the study: decision authority settles who makes the call, appeal and opt-out give the applicant something to invoke, and an audit examines outcomes across cases without offering any individual a route.

Those are three different functions, and compliance regimes treat them very differently. New York City's Local Law 144 requires a bias audit plus a notice explaining how to request an alternative process — and the implementing rules do not oblige the employer to actually provide that process (NYC DCWP, 2023). The EU AI Act takes the other route, requiring human oversight of specified recruitment systems under Annex III(4)(a) and Articles 14 and 26(2), while GDPR Article 22 grants a qualified right not to be subject to a solely automated decision, with rights to human intervention and to contest the outcome (EU AI Act, 2024).

If your AI screening stack was built to satisfy the first regime, you have system-level documentation. The 1,919 respondents valued documentation least of everything tested.

Where the Human Reviewer Actually Has to Sit

One more contrast carries real design weight. Respondents preferred "AI suggests, human decides" (+0.287) over "AI screens applicants first, human decides" (+0.257) by 0.030, at p < 0.001 — and that preference did not move with the error rate (change across the range +0.005, 95% CI [−0.026, 0.036]).

The difference between those two levels is small in wording and large in practice: only the first lets a human see the applications the AI would have rejected. Candidates charged a measurable premium for it.

The behavioural data points the same direction. Among US candidates, 38% have withdrawn from a hiring process at least once specifically because it included an AI interview, and the single most-cited abandonment trigger — named by 33% — was a pre-recorded video interview scored by AI with no human present (Greenhouse, 2026).

Do the Features Reinforce Each Other? Only Barely

It would be convenient if a human decider and an appeal multiplied. The evidence for that is thin. Appeal was valued more when a human was already involved — +0.025, 95% CI [0.004, 0.047] — but the same interaction loses significance under a logit specification in the joint model (p = 0.31). The opt-out interaction was effectively zero, at −0.0003.

So treat the attributes as additive, not synergistic. You are not buying a package where one feature upgrades another; you are buying four separate things applicants value at four different prices. The sequencing advice that follows from that is unglamorous: buy the expensive one first.

The Honest Counter

Three limits, stated plainly.

These are stated choices, not applications. A hypothetical paired choice with no status-quo option and no reject-both option is a clean instrument, not a payroll record. And the features were presented as costless — the survey's appeal has no queue, no delay, and no uncertain outcome, which is not the appeal you would actually run.

A nominal human is not a human. The authors flag the obvious failure mode: a reviewer in the loop can defer to the recommendation they are meant to check. Field evidence sharpens that. In a Harvard Business School experiment with 228 evaluators screening 48 real submissions, black-box AI recommendations improved decision quality, while the identical recommendations wrapped in a narrative explanation did not — evaluators disproportionately complied with rejection recommendations, increasing false negatives (Lane et al., HBS, 2026). A human plus a persuasive rationale can be worse than either alone. Candidates want a decider; your funnel needs that decider to actually decide.

Candidates will apply anyway, and that will mislead you. Intention to apply to employers using AI hiring averaged 3.24 on a five-point scale, above belief in the legitimacy of AI hiring at 2.85 (p < 0.001), and 7.8% of respondents combined below-midpoint legitimacy with above-midpoint intention to apply (Wang, Sturgis & de Kadt, 2026). Application volume is an imperfect indicator of whether candidates accept your process as legitimate. A healthy top of funnel is not evidence of consent.

What to Change on the Reject Path Before the Quarter Closes

Four moves, none of them a new budget line.

1. Stop booking the appeals portal as accuracy work. If your remediation plan for candidate-trust complaints is "tune the model, add an appeals form," you have two line items answering one demand and none answering the other. Price them separately, because applicants do.
2. Move the human from supervising to deciding — on the reject side specifically. The version that earns the +0.287 rather than the +0.257 is a human who sees what the model would have rejected, before it is rejected. The mechanic that makes that durable is a routing rule rather than a policy: scores below the statistical floor for their role-and-seniority cohort route automatically to a human reviewer, on a queue the hiring recruiter cannot close themselves. Scovai documents that routing and the weighting behind it publicly on its platform page if you want a worked reference.
3. Never trade re-review for an audit. Adding a system-level audit while removing individual re-review changes what an applicant can do, not only what you can document — and it swaps the attribute they valued at 0.156 for the one they valued at 0.068.
4. Replace application volume as your fairness metric. Instrument withdrawal rate at the AI-screening step and completion rate on AI-scored assessments. Those move when candidates object. Applications do not.

Procedural rights earn their keep on their own terms. They are not a hedge against your error rate, and your error rate is not a substitute for them. A human reviewer is not a safeguard you bolt on once the model proves unreliable — on this evidence it is the thing applicants were pricing all along, and the only feature that nothing else on the list replaced.

So open your rejection path this week and find the step where a person's name appears. If the only name on it belongs to a form, you have documentation — not a decision.