Scovai Scovai
AI & Operations 2026-11-16 1 min read

The Same 15 Minutes of AI Access, Moved to After the Screen, Bought Teams Nothing. Ten Frontier Models Could Not Tell the Difference.

DSL

Dr. Sarah Liu

The Same 15 Minutes of AI Access, Moved to After the Screen, Bought Teams Nothing. Ten Frontier Models Could Not Tell the Difference.

Same model. Same fifteen minutes. Same task. Move that window from the hour's first half to its second, and expert-rated creativity falls by 0.62 standard deviations (Huang, Yu, Chen & Zhou, 2026). Nothing about the tool changed โ€” only when the team was allowed to touch it. That finding makes AI access timing a configuration decision with a measurable price tag, and in most companies it is a decision nobody has made: the assistant is simply on, all the time, for everyone.

The study is a preregistered laboratory experiment with 148 three-person teams (444 participants), randomly assigned to five schedules of access to the same language model during a 40-minute product-design task. The task had a 15-minute search stage, a 15-minute selection stage, and a development stage. The decision gate โ€” the authors call it the screen โ€” fell at minute 15 in every condition.

What the experiment actually moved

Five conditions: the assistant during search only, during selection only, throughout, never, and during search with a standing overreliance warning. The platform enforced each window, and every one of the 2,124 logged queries landed inside the window it was assigned to. The assistant was a single model (DeepSeek V3.2) at temperature 0.1. Four raters with five to ten years of industry experience scored every submission blind.

The headline contrast is early-search access against later-selection access: 0.356 points on the rating scale, 95% CI [0.064, 0.648], Welch p = 0.018, which the authors report as 0.62 standard deviations. It survives their preregistered Benjamini-Hochberg correction across all eight contrasts.

Against the no-AI control, the pattern sharpens:

ConditionGain vs. no AIp-value
Access during search+0.486 points0.002
Access throughout+0.479 points0.002
Access during selection only+0.130 points0.43

Two things in that table deserve a second read. First, late access produced no measurable gain โ€” but it did not make teams worse than having no AI at all. The authors are explicit that they find no direct evidence of a penalty; the cost is a forgone premium, not damage. Second, adding selection-stage access on top of search-stage access bought nothing: continuous versus early-only is โˆ’0.007 points. The entire return sat in the first window.

Why AI access timing decides the return

The platform logs locate the mechanism precisely, and it is not about effort or usage rates. Teams used the model at similar rates across conditions.

Teams with early access arrived at the gate holding 19.8 distinct written ideas, against 3.9 for teams with no AI โ€” a pool five times larger. Most of it was machine-derived (18.6 ideas), and the teams' own drafted ideas actually fell from 3.9 to 1.3. The assistant displaced some of their own ideation while multiplying the pool. Then they pruned: about a third of the pooled ideas (32%) reappeared in the submission, the same share as teams without AI, and submissions carried 2.9 proposals.

That is the whole story. A wide pool, then a human screen, then a narrow commitment.

Teams with late access reached the same gate with 3.8 ideas โ€” statistically identical to the control's 3.9, exactly as randomisation implies. Then they were handed the model, and they did not use it to choose. They used it to search again. Holding the window, the system prompt and the model fixed โ€” late access versus continuous access, the clean comparison โ€” late-access teams devoted 29% of their selection-stage queries to generating new options, against 11% for teams that had already had the assistant during search.

The consequence shows up in what survived. Only 5% of late-access teams' submitted text existed before the gate, compared with 22% for teams without AI and 71% for teams with early access. Late options did not get added to the screened set. They replaced it. Sentence-embedding similarity gives the same pattern, so this is not the assistant rewording what teams already had.

Options that never met the team

One detail is worth flagging for anyone running a review process. Among late-access teams, machine-derived material entered submissions without ever being shared with teammates at 9.6%, against 3.9% under early access. When the authors hold request type fixed, most of the differences between the two conditions shrink and lose significance โ€” but unshared adoption survives. Late in a task, with a decision pending, output goes straight into the deliverable without passing a second pair of eyes.

That is the same failure the verification tax measures in the field: 52% of workers regularly correct AI-generated work produced by colleagues, and 42% spend more time verifying AI output than they save by using it (Adaptavist, 2026). Unscreened output does not disappear. It becomes someone else's screening problem, one step downstream.

Late access bought breadth, and breadth is not quality

Late-access teams were not idle. They produced more. Proposals per submission rose by 1.53 over the control (SE 0.54, p = 0.005) โ€” an increase of 73% over the control mean of 2.10 proposals, and twice the increase under early access.

The human raters did not reward it. Across teams, each additional proposal was associated with a 0.179-point lower rating. More options arrived and quality per option fell, which is what you would expect when a fixed development budget gets spread across more candidates.

This is the shape of the problem for an operator: late AI access reliably increases visible output and reliably fails to increase assessed quality. If your dashboard counts artefacts, late access looks like a win.

The honest counter: this finding has a scoring rule inside it

The authors run the counter themselves, and it is the most useful part of the paper. Credit breadth, and the verdict moves.

Add ฮณ points per proposal to each team's score, and the early-over-late advantage falls from 0.356 at ฮณ = 0 to 0.279 at ฮณ = 0.1, to 0.203 at ฮณ = 0.2 (p = 0.076), and vanishes entirely at ฮณ = 0.46. At a credit of just 0.1 points per proposal, late access outscores the no-AI control. Early access also leads by 0.356 on the mean proposal, by 0.221 on the best proposal, and it reverses on the total โ€” late-access teams sum to more points.

So the honest statement is conditional: if what you need is one strong commitment, give the assistant to the team before the screen. If what you need is a wide slate of options to carry forward, late access is not a failure mode. The error is not choosing. An always-on assistant sets this parameter by default, and the default is the expensive one for commitment work.

Three further limits belong in any internal summary of this study. The participants were 444 university students in China, mean age 21.6, on a 40-minute structured task with a single model โ€” the authors state plainly that their design "trades breadth of setting for clean identification." The design was powered for d โ‰ˆ 0.74 and found g = 0.62, so this is not a tightly estimated effect, even though it survives correction. And the early-versus-late comparison carries a prompt confound the authors disclose: the system prompt was set by stage, so the search-stage assistant was told to generate and the selection-stage assistant was told to refine. The comparison that holds prompt, window and model fixed is late access versus continuous access โ€” which is the 29%-versus-11% result, and it points the same way.

Your AI evaluator will not catch this

Here is the finding that should change a process rather than an opinion.

Every proposal was also scored by ten frontier language models from five model families, using the same rubric as the human experts. The panel was internally consistent and its team scores correlated 0.61 with the human panel's. It correctly detected the gross benefit of AI access, placing every access condition 0.58 to 1.41 standard deviations above the control.

It could not see the timing cost at all. The early-versus-late contrast was 0.59 standard deviations under human ratings (p = 0.019) and โˆ’0.08 under the model panel (p = 0.77). Excluding the model closest to the assistant leaves it at โˆ’0.10; excluding both DeepSeek models leaves it at โˆ’0.08, so this is not self-preference. The blindness appears on every dimension โ€” originality, appropriateness, elaboration, text quality โ€” where the human experts saw gaps of 0.5 to 0.7 SD.

Part of the explanation is mechanical. Human raters docked 0.179 points per additional proposal; the model panel docked 0.063. The panel did not price breadth the way the experts did, and breadth is exactly what late access produced.

The authors' own sentence is the one to carry into a steering meeting: an organisation that evaluates AI-assisted work with AI judges "will see that assistance helps but not that its timing matters."

This is not an isolated result about one rubric. Evaluation layers are measurement instruments with their own properties, and they drift: in a longitudinal study of agent output review, a stricter reviewer joining in May scored identical batches 0.79 SD lower and flattened a real 0.53 SD improvement trend in the raw data (Agent evaluation rater effects, 2026). Whether your judge is human or a model, it needs calibrating against something before you let it grade a deployment.

There is a parallel worth noting on the capability side. In a three-month randomised trial with 133 patent lawyers, AI access raised delivered work quality by 0.38 SD at 90 days โ€” and when the tool was removed, the retained advantage was concentrated among senior lawyers at 0.45 SD while juniors showed no average gain (Autor et al., NBER w35720, 2026). Delivered quality and underlying judgment came apart. The timing study is the same lesson one level up: the artefact looks fine while the process that should have produced it did not run.

What a Head of Operations should do about AI access timing this quarter

Four concrete moves, in order of cost.

  1. Find your screens and write them down. For your two or three highest-stakes recurring workflows, name the moment where the option set closes and commitment begins. Most teams cannot point to it, which is the precondition for scheduling access wrongly.
  2. Decide, per workflow, whether you are buying depth or breadth. This is the ฮณ question made concrete. Commitment work โ€” one proposal, one vendor, one plan โ€” wants the assistant before the screen. Slate work wants it after. Write the answer into the workflow definition rather than leaving it to whoever opens a chat window.
  3. Constrain post-screen AI use to development, not generation. The 29%-versus-11% split is the measurable failure. Where the assistant's first contribution necessarily arrives late, require that late output be compared against what has already been chosen instead of competing with it unscreened.
  4. Validate your evaluator before you trust a verdict. Score one batch with both your automated judge and two experienced humans, and compare the contrasts you actually care about โ€” not just overall agreement. A panel that correlates 0.61 on levels can still return โˆ’0.08 on the comparison that matters.

The third and fourth points are configuration questions, not training questions, and they are the reason per-task governance beats a blanket policy. Where that work happens on a platform rather than in a document, the relevant controls are the ones that sit at the level of the individual action: Scovai's Digital Employees are confirm-first by default, with autonomy earned per kind of action from a logged approval ledger and occupancy set per seat, so "the assistant may develop a chosen option but may not open a new one after the gate" is an enforced setting with an audit trail rather than a line in a handbook that nobody reads at minute 22.

Note what none of this requires. No new tool, no new budget line, no model migration. The experiment changed one parameter that your company is already setting by accident, every day, in every workflow where somebody has a chat window open.

Pick your single highest-stakes recurring decision this quarter and ask one question about it: does the AI arrive before the screen, or after it? If nobody can answer, you are not running an AI pilot. You are running the expensive condition by default โ€” and if an AI judge is grading it, nothing in your reporting will ever tell you.

Ready to go beyond the CV?

Scovai's AI-powered Talent Passport reveals what resumes can't: personality, potential, and true job fit.