Three AI assistants gave 189 knowledge workers the same answers. The workers paid them different amounts. In a VR-office experiment in the NordiCHI '26 proceedings, participants gave real money to a male-presenting assistant at a mean of 2.96 CHF, against 2.54 CHF for a functionally identical female-presenting one โ p = .026 (Cuber et al., NordiCHI '26, 2026). Asked whether presentation changed how they rated it, 30 of 34 interviewees said no.
Your AI agent evaluation has the same blind spot. Every pilot scorecard, trust survey and adoption dashboard you run collects the kind of answer that was wrong here โ about the one variable this study shows actually moved money.
The experiment held capability constant on purpose
This is the design detail that makes the result usable rather than merely interesting.
Published on 2 October 2026, the study put each of its 189 participants (191 recruited, 2 excluded after assistant response failures) through tasks in a VR office with three assistants: a text chatbot, a desk robot, and a human-like voice agent presented either as "Johan" or as "Johanna." The assistants were functionally identical โ same underlying system, same answers, same quality of work. Embodiment varied within subjects; gender presentation varied between subjects. Afterwards, participants allocated up to 4 CHF of real money to the assistant, and every franc they gave reduced their own payout.
So the experiment removed the explanation managers reach for first. Nobody paid the human-like agent more because it did better work. It did the same work. The only thing that changed was what it looked and sounded like.
The embodiment effect was larger than the gender effect and far more statistically robust: the human-like agent drew 2.76 CHF (SD 1.18) against 1.98 CHF (SD 1.26) for the robot, W = 1511.5, p < .001, with an effect size of r = .44. The plain text chatbot came last at 1.75 CHF. On a 4 CHF ceiling, the human-like presentation was worth roughly 0.8 CHF of other people's money for producing nothing extra.
The gender finding is narrower than the headlines
Be precise here, because the secondary coverage has not been. Gender presentation moved reward (2.96 vs 2.54, p = .026) and perceived human-likeness (5.09 for Johan vs 4.59 for Johanna, p = .029). It did not reach significance on trust (5.58 vs 5.25, p = .065) or on credit attribution (5.24 vs 5.05, p = .164).
The honest claim is therefore about monetary valuation and perceived human-likeness โ not about trust. If you see "female AI agents are trusted less" attributed to this paper, that is not what it found. You will also see a tidy "10.25%" figure circulating in write-ups of this study; it comes from a university press release and appears nowhere in the paper, and it is not recoverable from the reported means. Cite the means.
One moderation worth noting: the human-like advantage over the robot was stronger among women participants than men (interaction beta = 1.074, z = 3.73, p < .001). The persona effect is not uniform across your workforce, which means it will not average out across a team survey.
What they said and what they paid were different numbers
The interview result is the finding with the shortest path to an operational decision. Thirty of thirty-four participants said gender presentation was irrelevant to how they would trust or reward an assistant. Their own allocations, made with real money, contradicted them.
This is a textbook stated-versus-revealed preference gap, and it was not a case of people lying. They had no particular reason to deceive anyone and no obvious incentive to. They simply did not have accurate access to what was driving their own judgement.
The consequence is blunt: asking your team about agent design returns data with no demonstrated predictive value. In this study, self-report pointed the wrong way on the one variable that measurably moved behaviour.
Your AI agent evaluation is partly scoring the persona
Run this forward into a mid-market operations review and the problem stops being about avatars.
You are piloting two agents in different functions. One is a polished voice interface with a name and a face; the other is a utilitarian tool that returns text into a ticket queue. At the end of the quarter you compare satisfaction scores, perceived usefulness, team willingness to keep using them, and requests to expand the seat. Then you fund the winner.
What this study says is that some unknown share of that gap is embodiment, not performance. You have run an uncontrolled comparison and read the result as a capability signal. The agent that got funded may simply have been the better-presented one.
That failure mode compounds in a specific way. Presentation is cheap to change and capability is expensive to change, so a vendor โ or an internal team optimising for adoption metrics โ will reliably find it easier to improve the persona than the work. If your scorecard cannot tell those two apart, it will keep rewarding the cheaper one, and your measurement system will have quietly started selecting against actual quality.
This is a sharper version of a pattern already visible elsewhere in agent research. The 2026 Agent Confidence Index, which ranked 101 agentic tasks across 300 technology executives and practitioners, found that confidence tracked task verifiability and business-context completeness rather than model capability โ automated report generation scored 83.5 and boilerplate code 82.5, both of which have a single objective grading metric, while service-mesh configuration sat at 37.5 on the same underlying models (MIT Technology Review Insights, 2026; Microsoft, 2026). Where there is no clean success metric, human judgement fills the vacuum โ and the VR study shows what that judgement attaches to when it has nothing solid to hold.
What the study does not show
Three limits, stated plainly, because an argument this convenient deserves them.
It was a VR setting with a constructed task, not a live deployment of an agent inside a real workflow with real consequences. The stakes were small and one-shot: up to 4 Swiss francs, allocated once, not a recurring budget decision made by someone accountable for the outcome. And the trust and credit differences by gender presentation did not reach significance, so the effect is narrower than a casual reading suggests.
What survives those caveats is the structural point, and it survives intact: when capability was held constant by construction, presentation still moved monetary valuation, and self-report failed to predict it. You do not need the effect size to transfer from a VR lab to your operations floor to act on this. You need only accept that persona is a live variable in your evaluation and that you are currently not controlling for it.
The cost of measuring the wrong thing
The reason to care is budgetary, not ethical.
McKinsey QuantumBlack's August 2026 work on agentic economics found that for an agent performing a customer service task in banking, token costs frequently represent just 20 to 25 percent of variable run costs, while human oversight accounts for 70 to 75 percent (McKinsey QuantumBlack, 2026). Oversight โ how much human review a given agent needs, by whom, how often โ is the dominant line item.
That allocation is set by exactly the judgement this study shows to be contaminated. Decide an agent is trustworthy and you dial oversight down and widen its autonomy. Decide it is unreliable and you keep a reviewer on every run. If some of that confidence is tracking how human the interface feels, you will under-review the charming agent and over-review the useful one โ and you will be wrong on the largest cost in the stack.
What to do before your next agent review
Four changes, all of which can be made this quarter and none of which require new tooling.
Hold presentation constant across anything you intend to compare. If two agents are being evaluated against each other, give them the same interface modality, or accept that the comparison is confounded and stop reporting it as a performance result.
Stop surveying the team about agent design. Thirty of thirty-four participants got their own behaviour wrong. Ask instead about specific, observable events: what did the agent produce, what did you have to fix, how long did the fix take, what did you escalate. Those answers are checkable.
Instrument behaviour rather than sentiment. Error rates caught in review, rework volume, escalation frequency, time from output to accepted deliverable. All of these are indifferent to whether the agent has a face.
Make the quarterly agent review quantitative and signed. An agent is the one subject you can evaluate purely on numbers without the fairness constraints that rightly apply to people โ a periodic scorecard closed by a named supervisor with a written outcome is more defensible for a Digital Employee than it would ever be for a human report, which is why Scovai's Digital Employee seats are reviewed that way rather than by sentiment survey. Whatever system you use, the requirement is the same: a named accountable human, a fixed set of measures, and a decision written down.
The test for whether your AI agent evaluation passes is a single question, and you can answer it today: if the agent you funded last quarter had shipped with a different voice and no name, would your scorecard have produced the same number? If you cannot say yes, you measured the avatar.