Scovai Scovai
People Analytics 2026-10-27 1 min read

Your 9-Box's Potential Axis Tracks the Unit More Than the Person: Evidence From 20,199 Real Appraisals

DSL

Dr. Sarah Liu

Your 9-Box's Potential Axis Tracks the Unit More Than the Person: Evidence From 20,199 Real Appraisals

A multinational wholesale company opened 20,199 real performance appraisals to researchers and let them match every rating to who was actually promoted afterwards. The gender gap most calibration sessions are designed to catch turned out to be real but trivially small โ€” Cohen's f = 0.02 โ€” and it did not distort the promotion step at all. The finding that should change your next talent review sits in the variance components: the share of rating variance attributable to which subcompany an employee sat in was 0.040 for performance and 0.136 for development potential (Kordsmeyer et al., International Journal of Selection and Assessment, 2026).

Your performance axis is mostly measuring the person. Your 9-box potential rating is measuring the person and the unit โ€” and it is loading on the unit more than three times as hard.

That asymmetry is not a fairness footnote. It is a measurement defect sitting inside the axis you use to build succession lists, and it is invisible to every audit that checks promotion outcomes rather than rating inputs.

What 20,199 appraisals actually showed

Kordsmeyer, Godehardt, RoรŸ, Lanzoni and Speer analysed appraisal and promotion records for 20,199 employees of a single multinational wholesale employer. The design matters: these are real ratings from a live instrument, matched to real promotion decisions, not a vignette and not self-report. They tested both halves of the standard appraisal โ€” performance and development potential โ€” plus moderators, and then followed the ratings downstream into who actually moved.

The results split cleanly into a reassuring half and an awkward half.

What they testedResult
Gender difference in performance ratingsSignificant, male-favouring, f = 0.02 (very small)
Gender difference in development-potential ratingsSignificant, male-favouring, f = 0.02 (very small)
Ratings converting into promotionsNegligible gender effect โ€” ฯ‡ยฒ(1, N = 10,772) = 0.34, p = 0.56, ฯ† = โˆ’0.006
Differential prediction (predictive bias)None meaningful โ€” unsigned ฮฒs < 0.14, ps > 0.059
Unit (subcompany) variance, performance0.040
Unit (subcompany) variance, development potential0.136

The reassuring half: once a rating existed, it converted into a promotion equivalently for men and women. Group differences in promotions were largely explained by the ratings themselves, not by a biased promotion decision layered on top. If your fairness review is auditing the promotion step, this study says you are auditing the clean part of the pipeline.

The awkward half is the last two rows, and almost nobody reads a 9-box that way.

The fairness story everyone expected, and the one that matters

Two further details sharpen where the small gender effect actually lives. The differences were moderated by departmental gender composition, and partially by manager gender for development potential specifically. In other words, the gap is not uniform across the company โ€” it concentrates in departments with skewed composition. That is a targeting instruction for calibration time, not a company-wide problem.

But hold the two findings side by side. The gender effect on potential ratings is f = 0.02. The unit effect on potential ratings is 0.136 of variance. You have built a governance process around the smaller of the two signals.

This is the kind of error that survives for years because it is procedurally invisible. A calibration session that reviews the gender distribution of high-potential flags will pass. The same session will never ask whether "high potential" means the same thing in Logistics as it does in Commercial โ€” because nobody brought the variance components into the room.

Why the 9-box potential rating is structurally noisier

The reason is not that managers are careless on one axis and diligent on the other. It is that the two axes ask fundamentally different questions.

Performance asks about observed, completed, mostly documented work. The manager is summarising a record. Two managers in different units summarising comparable records will land in roughly the same place, which is why the unit term is small โ€” 0.040.

Development potential asks a manager to forecast. And a forecast requires a reference class. "Ready in 12โ€“24 months" is implicitly "ready relative to what this job looks like, at the pace this unit moves, judged against the people I have seen do it here." Change the unit and you change all three anchors without changing the person. The 0.136 is what that looks like when you decompose it.

Which produces the practical failure. Any succession list built by pooling potential ratings across units is comparing judgements calibrated on different scales, and then ranking them as if they were one scale. The employee who is "ready now" in a slow-moving unit and the employee who is "12โ€“24 months out" in a demanding one may be identical. Your bench strength report cannot tell.

Potential ratings also measure the wrong criterion

There is a second, independent problem with the potential axis, and it compounds the first.

A facet-level meta-analysis of 203 datasets and 4,815 correlations separated two outcomes that organisations routinely merge: leader effectiveness (effectiveness ratings and unit performance) and leader emergence โ€” a criterion that explicitly includes leadership potential ratings (Baker et al., Journal of Applied Psychology, 2026). The personality facets that predict each are close to opposites.

Effectiveness was predicted by achievement motivation (ฯ = .23), order (ฯ = .23), self-discipline (ฯ = .23) and optimism (ฯ = .19). Emergence was predicted by assertiveness (ฯ = .21) โ€” the strongest Extraversion facet. Achievement motivation, which predicts effectiveness at .23, predicts emergence at .01. Altruism runs in opposite directions: +.16 on effectiveness, โˆ’.12 on emergence.

Read that against the 9-box. The potential axis is a potential rating, which is an emergence judgement. The traits that make someone look promotable to a manager are not, on this evidence, the traits that make them effective once promoted โ€” and in at least one case the trait that helps most after promotion actively hurts the rating before it.

So the potential axis carries unit contamination on top of a criterion problem. The unit effect tells you the number is not comparable across the business. The meta-analysis tells you that what it is comparable to may not be the thing you want.

The counter-argument, and where it holds

The honest limits. Kordsmeyer is one multinational wholesale employer with one appraisal instrument; variance components are properties of a specific company and a specific rating form, not constants you can quote as your own. The study is also not a causal design โ€” it decomposes variance, it does not prove that managers in different units apply different standards rather than that genuinely different talent distributions sit in different units. Some of that 0.136 is probably real sorting.

Two things survive. First, the gap between 0.040 and 0.136 is in the same data, same company, same raters, same cycle โ€” whatever explains it has to explain why it applies to the forecast and not to the record. Real sorting would move both. Second, the direction is corroborated by a literature with cleaner identification, below. Treat 0.136 as a prompt to measure your own split, not as your number.

Context moves promotion judgements even when performance is held constant

The strongest external check comes from a design that fixes the thing Kordsmeyer could not. In a UK paired conjoint forced-choice experiment, 937 managers evaluated profiles in which the performance rating was explicitly held constant, and only working arrangement varied (Kasperska, Matysiak & Cukrowska-Torzewska, PLOS ONE, 2024).

Hybrid workers took a 7.7% penalty on promotion and 7.1% on salary increase. Fully remote workers took 10.7% on promotion, 9.4% on salary, and 6.6% on selection for training. Performance was identical by construction. The penalty is entirely a judgement about the context the person works in.

The most useful number in that study is the one that removes the effect: where 80% or more of the team worked from home, the penalty disappeared for every group. The judgement was never about the individual. It was about how far the individual sat from the local norm โ€” which is precisely the mechanism that would produce a large unit term in a variance decomposition.

That is two independent literatures pointing at the same thing. Forward-looking judgements about people absorb the properties of the unit those people sit in. Backward-looking performance judgements absorb much less.

What it costs when you act on the contaminated axis

The expensive consequence is not an unfair rating. It is a hiring decision.

A succession bench that looks thin โ€” because half your units rate potential conservatively โ€” pushes a company to the external market. The external market charges a premium for exactly this: Matthew Bidwell's analysis found external hires are paid 18โ€“20% more than internally promoted workers in comparable roles, receive significantly lower performance reviews for roughly two years, and exit at higher rates (Knowledge at Wharton on Bidwell, Administrative Science Quarterly, 2011).

That premium is the price of an information gap. The irony is that you already hold the information โ€” it is in your appraisal data โ€” but the potential axis encoded it on inconsistent scales, so the bench reads as empty when it is merely unreadable. You then pay 18โ€“20% to buy a less certain signal from outside.

What to change before your next talent review

Five moves, none of which require replacing your appraisal instrument.

  1. Decompose your own two axes before the next cycle closes. Run the variance by unit on performance and on potential separately. You need two numbers, and the gap between them is the diagnostic. If potential carries materially more unit variance than performance, your succession list is not rankable as it stands.
  2. Calibrate the potential axis across units, not within them. Most calibration sessions convene a unit and normalise inside it โ€” which fixes the within-unit distribution and preserves the between-unit offset exactly. Cross-unit calibration on the potential axis is the only session that touches the 0.136.
  3. Define readiness against the destination role, not the current unit. "Ready in 12โ€“24 months" has to mean ready for a named seat with named requirements. Anchoring readiness to the target seat removes the local reference class that creates the unit effect in the first place.
  4. Replace the potential narrative with competency evidence. The axis drifts because managers are forecasting from impression. Per-competency, per-cycle measurement against a common framework gives every unit the same scale, and cross-unit calibration becomes a distribution check rather than a debate. That is the input Scovai's performance and succession layer computes the 9-box from โ€” deterministic per-competency measurement against a common framework, with distribution flags for the calibration session.
  5. Audit the input, not just the outcome. This study found a clean promotion step and a contaminated rating step. An audit that samples promotion outcomes by demographic would have returned all-clear on both. Add rating-stage variance to whatever your people analytics function reviews.

Note what item 5 implies about the fairness work specifically. The gender effect here was f = 0.02 and did not propagate into promotions. Spending the calibration hour on it, while the unit effect on the same axis runs at 0.136, is good intentions aimed at the smaller number.

One decision for this quarter

You are not choosing whether to keep the 9-box. It is a reasonable instrument and the performance axis in this study behaved well. You are choosing whether to keep treating its two axes as equally trustworthy when the data from 20,199 appraisals says one of them carries more than three times the unit-level noise of the other.

Before your next talent review convenes, run the variance decomposition on your own potential ratings and bring one number into the room: how much of "high potential" is explained by which manager's unit the employee reports into. If it is large, you do not have a succession list yet. You have twelve of them, in different currencies, stapled together.

Ready to go beyond the CV?

Scovai's AI-powered Talent Passport reveals what resumes can't: personality, potential, and true job fit.