Sit back, relax and enjoy our latest articles...

Sit back, relax and enjoy our latest articles…
What Being Overlooked Actually Looks Like - blog by Jacqueline Frost

We’re About to Automate a Mistake We Already Know About

We're About to Automate a Mistake We Already Know About - blog by Jacqueline Frost

So far in this series I’ve argued three things. That our leadership problem is less about developing people and more about spotting them. That the potential ratings driving promotion decisions have been shown not to predict who actually does well. And that there’s a recognisable group of capable people our criteria aren’t built to see.

All of that has been survivable, in a way, because human judgement is patchy. Some managers look past the surface. Some people get lucky with a sponsor who notices them. The mistakes are real, but they’re scattered – and scattered mistakes leave gaps that people occasionally fall through in the right direction.

That’s now changing, and quite fast.


The decisions are moving

Talent platforms can now scan career histories at enormous scale and predict who has leadership potential from patterns in how careers have unfolded. In other words, they learn what the people we promoted looked like, and look for more of them.

That’s fine if we promoted the right people. But the study I wrote about earlier, nearly 30,000 employees, published this year in the American Economic Review, found the potential ratings behind those promotions didn’t predict who went on to do well.


The problem nobody is talking about

AI career tools also work by asking you to describe yourself: your strengths, your achievements, what you’re ready for next. Then they assess what you’ve written.

Now put that next to our data. Across five years, the share of people who strongly agreed they make sure others know the value they bring never went above 12%. In 2026, just 9% said they actively manage how they’re seen.

And it isn’t only our data. Research in the Quarterly Journal of Economics found women consistently describe equivalent work less favourably than men. The gap stayed when there was no reason to self-promote, and it stayed when confidence was accounted for. It even showed up in schoolchildren.

If a system judges you on how you describe yourself, and you undersell yourself, it will underrate you by exactly that amount. And nobody will see it happen, because the underselling is the data.


Sometimes there’s nothing to read at all

There’s a part of this I only saw clearly recently.

Over the last three years, almost three thousand people started our programme with the same self-assessment. One statement was: “Others speak well of me when I am not in the room.”

Four in ten said they didn’t know. Only 2% said no.

It was a similar story elsewhere. Three in ten didn’t know how well they handle the politics of their organisation. One in five didn’t know whether people follow when they take the lead. And the answers were the same in every group, across different years and different employers.

Yet nine in ten said the people around them see their potential, and 96% said their colleagues trust them. These people know exactly how they’re seen up close. What they don’t know is how they’re talked about in the rooms where decisions get made. That isn’t a confidence issue. Nobody has told them.

So when an AI tool asks someone how senior stakeholders see them, a large share of capable people will give it a hedge or a blank. The organisation has a view. It’s sitting in the talent review. The tool is asking the one person who has never heard it.


Why the usual fixes don’t reach this

The standard safeguards are to check training data for imbalance and strip out names and gender. Both are sensible. Neither touches this. Removing a name doesn’t change how someone described their work or fill in an answer they were never in a position to give.

The serious assessment companies solved part of this years ago, building tests you can’t pass simply by rating yourself generously. The new AI tools being bolted on top haven’t inherited that solution.

To be fair, these tools aren’t broken. One audit of more than 150 AI hiring systems found around 85% met accepted fairness standards, and many are tested more rigorously than the manager nominations they replace. But underselling, or simply not knowing how you’re seen, isn’t something those audits are designed to catch. It isn’t a flaw in the system. It’s a flaw in what we’re feeding it.


Let me be fair about this

I want to be careful here, because “AI is biased” has become a lazy thing to say and I don’t want to add to it.

There is real evidence of problems. One 2025 study tested five major AI models with identical professional profiles that differed only by gender, and found the models advised women to ask for substantially lower salaries – in one case the difference ran to six figures. But the same researchers ran three experiments, and only one produced that result clearly. One showed no meaningful difference, and one went the other way.

There’s also evidence on the other side. One auditing firm reviewing more than 150 AI hiring systems, across over a million test samples, found roughly 85% met accepted fairness standards. That’s not nothing.

So I’m not saying these tools are broken. Many of them are tested more rigorously than the manager nomination process they’re replacing, which has mostly never been tested at all.

What I’m saying is narrower. People underselling themselves, or simply not knowing how they’re seen, isn’t something those audits are designed to catch. It isn’t a flaw in the system. It’s a flaw in what we’re feeding it.


Three questions worth asking

What is it learning from? If it’s your own promotion history, how confident are you that history was right?

Where does self-description feed a decision, and what happens when someone answers modestly, or says “I’m not sure”? That’s where this problem lives.

Has anyone tested whether confident phrasing scores higher than modest phrasing for identical work? Almost nobody is checking. It’s a simple test.


Where this leaves us

None of this is an argument against AI in talent decisions. Done well, it could be fairer than what we have now.

But there’s a difference between building something better and building something faster. Right now we risk taking a judgement we know to be unreliable and making it quick, consistent, and much harder to argue with.

Getting the definition of potential right has been a slow-burning problem for decades. It’s about to become an urgent one.


This is the final piece in this series. If any of it has made you wonder what your own data would show, I’d be glad to hear from you.


Sources

  • Benson, A., Li, D. and Shue, K. (2026) ‘”Potential” and the Gender Promotion Gap’, American Economic Review, 116(2)
  • Exley, C. and Kessler, J. (2022) ‘The Gender Gap in Self-Promotion’, Quarterly Journal of Economics, 137(3)
  • Yamshchikov, I. et al. (2025), study of gender-differentiated salary advice across five large language models
  • Warden AI (2025), audit report covering 150+ AI hiring systems
  • Korn Ferry, Assessment of Leadership Potential technical materials on forced-choice design
  • Elevate Talent participant data, 2016–2026, including EPIC Career Self-Assessment results from nine programme groups, 2022–2025