I don't care which AI model wins the exam if it still needs a grown-up to check the invoice.
AI benchmarks are controlled tests used to compare models. They're useful. They aren't a job interview for your business, and I think treating them as one is a splendid way to buy the wrong software with impressive supporting paperwork.
Hiring the winner of the decathlon to fix your boiler is a bold interpretation of transferable skills.
I treat the official "best model" story as the sales pitch somebody's paid to deliver, not the conclusion. That doesn't make the underlying research rubbish. It means the jump from a good test result to a sensible purchase deserves rather more attention than the trophy.
Whose exam is it?
Artificial Analysis combines ten evaluations in its Intelligence Index, with weighted categories covering agents, coding, scientific reasoning and general ability. Agents, in this context, are AI systems attempting tasks through several steps, sometimes using other software. Its index is primarily text-based and English-language; image inputs, speech inputs and multilingual performance are tested separately.[1]
That's a useful comparison. It also contains choices about what matters and how much each thing should count.
Your business gets to make different choices. A model's scientific reasoning score won't settle whether it can extract the right delivery conditions from a supplier's awkward PDF. Nor does an overall intelligence score tell you how much correction that particular job will need.
I don't think the answer is to find a more flattering leaderboard. It's to stop confusing someone else's mix of tasks with your own.
Call me cynical, but turning "strong across this test suite" into "the model your company needs" is excellent business for AI labs and software vendors. They get a simple selling point. You inherit the work of finding out whether it applies.
I'm not suggesting the evaluator's rigged anything. I'm questioning the sales leap made after the measurement.
Half right isn't half a colleague
METR, an AI evaluation organisation, measures something called a task-completion time horizon. Its 50% measure estimates the length of tasks, measured in human expert working time, at which an AI agent is predicted to succeed half the time.[2]
It isn't a clock showing how long the machine can work independently. METR explicitly warns against that reading.[3]
That distinction matters. A chart about the difficulty of tasks a system can complete at a particular success rate isn't a promise that it'll take a full shift off your hands.
The tasks are mainly software engineering, machine learning and cybersecurity, with clear instructions and checkable outcomes. METR also makes clear that most jobs aren't composed of tidy tasks like these.[2]
I like that honesty. It's considerably more useful than pretending a research graph comes with payroll savings attached.
At the 50% point, we're talking about succeeding half the time on tasks of that difficulty. That's not the same proposition as reliable delegation. "Excellent candidate, provided we only count the work that went well" belongs in a reference letter written by someone trying to get rid of him.
Give it the awkward file
My preference is a small, dull audition built around work that actually needs doing. Not a theatrical contest to find the cleverest chatbot.
Imagine a business comparing two models for supplier enquiries. Give both the same sample of old, anonymised enquiries and the same approved product information. Include an ordinary question, an ambiguous delivery request and a case where the information simply isn't there. Decide what a usable answer looks like before seeing either response.
The missing-information case matters most to me. A helpful admission that the answer isn't available beats a beautifully worded delivery promise nobody authorised.
Then count the whole job. How long did preparation take? What did the software cost? How much checking and correction was needed? Could the finished answer actually be sent?
Those aren't secondary details spoiling an elegant model comparison. They're the comparison.
METR's own discussion of translating capability into time savings includes the human time spent prompting, waiting and checking output.[3] I think business buyers should be at least as fussy as the researchers whose charts they're borrowing.
A cheaper model that needs less correction on one job could be the better choice for that job. A stronger model might justify its extra cost on another. Neither conclusion requires declaring a permanent winner for all of artificial intelligence.
The valuable business, to my mind, is one that can demonstrate finished work with less human effort, not one that can borrow the shiniest score. I'd rather see an unglamorous comparison survive contact with a messy file than another victory lap around a graph.
A leaderboard can earn a model an interview. It shouldn't sign the employment contract.
Who gets paid when we confuse the two?
Sources
[1] https://artificialanalysis.ai/methodology/intelligence-benchmarking
[2] https://metr.org/time-horizons/
[3] https://metr.org/notes/2026-01-22-time-horizon-limitations/