If you spend ten minutes on tech Twitter, you’d think we were about twenty minutes away from mass economic obsolescence. Founders are bragging about their “fully automated” startups, venture capitalists are warning liberal arts majors that their degrees are now decorative toilet paper, and lab researchers are taking victory laps because their latest model scored a 94.7% on some incomprehensible acronym of an evaluation.
It’s intoxicating. It’s terrifying. And, as it turns out, it’s mostly complete bullshit.
A fascinating piece of research from Mercor mapped actual US labor data against existing AI benchmarks. Their finding should make every executive who just replaced their support staff with a brittle chatbot sweat through their Patagonia fleece: roughly 90% of all US working hours are spent on tasks that are not covered well—or covered at all—by current benchmarks.
Ninety percent.
Read that again, let the venture hype drain out of your ears, and look at the reality. The entire multi-trillion-dollar narrative of automated cognitive replacement is being measured against a sliver of economic activity so thin it barely qualifies as a rounding error.
We’ve built an entire industry around gaming standardized tests, and then we’re acting shocked when the student can’t figure out how to operate a mop bucket in the real world.
Here’s the open secret nobody in Silicon Valley wants to admit on a podcast: AI benchmarks are synthetic, sterile, and deliberately isolated. They exist because they’re easy to score. They test multiple-choice trivia (MMLU), canned algorithmic coding puzzles (HumanEval), or neatly framed mathematical proofs. They hand a model a clean prompt, with a clean context window, with a pre-determined right answer, and they celebrate when the model spits out the correct token.
Real work isn’t an AP History exam. Real work is a toxic swamp of undocumented context, fragmented incentives, and bureaucratic rot.
A nurse doesn’t spend eight hours a day answering multiple-choice clinical questions; she spends it de-escalating a panicked family member while battling a malfunctioning electronic health records system that hasn’t been updated since 2004. A junior attorney doesn’t just “summarize contracts”; they navigate the unspoken politics of a senior partner who refuses to read bullet points and wants documents formatted in a very specific, superstitious way. A logistics coordinator doesn’t solve high-level optimization math; they call a stubborn warehouse manager in Ohio who refuses to pick up the phone because he’s on his smoke break.
That is the 90%. It’s the messy, relational, edge-case-ridden friction that makes up the actual machinery of the economy. And right now, the benchmark suites treat that entire reality like it doesn’t exist.
Why? Because the labs can’t measure it. And what you can’t measure, you can’t put on a splashy slide deck to justify a $20 billion compute cluster.
So instead, we get Goodhart’s Law on industrial-grade steroids. AI labs optimize for what shows up on the leaderboard. If a new benchmark measures Python function generation, they dump millions into synthetic coding data. Then they hold a press conference declaring that software engineering is dead. Meanwhile, actual engineers spend 15% of their day writing fresh code and 85% of their day untangling an unholy spaghetti architecture left behind by a guy who got fired four years ago, trying not to break a legacy production database that lacks backups.
The tech industry has mistaken the ability to ace a test for the ability to do a job. It’s the classic Ivy League mistake, just repackaged into a matrix multiplication algorithm.
We keep wondering why the massive enterprise “productivity boom” hasn’t shown up in the macroeconomic numbers despite two straight years of breathless hype. This is why. Companies buy the software assuming the 95% benchmark score translates to real-world competence. Then they drop the tool into a real workflow, watch it hallucinate in the face of slightly ambiguous instructions, and realize they have to hire two humans just to supervise the machine that was supposed to replace three.
We haven’t built artificial general intelligence. We’ve built the world’s most articulate, overconfident intern who tests brilliantly on standardized exams but completely freezes the second you ask them to troubleshoot an unformatted spreadsheet while someone yells at them over Slack.
Hot take: The AI labs aren’t failing to benchmark real work because it’s hard; they’re avoiding it because measuring the messy 90% would instantly pop the valuation bubble. As soon as the market realizes these models aren’t replacing the labor force—just mildly speeding up the 10% of our jobs that were already boring—the trillion-dollar music stops.