George Hotz, the hacker known for cracking the iPhone and PlayStation 3, spent half a year putting AI coding agents through real-world projects. His verdict: the tools are churning out an avalanche of undetectable slop—and large organizations won't spot the mess until it's too late.
Six months of real testing
Hotz, who now runs the self-driving startup Comma.ai, didn't run benchmark tests or read white papers. He used AI coding agents on actual software tasks over six months. The results, he said in a recent post, paint a grim picture for anyone relying on these tools to produce production-grade code.
The problem isn't that the code doesn't run. It runs. But it's filled with subtle bugs, nonsensical logic, and hidden shortcuts that look reasonable to a reviewer. Hotz called it “slop”—a term he used to describe code that passes surface-level checks but fails under scrutiny.
The undetectable slop problem
What makes this different from earlier code-generation tools, Hotz argued, is the sheer volume. AI agents can produce thousands of lines per hour. A human reviewer can't read every line, so bad code slips through. Over weeks or months, the accumulation of such slop creates a codebase that's fragile and hard to debug.
Hotz didn't name specific agents or companies. He simply warned that the industry is sleepwalking into a crisis. “Large organizations won't realize the problem until it's too late,” he wrote. The warning carries weight because Hotz has a track record of finding flaws others miss—both in hardware and in software.
The implications for companies that have rushed to adopt AI coding agents are uncomfortable. A manager might see productivity metrics going up—more pull requests merged, faster feature delivery. But if the underlying code is slop, that speed is an illusion. Technical debt compounds silently.
Hotz didn't offer a solution. He didn't call for regulation or for abandoning the tools. He simply presented his findings and let the data speak. The question he leaves open: how many teams will realize they have a problem only after a production outage, a security hole, or a failed audit?
For now, the industry has no standard way to measure code quality from AI agents. No widely accepted tests exist to flag slop. That means every team is on its own—trying to figure out whether the code their agent just wrote is solid or secretly broken.
Hotz's test ran for six months. The clock is ticking for everyone else.
