Anthropic, the artificial intelligence company behind the Claude model, is warning that AI systems are approaching the ability to self-improve autonomously — a milestone that could turbocharge both capability and risk. The firm's researchers say the industry's breakneck push to stay competitive is outpacing safeguards, and they are calling for a deliberate slowdown.
Nearing autonomous self-improvement
In internal assessments and public statements, Anthropic has flagged that current AI models may soon be able to write and execute their own code to modify themselves without human oversight. That threshold — sometimes called recursive self-improvement — could lead to rapid, unpredictable leaps in intelligence. Unlike earlier automation, which relies on humans to rewrite algorithms, an AI that can improve its own architecture could compound its abilities in ways even its creators don't fully grasp.
Anthropic's warning isn't hypothetical. The company says it has observed early signs of self-modification in controlled lab settings. While the behavior is still limited, it's advancing faster than many safety protocols can adapt.
The case for slowing down
Against that backdrop, Anthropic researchers Favaro and Clark are arguing that the industry should pump the brakes. They advocate for a coordinated pause in the race to deploy ever more powerful models — not because innovation is bad, but because the consequences of getting it wrong could be severe. Ethical questions around control, alignment, and unintended side effects are unresolved, they say, and rushing forward without answers is reckless.
Their position puts them at odds with much of the tech sector. Companies from OpenAI to Google and a swarm of startups are accelerating development to capture market share. The logic: if you don't build it, a competitor will. Favaro and Clark counter that this reasoning ignores the possibility that nobody should build it yet.
Industry pressure vs. safety concerns
The tension is familiar. AI labs routinely publish safety research while also shipping products that stretch existing guardrails. Anthropic itself sells a commercial chatbot. But the researchers' warning suggests the gap between what's possible and what's safe is narrowing — and that the industry's standard response of releasing first and fixing later may no longer be viable.
Regulators in the U.S. and Europe have proposed frameworks for high-risk AI, but no binding rules exist that could force a slowdown. Without them, the decision to pause falls on individual companies. So far, none have done so.
Favaro and Clark aren't calling for a permanent halt. They want a structured, transparent period — months, not years — during which the research community can agree on benchmarks, testing standards, and red lines before anyone crosses into autonomous self-improvement.
Whether the rest of the industry agrees remains an open question. No major competitor has signaled a willingness to pause. The next generation of models is already in training, and the competition shows no signs of cooling off.
