Anthropic, the AI company behind the Claude model, has called on the US government to prevent the release of dangerously capable artificial intelligence systems — but the company also warned that the wrong kind of regulation could hand more power to the biggest tech firms and crush smaller competitors.
Why Anthropic is asking for a block
The company argues that certain AI models pose risks severe enough to warrant government intervention before they reach the public. In a new policy document, Anthropic said the US should establish clear criteria for identifying dangerous models and block their release unless developers can prove they are safe.
The proposal would give regulators veto power over the most advanced systems, a step Anthropic describes as necessary to prevent catastrophic harms. The company has long warned that future AI systems could be used to accelerate cyberattacks, design bioweapons, or spread disinformation at an unprecedented scale.
But the same document also lays out what the company sees as serious downsides to broad restrictions.
The risk of centralizing power
Anthropic cautioned that poorly designed regulation could entrench the dominance of a few large technology companies. Big firms with deep pockets can afford to hire armies of lawyers and compliance officers, while startups and academic labs would struggle to meet the same requirements.
“Regulation that is too heavy-handed could make it nearly impossible for smaller players to compete,” the company wrote. “That would concentrate control over AI in the hands of a few giant corporations, which is itself a risk to safety and democracy.”
The warning reflects a tension at the heart of the current debate over AI governance. Many policymakers want to act quickly to prevent harm, but they also worry about stifling a technology that could boost the economy and solve hard problems.
Anthropic’s position is notable because it is both pushing for tougher oversight and cautioning against overreach. The company, co-founded by former OpenAI employees, has positioned itself as a safety-conscious alternative in the industry.
Compliance costs and the innovation gap
The document estimates that compliance with a comprehensive AI licensing regime could run into the tens of millions of dollars per model. For a venture-backed startup, that sum might mean the difference between launching and folding. Anthropic itself has raised billions of dollars, but it says the broader ecosystem needs room for new entrants.
“If every AI developer has to pay millions just to get a license, you won't see many new ideas,” one section of the paper reads. The company argues that regulators should focus on the highest-risk systems and leave less powerful models relatively unregulated.
Critics of heavy regulation have made similar arguments. But Anthropic’s endorsement of a block on the most dangerous models gives the idea a different weight — coming from a company that builds the very systems it wants controlled.
Next steps for policymakers
Anthropic’s submission is part of a broader consultation by the US government on AI regulation. Lawmakers and federal agencies are still weighing how to handle the fast-moving technology. Several bills have been introduced in Congress, but none have passed.
The company’s proposal includes specific thresholds for what counts as a dangerous model, based on benchmarks for capabilities like autonomous hacking or biological threat creation. It suggests that any model exceeding those benchmarks should require a government-issued license before being deployed.
Whether regulators will adopt such a tiered approach — and whether they can avoid the centralization risks Anthropic warns about — remains an open question.
