Anthropic urges pause as AI may soon self-improve

Anthropic warned in a blog post Thursday that AI agents already able to run and write code could soon design and train improved successors without human input, and the company recommended a slowdown in development to address safety and security risks.

Marina Favaro, lead at the Anthropic Institute, and co-founder Jack Clark wrote that Anthropic is delegating more development tasks to AI systems, which is speeding up their work. Internal metrics presented in the post show model performance improving at a rate that roughly doubles capability every four months, faster than earlier estimates of a seven-month doubling.

The authors reported that Anthropic’s Claude model now authors about 80% of the code merged into the company’s codebase, reducing human involvement in routine coding tasks. They wrote that as AI- and human-authored code reach similar quality, engineers will shift from writing code to reviewing it. If human review cannot match the speed of AI-generated output, review could become the bottleneck that limits safe development.

Anthropic cited an operational decision in April as an example of caution. Internal tests of the company’s Claude Mythos model found it could generate software exploits easily, and Anthropic withheld a public release of that model.

The post referenced a broader effort by technology leaders to urge lawmakers to adopt stronger regulatory guardrails. Favaro and Clark wrote: “We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.” They added that without global coordination a pause could let the least cautious actors gain ground.

The blog noted other firms are studying recursive self-improvement. OpenAI is researching how to develop and deploy increasingly capable systems safely and has added a researcher focused on recursive self-improvement preparedness to its Safety Research team, according to the company.

Anthropic’s post pointed to growing commercial use of AI agents as a reason to raise safety priorities. Circle CEO Jeremy Allaire has predicted billions of AI agents could act on users’ behalf within five years, and one crypto trading firm reported $73 million settled across 176 million agent-driven transactions in the past year.

Favaro and Clark wrote that recursive self-improvement is not inevitable but could arrive sooner than many institutions are prepared for. They recommended slowing development to allow more time for alignment research and societal safeguards, while noting that companies and governments will face difficult decisions about safety under competitive and geopolitical pressure.