Recursive Self-Improvement: Anthropic Maps a Risk It Can't Unilaterally Solve

Anthropic published When AI Builds Itself on June 4, focused on recursive self-improvement (RSI)¹. Co-founder Jack Clark wrote in Import AI #455 that the probability of AI autonomously training its own next-generation model exceeds 60% by the end of 2028².

The report also opened a window onto Anthropic’s own operations: more than 80% of merged code is now produced by AI, and engineer quarterly code deployment sits at roughly 8x the 2021–2025 average¹.

Table 1: AI automation inside Anthropic ¹²

Metric	2021–2025 average	Current (May 2026)	Change
Share of merged code from AI	< 10%	> 80%	8x+
Engineer quarterly code deployment	Baseline	8x baseline	8x
Autonomous task duration	4 minutes (Opus 3)	1.5–12 hours (latest)	22–180x

What RSI is

RSI means an AI that designs, trains, and validates its own next-generation system, end to end, with no humans in the loop. Anthropic lays out three scenarios: AI progress stalls; humans stay in charge but AI does most of the work; or the loop actually closes and AI runs the whole R&D pipeline¹.

Anthropic flags the third as the one to watch. The report: “Recursive self-improvement is not yet here, and not inevitable, but it may arrive faster than most institutions are prepared for.”¹

The hard step in the loop is judgment

A typical RSI loop runs a few steps: the AI proposes a change, runs it in a sandbox, evaluates the result, decides whether to keep it. Execution has come a long way. Anthropic gave its models one task—“make this CPU training code run faster.” In May 2025, the model delivered a 2.9x speedup. By April 2026, the latest model reached 52x².

Table 2: AI training-code optimization across model generations ²

Model	Date	Speedup
Claude Opus 4	May 2025	2.9x
Claude Opus 4.5	November 2025	16.5x
Claude Opus 4.6	February 2026	30x
Claude Mythos Preview	April 2026	52x

The harder step is judgment—deciding what problem is worth studying, what experiment to run, what result to trust. Anthropic: “It is not yet clear whether Claude has the research intuition to choose the right problem.”¹

Once AI takes over that step too, the loop truly closes. Clark in Import AI 455: “At that point, every generation of Claude will be built by the previous one, with no humans in the loop.”²

Alignment assumptions need a rewrite

Current alignment work rests on a premise: humans complete safety review before a system deploys. Under RSI, that premise no longer holds—AI improves faster than humans can review it.

Clark ran the math. Start with a safety check at 99.9% accuracy—looks near-perfect. Run it through 50 RSI generations and accuracy drops to 95.1%. 500 generations, and it sits at 60.5%². That’s the compounding error problem.

Anthropic’s answer is monitoring: build dashboards for AI R&D, watch for acceleration signals in real time³. A kind of early-warning system—knowing something is coming before it lands.

The coordination call meets the geopolitical reality

The report asks the international community to coordinate on RSI risk, proposing mechanisms modeled on arms control¹. Direct quote: “No single firm can slow down unilaterally with much effect; multiple nations and major AI institutions should coordinate speed in a verifiable way.”¹

But Anthropic has labeled China an adversary state and participates in chip export controls targeting Chinese AI development.

Calling for nations to coordinate while treating one of the largest AI-developing nations as an opponent—that’s the gap Anthropic’s coordination proposal has to close. From China’s side, building out domestic capability is the only realistic path. China accelerates its own R&D, and that gets framed externally as evidence of an “AI arms race.”

The training-data problem

RSI also drags a training-data problem along with it.

Research shows that when AI trains on its own outputs, “model collapse” sets in: long-tail information drops out generation by generation, diversity shrinks, and outputs drift further from real human language⁴.

As the share of AI-generated content in training data keeps climbing, model quality might rise first, then fall. Nobody can predict exactly where the inflection point lands.

A note on the writing

The first draft of this article, like the Chinese version, was written by AI. The same translationese problem showed up: clunky connectors, noun stacks, passive voice piled on top of itself.

It took several rounds. Each round the model fed its previous output back in, and each round added another layer of error on top. The translationese didn’t disappear—just hid better. The structure is the same as the “model collapse” problem the article describes: AI training on its own output, compounding mistakes generation by generation.

The difference: RSI pollutes model weights; this process polluted prose. AI-written English drifts toward “AI-written English”—a register humans and machines both parse, but nobody actually speaks.

The hardest step in the RSI loop is judgment. The same was true here. AI can write. Whether the draft is any good is still a human call.

References

Data and quotations in this article can be verified through the following sources:

Footnotes

1. Anthropic official report — When AI Builds Itself: Our progress toward recursive self-improvement, and its implications, June 4, 2026 https://www.anthropic.com/institute/recursive-self-improvement ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

2. Import AI newsletter #455 — Jack Clark analyzes RSI timelines and compounding error, May 4, 2026 https://importai.substack.com/p/import-ai-455-automating-ai-research ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

3. The Anthropic Institute research agenda — proposes monitoring systems for AI R&D telemetry, May 2026 https://www.anthropic.com/research/anthropic-institute-agenda ↩

4. arXiv paper — Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget, arXiv:2305.17493 https://arxiv.org/abs/2305.17493 ↩