When AI Starts Improving Itself, Who's Actually in Control?

A lot has happened in artificial intelligence over the past few days. Enough, actually, that several stories I initially viewed separately started to look like parts of the same conversation.

First, Jacob Coxon, an AI researcher who worked at both OpenAI and Anthropic, abruptly resigned from Anthropic, accusing his former employers of “gambling with our lives” as they race to develop super human artificial intelligence. Then Anthropic CEO Dario Amodei called for slowing the pace of AI development.

Amodei has been sounding alarms about AI risk for some time, so his concern didn't surprise me. What certainly wasn’t on my 2026 bingo card was Sam Altman and Elon Musk agreeing with him, yet here we are.

That doesn't mean these men suddenly agree on AI policy broadly, they most certainly don’t, but when leaders who compete with one another, and often publicly disagree with one another, begin converging around the idea that something about the pace of frontier AI development requires more attention, I think it is worth paying attention.

The question that interests me is: What are the people closest to the development of these systems seeing that is causing this conversation to change?

One answer may lie in something called recursive self-improvement, or RSI.

A research paper published September 10, The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement, via the University of Shanghai (China) examines the progression toward AI systems taking increasing responsibility for their own improvement.

In regular-person English, imagine an AI that doesn't get better because engineers improve it. Imagine a system that increasingly participates in figuring out what is wrong with itself, deciding how to improve, testing whether the solution worked, retaining what it learned, and improving the very process it uses to make the next improvement.

We are no longer talking about AI performing a task faster than a human. We are talking about AI potentially participating in deciding how the AI that comes next gets better.

The more I read about RSI, the more I realized that my biggest concern is the closed loop.

Imagine an AI identifies something it isn't doing particularly well. The AI diagnoses the problem, proposes a solution, implements it, tests the results, determines whether the solution worked, and retains the change so what it learned can influence the next round of improvement.

At first glance, that sounds efficient. From a governance perspective, I immediately have questions.

Who independently verifies that the AI correctly identified the original problem? Who determines that the proposed solution didn't create another problem somewhere else? Who decides that the test used to evaluate the improvement was actually a good test?

And who is checking the checker?

This is why I think organizations need to reconsider a common phrase in responsible AI: human in the loop.

Inserting a person somewhere in an autonomous process doesn’t create meaningful human oversight. If an AI workflow can execute tasks, evaluate results, modify its approach, retain lessons from previous attempts, and continue operating between human reviews, technically there may be a human in the loop. But how many autonomous cycles can happen between those reviews? What can the human actually see? What can the AI change without permission? Can the human reverse those changes?

And what happens when the human goes home?

If the person responsible for supervising an autonomous workflow signs off at 6 p.m. and the system continues operating overnight, encountering problems, changing strategies, evaluating those changes, and retaining what it learns, was that human really in the loop, or was the human near the loop occasionally?

Maybe the governance standard we need isn't human involvement. Maybe it’s human control.

Then I looked at what China is doing.

The timing is particularly interesting. The RSI paper was published on September 10. Just three days later, Xi Jinping outlined a BRICS agenda that included an AI open-source community, cooperation on large language models, AI training, digital infrastructure, engineering talent development, and a larger role in global AI governance.

I am not suggesting those developments are directly connected. What interests me is what they represent when viewed side by side. Within the span of a few days, researchers were describing a potential future in which AI systems assume progressively greater control over how they improve, while China was laying out pieces of an international ecosystem for how increasingly powerful AI might be developed, shared, supported, and governed.

Meanwhile, here in America, we continue to have pissing contests that force these issues into opposing camps: innovation versus regulation, open source versus closed source, development versus safety, business versus government, competition versus cooperation, and China versus America.

It's getting ridiculous. In fact, it's been ridiculous.

The United States pioneered much of the technology behind today's AI boom and remains home to many of the world's most important AI companies and researchers. But technological leadership doesn’t guarantee leadership over everything that develops around the technology.

A country can lead in model capability while falling behind in governance. It can lead in investment while falling behind in workforce preparation. It can lead in advanced chips while falling behind in education. And it can build some of the most powerful AI systems in the world while other countries build the standards, training programs, international relationships, technical ecosystems, and governance structures that influence how billions of people encounter those systems.

Those things aren't secondary to AI. They are part of the AI ecosystem.

That is why I don't think the AI race is about who builds the smartest model first. It’s also about whether our ability to govern autonomous systems can evolve as quickly as the systems themselves.

So I'm left with the same questions I raised in the full article: How do we know when an AI has genuinely improved rather than just gotten better at passing our tests? At what point does “human in the loop” become more than language that makes us feel better about a process humans no longer meaningfully control? What happens when the speed of autonomous change exceeds the speed of human oversight?

And perhaps the deeper geopolitical question: What happens if America wins the race to build AI, but someone else wins the race to govern it?

I explore the research behind recursive self-improvement, the limits of “human in the loop,” and the emerging global AI governance race in the full Thinking Through AI article: https://www.linkedin.com/pulse/3-meisa-bonelli-frjqc

Sources:

Scientific American — Jacob Coxon, Anthropic, and AI risk https://www.scientificamerican.com/article/ai-jacob-coxon-quit-extinction-fears-security-experts-see-familiar-fight/

The New York Times — Dario Amodei and calls to slow AI development https://www.nytimes.com/2026/09/12/technology/anthropic-dario-amodei-ai-slowdown.html

The Guardian — Sam Altman and Elon Musk back calls to slow AI development https://www.theguardian.com/technology/2026/sep/13/openai-sam-altman-elon-musk-back-anthropic-calls-brakes-ai-development

POLITICO — Barack Obama on AI, Democrats, jobs, and policy https://www.politico.com/news/2026/09/13/obama-ai-democrats-fundraiser-01073778

The Black Wall Street Times — Black women and workforce losses https://theblackwallsttimes.com/2026/08/05/600000-black-women-pushed-out-of-the-workforce-new-iwpr-report-examines-why/

arXiv — The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement https://arxiv.org/abs/2609.11873

CNBC — Xi Jinping, China, AI, and BRICS https://www.cnbc.com/2026/09/13/china-xi-ai-tech-brics.html

China Ministry of Foreign Affairs — Xi Jinping's BRICS statement https://www.fmprc.gov.cn/mfa_eng/xw/zyxw/202609/t20260913_12021300.html

China Ministry of Foreign Affairs — AI governance and World Artificial Intelligence Conference https://www.fmprc.gov.cn/eng/xw/zyxw/202607/t20260717_11984910.html

China Ministry of Foreign Affairs — AI cooperation and the Global South https://www.fmprc.gov.cn/eng/xw/zwbd/202605/t20260512_11908944.html

Bastille Post — Former Jordanian Prime Minister Omar Razzaz on China's AI development and governance approach https://www.bastillepost.com/global/article/6150770-former-jordanian-pm-praises-chinas-ai-development-approach-to-global-governance

arXiv — Research on AI self-improvement and evaluation https://arxiv.org/abs/2607.07663

arXiv — Research on AI-assisted research feedback and self-amplifying development https://arxiv.org/abs/2609.00137

Thinking Through AI — When AI Stops Being (Just) a Tool https://www.meisab.com/thinking-through-ai/when-ai-stops-being-a-tool