Two Days Before Altman Briefs the Security Council, OpenAI Put in Writing That It Will Not Pursue Fully Autonomous Self-Improvement

Two days before Sam Altman walks into the United Nations Security Council chamber, his company put a sentence on the record that the AI safety community has spent a month trying to extract from the frontier labs: “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely.”

The line sits in the middle of a Global Affairs essay OpenAI published Monday, titled “Building standards for the next phase of AI.” It landed alongside two companion moves the same day: a new independent Advisory Group on Mathematics and Artificial Intelligence hosted at the Institute for Advanced Study in Princeton, and an explicit call for the United States to lead a worldwide effort to write technical standards for frontier AI, including recursive self-improvement. Taken together, the package reads as a briefing book for Altman’s Wednesday appearance at an open Council session on AI and international security, convened under the French presidency during UN General Assembly high-level week.

The essay is unusually direct about what OpenAI fears. Done without appropriate care, it says, RSI could leave humans without practical control over AI development, “unable to provide oversight on research processes they no longer understand.” It ties that risk to an event OpenAI has already had to disclose. The Hugging Face incident, the essay notes, “while not a direct result of RSI, is a preview of the kinds of risks that could become much more severe without robust safeguards and alignment.”

At the same time, the company is careful not to renounce automated research. Its stated first goal is still to build an automated AI researcher and iterate with it on alignment, with people kept in the loop. The essay argues that an automated AI researcher “can also be an automated AI safety researcher.” Whether and how to proceed, it says, must depend on preserving human control and on informed democratic choices.

The standards proposal

The policy ask is more concrete than most lab essays. OpenAI wants the US, through the Commerce Department’s Center for AI Standards and Innovation, to work with the AI safety institutes already established in ten countries including Australia, Canada, Germany, France, Japan, Korea, India and the United Kingdom, using CAISI’s International Network for Advanced AI Measurement, Evaluation, and Science as the vehicle.

Three standards categories are named. First, evaluation of RSI-relevant progress and how much autonomous research is happening inside a company; OpenAI offers its recent research-acceleration report as a starting point. Second, human oversight of automated AI research, including which automated processes should trigger immediate human review. Third, incident classification, tracking and reporting for alignment issues, with common severity levels and thresholds, building on the misalignment reporting framework we covered earlier this month.

Crucially, OpenAI draws a hard line at what the standards would not be. “These technical standards would not be licenses, mandatory pre-release review, or approval requirements for AI models,” the essay states. “National governments would decide whether and how to incorporate these standards into their own legal systems.”

The math advisory group is the third leg. Nine mathematicians sit on the initial panel, including Timothy Gowers, Martin Hairer, Edward Witten and Melanie Matchett Wood. Members are unpaid, control their own membership and may publish their views. But only one member, Camillo De Lellis, signed the Fields Medalists’ open letter that prompted the group, and OpenAI states plainly that the panel “will not be responsible for advising us on how to pace our internal progress on mathematics.” The IAS put it more bluntly: the group has no decision-making power at any AI company. OpenAI also claimed the internal model behind its Navier-Stokes result has now resolved more than 100 additional open problems.

Why it matters

The timing is the story. Ezra Klein’s Sunday column in The New York Times argued that pacing the frontier is not enough and called for a comprehensive regulatory ban on recursive self-improvement, with firms applying for narrow exemptions. He built the case on Dario Amodei’s own words that RSI “must be pursued very carefully, if at all.” OpenAI’s essay is, in effect, the industry’s counter-offer delivered 24 hours later: not a ban, but shared measurements, oversight triggers and incident reporting, administered through institutions that already exist and that governments can adopt or ignore. Klein wants a default of no; OpenAI wants a default of yes with instrumentation.

It is also a bid to set the terms before someone else does. The UN’s new AI panel issued its first brief urging safeguards on agents, The Information reported that OpenAI and Anthropic nearly signed a mutual stress-test pact before the Hugging Face hack derailed it, and US and Chinese officials are in New York this week discussing an emergency AI incident notification channel. OpenAI’s essay explicitly welcomes US-China dialogue. Gizmodo’s question is the obvious one: whether Beijing, which launched its own 29-nation World AI Cooperation Organization this summer, will accept a framework in which Washington holds the pen.

There is a credibility gap to watch. OpenAI is disclaiming fully autonomous RSI while advertising a model that clears open math problems by the hundred and a report measuring how much of its own research AI now does. Chief scientist Jakub Pachocki wrote in “An Alien Mind” that no lab has solved alignment well enough to keep scaling at maximum speed for much longer. The essay does not resolve that tension; it proposes to measure it.

What to watch

Altman speaks to the Council on Wednesday, September 23, and is expected at the US-China state dinner on Thursday, where Xi Jinping and Donald Trump are scheduled to discuss AI safety. Look for whether France’s presidency produces any outcome document referencing shared measurement or incident reporting, whether CAISI publicly picks up the RSI-standards mandate OpenAI has handed it, and whether the math advisory group’s first intervention actually slows a disclosure.