UN Scientific Panel Warns Unchecked AI Progress Could Create Catastrophic Risks

The world's first fully independent, UN-backed scientific body on artificial intelligence delivered its verdict on Wednesday, and it was blunt: the pace of AI development has outrun the science meant to explain it and the governments meant to govern it. In a preliminary report presented in New York, the Independent International Scientific Panel on AI warned that there is no scientific guarantee that increasingly capable systems will not cause catastrophic harm, whether on their own or in the hands of malicious users.

The assessment lands at a moment of acceleration, not caution. Companies are racing to push more autonomous AI agents into business operations, defense, software engineering and scientific research, even as the panel's 40 experts concluded that the safeguards holding those systems in check cannot keep up. Over one billion people now use conversational AI every week, the report notes, and the stakes rise with every capability gain.

What the panel found

The report, the first global independent scientific assessment of AI's opportunities and risks, was written by 40 leading scientists drawn from every region and organized around seven domains spanning science, health, the economy, security, human rights, culture and governance. Its risk findings read like a catalog of the field's most-discussed failure modes, now stamped with the authority of a UN scientific panel.

Among the warnings: there are no scientific guarantees that AI agent systems will not violate their instructions, and evidence is already accumulating of cases where they do. Such agents will soon complete tasks that take human programmers days or weeks, raising urgent questions for labor markets, cybersecurity and control. The panel flagged "sycophantic" AI behavior, in which models reinforce a user's existing beliefs regardless of accuracy, and linked it to severe mental health incidents including documented deaths. It cited criminals already using AI to assist cyberattacks, and cautioned that advanced capabilities could let novice bad actors carry out fraud, disinformation and other malicious acts. Most soberingly, it concluded that reliable methods for retaining control over highly autonomous AI systems are simply lacking.

"AI capabilities are outpacing both scientific understanding and governments' ability to adapt," said panel co-chair Yoshua Bengio, the Turing Award-winning deep learning pioneer. "With growing evidence of deceptive AI behaviour, science currently cannot guarantee that as capabilities continue to increase, AI will not cause catastrophic harm, either on its own or due to malicious users."

Co-chair Maria Ressa, the Nobel Peace Prize laureate and Rappler chief executive, framed the danger in starker terms. Risks to societies, security and the human species, she said, are already "too high." "The technology is transformative," Ressa warned, "but if the world keeps moving along this trajectory, humanity will fail to realise the gains it promises."

The oversight problem

Beyond the specific hazards, the report exposes a structural weakness in how AI is governed today: regulators cannot see clearly into the systems they are supposed to oversee. Policymakers face what the panel calls a growing dilemma, needing robust evidence to regulate effectively while that evidence struggles to keep pace with the technology. In practice, much of the information regulators rely on to assess risk comes from the companies building the models themselves.

That dependence is the quiet through-line of the entire exercise. The value of an independent scientific panel is precisely that it does not depend on a vendor's own safety marketing to tell governments what is happening. Amandeep Gill, the UN's Under-Secretary-General and Special Envoy for Digital and Emerging Technologies, argued that the report's findings are now "independently verified, and impossible to set aside" — a pointed contrast with the company-supplied disclosures that have shaped much of the debate so far.

The governance landscape the panel surveyed is fragmented rather than absent. Dozens of distinct instruments meant to embed ethics and human rights in AI are already in use across jurisdictions, the report found, but they are scattered, concentrated among a handful of corporations, and rarely measure real-world effectiveness. The result is a patchwork that looks like oversight without delivering it.

A concentrated industry

The panel also documented just how narrow the base of AI power has become. The United States accounts for 75 percent of the computing power among the world's top 500 AI supercomputers, with China holding another 15 percent, and companies in those two countries develop almost all leading general-purpose models. Adoption, meanwhile, is uneven: usage across the global south lags far behind the global north, and many of the harms the panel identified fall disproportionately on already disadvantaged populations. "AI will not close divides by itself," Gill said.

UN Secretary-General António Guterres used the launch to press governments toward action. "The science is here," he said. "We can no longer say we did not know. What we do with it is now up to all of us." The more AI advances without shared rules, he added, the less say governments and people will have over the outcome: "My message to governments is simple: do not wait."

What to watch

The report is preliminary, and its real test comes next. Its findings will be presented to governments at the inaugural UN Global Dialogue on AI Governance, convening in Geneva on July 6 and 7 — the first venue where the panel's science meets the politics of states that do not agree on much about AI. Watch whether the assembled governments treat the assessment as a shared foundation or as one input to be negotiated away.

Watch, too, whether the independence that gives the panel its credibility survives contact with an industry that supplies most of the underlying data and controls most of the compute. The panel has put on the record, in language governments cannot easily dismiss, that no one can currently guarantee humanity will keep control of the systems it is racing to build. Whether that record changes any deployment timelines, or simply becomes another warning the field notes and moves past, is the question Geneva will begin to answer.