'Agentic AI Scientists Are Not Built for Autonomous Scientific Discovery,' a New Paper Argues

Most critiques of AI scientists argue the systems are not good enough yet. A position paper posted to arXiv on May 9 makes a harder claim: that they are built wrong. Five researchers from IIT Delhi and Friedrich Schiller University Jena argue that what separates today's agentic research systems from genuine autonomous discovery is not a deficit of scale or scaffolding, but consequences of how the underlying models are trained and how the field measures success.

The paper, "Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery" (arXiv:2605.08956), is by Harshit Bisht, Vinay Kumar, Kevin Maik Jablonka, Mausam, and N. M. Anoop Krishnan. Its position is not that AI should leave the lab. "The position advanced here is not that AI systems should be removed from scientific workflows," the authors write — "they are useful as a co-scientist: a capable collaborator that extends human scientific judgement rather than displacing it." What they reject is the stronger claim, embodied in systems like Sakana's AI Scientist-v2 and the Nobel Turing Challenge framing, that the whole loop can be closed without a human in it.

Four gaps, none of them about scale

The paper organizes its case around four failures, each traced to a design decision rather than a capability ceiling.

The first is problem selection, which the authors describe through the McNamara fallacy — measuring what is easy to measure, then presuming the unmeasured does not exist. Because machine learning needs quantifiable objectives and large, tractable datasets, AI-driven science drifts toward problems that have those properties. They cite a study of 41.3 million research papers finding that AI-augmented scientists publish 3.02 times more and collect 4.84 times more citations, while the collective volume of topics studied shrinks by 4.63% and scientist-to-scientist engagement falls 22%. Acceleration, in other words, with narrowing.

The second is what the published record leaves out. Journals select for conclusions over process and positive results over negative ones, so a model trained on the literature inherits the output of that filter rather than the knowledge behind it. Tacit laboratory knowledge is largely unwritten; so is failure knowledge. The authors' conclusion is blunt: "a corpus filtered for successful conclusions cannot teach how and when to abandon a hypothesis, an essential skill for autonomous discovery."

The third is post-training: RLHF and DPO push outputs toward annotator consensus, and those annotators are shaped by the same published literature the pre-training corpus encodes. The fourth is evaluation. The paper cites a survey of 294 papers across 17 fields in which correcting for eight common data-leakage types collapsed the apparent advantage of complex ML models over classical baselines, and an agentic evaluation spanning more than 25,000 runs across eight scientific domains where evidence was ignored in 68% of traces and refutation-driven belief revision occurred in only 26% of cases. On physical validation, they note that of GNoME's 2.2 million predicted crystal structures, 736 had been independently experimentally realized at the time of publication.

The hypothesis hivemind

The paper's one original experiment tests the diversity claim directly. The authors took 50 papers from the NeurIPS 2025 AI4Mat track and posed two tasks to six frontier models from two providers — Claude Haiku 4.5, Sonnet 4.5 and Sonnet 4.6 from Anthropic, and GPT-5 Nano, Mini and GPT-5 from OpenAI. Task one asked models to recover a paper's underlying hypothesis from an experiment summary, where convergence is expected. Task two gave models the full text and asked for novel hypotheses extending the work, where divergence is the whole point. Ten samples per model per task were embedded and compared by cosine similarity.

Similarity stayed high in both. "A research community that queries multiple AI systems for novel hypotheses is, from an epistemic standpoint, effectively sampling from a single model," the authors write. "The effective epistemic sample size, for generating directions the field has not already taken, is close to one regardless of how many systems are consulted." Control comparisons confirmed the embedding model could distinguish outputs from different source papers, ruling out the trivial explanation that scientific text simply embeds close together.

Analysis: the counter-evidence is real, and it lands on one side of the line

Three months after this paper appeared, OpenAI announced that an internal Astra model had produced new results on ten open problems in mathematics and theoretical computer science, shipping machine-checkable Lean 4 certificates for roughly $2,000 in compute. DeepMind's Aletheia has autonomously solved open Erdős problems. Neither existed when the authors wrote, and both are genuine.

But they sit almost entirely on the computational side of the boundary the paper draws. The authors anticipate the objection in a section titled "Alternative views": "AlphaFold solved a problem formally stated for fifty years, with an unambiguous evaluation criterion available at experimental scale. GNoME searched a compositional space against a computable stability criterion." Formal mathematics is the extreme case — the verifier is a compiler, the objective is stated in advance by a human, and no wet lab is involved. Astra is arguably the strongest possible evidence for exactly the regime the paper concedes AI already excels at, and close to silent on tacit knowledge, physical feedback, or whether a system can decide which problem was worth attacking. Whether that boundary is a wall or a frontier remains open — but the fastest-moving counter-evidence has so far accumulated on the near side of it.

The recommendations follow from the diagnosis rather than calling for more compute: simulators as scalable verifiers for training, persistent world models carrying an agent's epistemic state across an investigation, and a centralized, machine-readable preregistration repository for AI-generated hypotheses — cheap to enforce via an API call, and doubly useful as an archive of failure knowledge.

What to watch

Two things. First, whether anyone replicates the hivemind result with more providers — six models from two heavily RLHF'd labs is suggestive rather than decisive, and adding open-weight or differently post-trained models would sharpen it. Second, whether an autonomous system closes a loop that ends in a physical measurement contradicting its own prediction, and then revises the model that produced it. That is the test the paper argues no current benchmark administers, and the one that would move the argument.