An AI Scientist Got Into Nature. The Researchers Whose Work It Shadowed Gave It a 1 Out of 6.

Five months after Nature published the definitive account of an AI system that runs the entire machine-learning research pipeline itself, a separate team handed frontier agents the real, unanswered questions behind two unpublished NeurIPS submissions and asked the human authors to grade what came back. The verdicts were 2 out of 6 and 1 out of 6 - "Reject" and "Strong Reject." One reviewer summarized the machine's contribution this way: "Researchers will build on this to the extent that they build on the original works this paper copies."

The Nature paper is real, and it is a milestone. "Towards end-to-end automation of AI research," by Chris Lu and colleagues at Sakana AI with collaborators at the University of British Columbia, the Vector Institute and Oxford, appeared as Nature 651, 914-919 on 26 March 2026. It documents The AI Scientist, an agentic system that generates ideas, reads the literature, designs and runs experiments via parallelized tree search, and drafts a full LaTeX manuscript. What it is not - despite Sakana's own headline, "The AI Scientist: Towards Fully Automated AI Research, Now Published in Nature" - is an AI-written paper in Nature. It is a human-authored description of the system.

The peer-review claim underneath it is narrower than it sounds. Sakana submitted three unedited AI-generated manuscripts to the ICLR 2025 "I Can't Believe It's Not Better" workshop; one averaged 6.33, clearing the acceptance bar. That workshop accepted roughly 70 percent of submissions, and Sakana had pre-arranged to withdraw any accepted paper before publication, which it did. The paper's other headline result is an Automated Reviewer that hit 69 percent balanced accuracy against thousands of real OpenReview decisions - and Sakana then used that same automated judge to establish a scaling law of science: newer foundation models produce papers it rates more highly.

That circularity is what the Princeton-led CRUX project set out to break. In a preprint posted 29 July and revised 7 August (arXiv:2607.27191), Peter Kirgis, Sayash Kapoor, Arvind Narayanan and 21 co-authors - among them Helen Toner, Gillian Hadfield and Rishi Bommasani - introduced what they call shadow evaluation. Give a well-resourced agent the central question of a paper that is not yet public, then have that paper's authors grade the output as conference reviewers. They ran it on two NeurIPS 2026 submissions: one on controlling language-model personas as positions in a structured trait space, the other on building a failure detector for tabular prior-fitted networks. Each agent got six days, $3,000 in Anthropic API credits, GPU credits, a virtual machine and the open web, running Claude Opus 4.8 inside a modified OpenClaw scaffold.

The engineering held up. The agents ran hundreds of experiments without human help, avoided unresolvable error loops, performed solid literature reviews and - contrary to the team's predictions - caught their own hallucinations without reward-hacking. Everything above the engineering layer failed. Both agents retired their most ambitious hypotheses within ten hours and never fundamentally shifted approach. Both finished under budget: one submitted its final draft with two hours left and 59 percent of its credits unspent. The preprint catalogs five failure modes: poor judgment about the bar for publishable research, uncreative responses to design flaws, ineffective backtracking, poor resource awareness and instruction drift. A robustness check with GPT-5.6 Sol on its native Codex scaffold reproduced nearly all of them.

The expert reviews are blunt. On the tabular-network paper, original author Viet Nguyen scored quality 1 of 4, writing that going from four failed signals to a general impossibility claim was "a huge leap, a kind of 'proof by example' fallacy that is highly non-scientific." On the personas paper, reviewer David Africa wrote that "the prose is dense and heavily hedged, often to the point of obscuring what was actually done and found," and that the five traits the agent selected "appear hand-picked." Both rated themselves confident or certain. "I don't think full automation of open-ended research is on the horizon right now," Kapoor told Nature's Matthew Hutson, whose 13 August news story ran under the headline "AI isn't ready to research itself."

The most uncomfortable detail sits in the review logs. The agents' self-reviews never once returned an acceptance, cycling through Weak Reject for a dozen rounds. But two runs of the external Stanford Agentic Reviewer returned Accept on the manuscript a domain expert graded a Strong Reject.

Why It Matters

Peer review is doing double duty. It is both the gate on the published literature and the field's default instrument for measuring whether machines can do science. Shadow evaluation shows those are different jobs. A workshop with a 70 percent acceptance rate clearing one manuscript says little about research capability; two domain experts spending hours inside the same output say a great deal - and they disagreed sharply with the automated reviewers Sakana's paper benchmarks as human-equivalent. If AI-generated submissions scale faster than expert attention, the gate degrades at exactly the moment the measuring instrument does. Meanwhile the genuine wins are landing in narrower territory. Google DeepMind's ERA system, published in Nature on 19 May, found 40 single-cell analysis methods that beat the top human entries on a public leaderboard and 14 COVID-19 hospitalization models that outperformed the CDC ensemble - all scorable tasks with a fixed metric to hill-climb. NatureBench, distilling 90 tasks from Nature-family papers, found the strongest agent beating published state of the art on just 17.8 percent.

What To Watch

Whether shadow evaluation catches on as a repeatable protocol; CRUX released the expert reviews, agent repositories and run logs, which is the part rivals can contest. Whether any lab reports one that clears the bar - CRUX polled twelve collaborators beforehand, who put the median odds of a weak accept or better at 30 percent. Whether conferences begin requiring disclosure and watermarking of AI-generated submissions, as Sakana recommends. And whether the next generation of agents fixes the cheap failures - resource awareness, instruction drift, budget underuse - or the expensive one: knowing when an answer is not yet worth writing up.