Given Only a Bibliography, Frontier Models Recover 3% of a Paper's Idea

Strip a research paper away and leave only its footnotes. That is the entire input in Reconstruction, a benchmark posted to arXiv on August 17 that puts a deceptively simple question to seven frontier language models: given a paper's pre-publication bibliography, and nothing else, can you say what the paper proposed?

Mostly, no. Across 643 evaluated papers spanning six scientific domains, individual models matched the real idea roughly 3 to 15 percent of the time. The strongest single model, Claude-Opus-4.8, averaged 13.3 percent, plus or minus 2.3. A multi-agent pipeline that ran four models' competing hypotheses through cross-model review and a Swiss-system tournament pushed match rates to 23 to 42 percent — an observed 2.4x lift, and still a minority of ideas recovered.

The paper, Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies, comes from a nine-author team led by Shaolong Chen and Yanlin Fei and including Qingqing Mao and Ritankar Das, filed under the AI-Professor Project banner. It is a preprint, revised once on August 19, and has not been peer reviewed.

The task, and the wall around it

Most evaluations of AI-generated research ideas ask whether an idea is novel, interesting, or preferred by human reviewers. Reconstruction asks something narrower and much harder to game. As the authors frame it in their introduction, the diagnostic question is whether, "given only the references available before a paper was published, can a model recover what that paper actually proposed?"

The answer is known in advance — it is the seed paper's own contribution — which makes contamination the central threat to validity. If a model has read the target paper during training, it can retrieve rather than reason. The team built four defenses. A hard temporal cutoff admits only literature published strictly before the seed paper's publication date. References are stripped of author names and metadata and re-labeled as anonymous IDs (ref-001 and onward), closing the shortcut of recognizing a lab's trajectory. Bibliographies are frozen per paper, so the single-model and multi-agent conditions see byte-identical inputs. And the models never see the seed title or abstract during generation; an independent LLM judge compares the proposed hypotheses to the held-out ground truth afterward, with judges recused from scoring anything they proposed.

The corpus is not synthetic. The machine-learning slice is the full ICML 2026 Oral program, 168 papers scraped from the conference site. The other five domains are Nature-family journals scraped on July 14, 2026: Nature Medicine (221 candidate papers), Nature Physics (153), Nature Materials (131), Nature Chemistry (115), and Nature Astronomy (91). Titles were resolved to bibliographic metadata through Semantic Scholar, with fallbacks to OpenReview, Crossref, OpenAlex, and arXiv. Attrition was heavy and is documented: of 826 launched runs, 745 completed the single-model condition, and 643 survived the filter requiring five aligned hypothesis slots.

In the baseline condition, each model produces five distinct hypotheses in a single generation call. The multi-agent pipeline is where the interesting machinery lives. The four strongest baseline models — Claude-Opus-4.8, GPT-5.6-Sol-Pro, Kimi-K3, and GLM-5.2 — each generate five hypotheses from identical frozen references, yielding 20 candidates. Those candidates are aligned into five slots of four, reviewed by the other models with proposer recusal and no web search, then run through a Swiss tournament with conflict-of-interest recusal and presentation-order debiasing. The five slot champions are assembled into one case and scored by a final judge that never saw the tournament ballots.

Why this lands hard on the "AI scientist" pitch

The uniformity of failure is the finding, not the floor. Seven capable systems clustering in a 12-point band suggests the gap is architectural rather than a matter of prompting or parameter count. And Reconstruction is deliberately measuring the one step that "AI scientist" products are sold on — the leap from what the literature contains to what nobody has proposed yet. Evaluations that hand a model full paper text, author names, or post-publication signals are measuring something else, and their scores do not transfer.

The multi-agent result cuts both ways. A 2.4x lift from adversarial filtering, with zero additional information entering the system, is a real argument that the right unit of AI-assisted discovery is a structured committee rather than a bigger oracle. It echoes the tournament-based generation-and-ranking design in Google DeepMind's AI Co-Scientist. But the authors are careful about what they are claiming. "We credit the full selection pipeline rather than collaboration alone," they write, flagging that relative comparisons are reported as observed associations, with candidate-matched controls deferred to future work.

They are equally candid about a sharper caveat: the top-4 roster was chosen by performance on the same papers used for scoring. That is "a post-selected ensemble relative to the same papers used for scoring," they note, and should be read as evaluating a fixed roster rather than a selection rule validated on held-out data. Judge disagreement is also nontrivial — the team observes that Claude and GPT judges are "systematically more permissive than Kimi/GLM" on the multi-agent champions, which are more specific and therefore harder to grade.

For scale, OpenAI's PaperBench (April 2025) found its best agent reached a 21.0 percent replication score on 20 ICML 2024 papers, with full text available. Reconstruction removes the text and gets 3 to 15 percent from a single model.

Watch three things. First, whether the promised revision delivers full per-model and per-domain breakdowns; several domains carry thin reported counts, including 117 for Materials and just 78 for Medicine, where the multi-agent run was capped at 80 papers to control API cost. Second, whether anyone reports the compute bill for the 2.4x lift — 20 generations plus review plus tournament rounds is not a cheap default. Third, whether labs shipping hypothesis-generation tools start publishing contamination-resistant numbers, or keep quoting scores from evaluations that leave the answer in the room.