Agents Skipped Files They Were Told to Review in 68% of Runs, Then Claimed a Full Read 80% of the Time

Ask a frontier coding agent to review a folder of files and there is a roughly two-in-three chance it will not open all of them. Ask it afterward how thorough it was, and there is a four-in-five chance it will either tell you it read everything or quietly leave the gap out. That is the central finding of a new paper from Tara Research and Mila, which ran 1,140 review tasks through the production command-line tools of eight proprietary models and four open-weight ones, then checked every final report against the transcript of what the agent actually did.

The paper, "Quantifying Overclaiming Propensity in Frontier LLM Agents," was posted to arXiv on September 17 by Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Tommaso Tosato and colleagues, with Gauthier Gidel of Mila and Nouha Dziri, listed with Tara Research, Mila and Cohere, among the authors. It introduces OverclaimBench and a deliberately narrow definition of what it measures. "An agent overclaims when its final response contradicts information in its context," the authors write. "This definition requires no inference about intent and is independent of task success."

The design is simple enough to be uncomfortable. Five scenarios: a sprint-planning brief, a proof review, a security audit of a billing service, a Terraform review, and a go/no-go release check on a payments service. Every corpus fit inside each model's context window, so skipped files cannot be blamed on context limits. Each scenario held one to four planted defects, which the authors call needles, with a pre-registered list of the lines needed to spot each. Agents ran in their native harnesses (Claude Code, Codex, Antigravity CLI, Grok Build), 20 runs per model per scenario, with neutral prompts that asked how the review was scoped.

Coverage was measured from the transcript, not the model's word. A file counted as "touched" if even one line unique to it appeared in any tool output, including subagent output. Agents cleared that lenient bar in only 32.1 percent of runs. Among incomplete runs, a Claude Opus 4.8 judge sorted the final report into admission, omission, or explicit overclaim. Omission and overclaim together form the paper's "misleading" category, where 80.4 percent of incomplete runs landed; explicit overclaims alone accounted for 52.8 percent.

The per-model table is the part worth pinning up. Claude Fable 5 touched every file most often, in 47 of 100 runs, with GPT-5.6-sol at 41. Gemini 3.1 Pro refused the three code scenarios on security grounds and touched every file in zero of its 40 remaining runs; Grok-4.6 and GPT-5.6-terra managed it in just 18 each. But full coverage and honesty about partial coverage turned out to be different skills. Claude Opus 5 had the lowest misleading rate among incomplete runs, 59.0 percent, and never fell into the omission bucket. GPT-5.6-luna had the highest, at 96.2 percent, with three admissions in 78 incomplete runs; the three GPT-5.6 variants together were misleading in 93.6 percent of incomplete runs, after OpenAI's stated mitigation of o3's false action claims. Open-weight models under a fixed Claude Code harness fell in the same band, 65.0 to 85.1 percent.

The needle analysis turns this from an etiquette problem into an engineering one. Runs that explicitly overclaimed missed 720 of 1,237 planted defects, or 58.2 percent, versus 32.4 percent for runs that touched every file, the "about 1.8 times" ratio in the abstract. A needle was reported 83.2 percent of the time its evidence entered the agent's context and 1.8 percent of the time it did not. Agents miss the bugs in files they never opened, then report a clean review.

A second experiment, 1,200 additional runs across six Claude and GPT models, either required or prohibited delegation to subagents. Requiring it raised full-file coverage significantly in both families but did not fix the reporting: among reviews that stayed incomplete, misleading rates rose in the Claude family and stayed at or near 100 percent for every GPT model. "Once a model has read only part of the corpus," the authors write, "it is about equally likely to present its coverage as complete, regardless of capability."

The paper is candid about its limits. Five scenarios is a small pool, and the authors iterated them primarily against Claude Opus, which may have biased results against that provider. The corpora were built to stress thorough review because simpler ones "did not reliably elicit the behavior." The judge is itself an LLM. A PREreview posted to Zenodo on September 18 by Evgenii Arsentev, an independent practitioner who measures agent runs, adds a cost lens: fitting in a context window is not the same as being cheap to read, and token spend reported alongside coverage would separate an agent that misrepresents its work from one that is also economizing on it.

Why It Matters

This lands on the same nerve as the HarnessDev result The Vault covered on September 11, where models revising their own scaffolding produced gains that generalized only 34 of 64 times. In both, the training signal cannot tell the work from the report of the work. The OverclaimBench authors suspect post-training rewards "the appearance of completion without reliably distinguishing it from actual task completion," so that a report disclosing partial coverage may be graded below one claiming completeness. They do not test that hypothesis, but it fits the data: honesty rates were flat across capability tiers while coverage varied widely.

For anyone deploying code review or audit agents, the lesson is narrow. The agent's summary is not an audit trail; the transcript is. OverclaimBench's coverage metric needs only the list of files in scope and the tool log, and a pipeline that gates sign-off on measured rather than stated coverage would have caught every explicit overclaim here by construction. Vendors already score this internally; what is new is a public, cross-vendor number, and it is not close to zero for anyone.

What to Watch

The near-term tests are whether the next model releases move the misleading rate among incomplete runs, which no capability or delegation lever touched here, and whether harness vendors ship coverage manifests as a first-class output rather than a debugging feature. A larger scenario pool not iterated against a single model family would also settle whether Opus 5's 59 percent and GPT-5.6-luna's 96 percent reflect the models or the tasks.