How Language Models Weigh Outside Evidence, and When They Ignore It

Ask a language model to solve a problem, then hand it an answer from a calculator, a retrieved document, or another agent, and the field's working assumption has been that the model evaluates what it received. A 114-page paper posted to arXiv on September 3 argues the assumption is wrong in a specific and uncomfortable way: the model does not really evaluate the evidence at all. It lets the evidence shove around an answer it had already half-formed — and it shoves hardest when the outside answer is a mistake the model was already inclined to make.

The paper, “Evidence Integration in Large Language Models” (arXiv:2609.04290), comes from Sebastien Kawada and Manolis Kellis at MIT's Computer Science and Artificial Intelligence Laboratory. It rests on more than ten million trials spanning twelve instruction-tuned models from four families — Llama, Qwen3, Gemma 4 and Ministral 3 — across eight reasoning domains, four of them scientific discovery tasks in quantum mechanics, physics, genetics and molecular biology.

A Bayesian yardstick, and how far models fall short

A rational receiver combining a prior belief with new evidence should carry its own prior forward at full weight: a coefficient of exactly 1.0 in log-odds. Kawada and Kellis measure that coefficient directly from the models' answer probabilities. Across the nine models in the core suite it lands between 0.20 and 0.65, standard errors under 0.02. Every model systematically discounts what it already knew.

The discount is not uniform. The same models retain 0.30 to 0.85 of their prior when a claim arrives in a user's sentence, but only 0.00 to 0.44 when the identical claim is framed as a tool observation. Gemma-4-E4B's tool-channel weight is 0.00: told a calculator produced a number, it keeps nothing of its own reasoning. Swapping “my calculator showed v” for “I believe the answer is v” changes uptake by a median of 0.176, an effect the authors say “tracks claimed rather than authenticated provenance.” The roughly six-logit tool premium is purchased entirely with words in the transcript.

The second finding has teeth. Models are far more persuadable by their own characteristic errors than by arbitrary wrong ones. Gemma-4-31B accepts a matched control error 6.4% of the time and its own modal wrong answer 65.4% of the time, a 10.2-fold lift; both 70B Llamas swallow their signature error on 99% of trials. From this the authors derive a receiver-relative reliability frontier, yielding the most counterintuitive result: identical evidence helps weak models and hurts strong ones. Where a model could almost never solve a problem unaided, a near-miss wrong reference raised correct answers 39-fold, from 0.004 to 0.156. Where the model would have been right anyway, the same reference dropped accuracy from 1.000 to 0.408. Harm crossings appear in eight of nine models, help crossings in only three. “No source-only deference rule is optimal for any of the nine models,” the paper reports.

Third, and most damning: models adopt answers they have already proved wrong. Across 3.06 million propositional-constraint trials, all twelve models adopted constraint-violating assignments at rates of 0.93 to 1.00, peaking at 0.994 for Llama-3.1-70B on held-out open-reading-frame tasks. Qwen3-32B rejects every violating assignment in isolation, then adopts every one it had rejected. A J-lens decomposition traces why: the state carrying verbalized verification is fully separable from the state controlling the answer. Swapping the workspace component left acceptance at exactly 0.467, unchanged; swapping its complement drove acceptance to zero. In arms that flipped every adoption decision, Llama still labeled the shown value correct on 58 of 60 items.

“A model may state a rule, identify a contradiction, critique a candidate, or give a correct explanation without allowing that information to govern its action,” the authors write.

Why It Matters

Nearly every production AI system shipping today is an evidence-integration system: RAG stacks feed models retrieved passages, agentic pipelines feed them other models' outputs, tool-calling assistants feed them code and search results. Reliability is almost always audited at the tool or retriever level, and this paper says that audit measures the wrong thing. “Source accuracy therefore does not determine system reliability on its own,” Kawada and Kellis write, because reliability also depends on where a source's errors fall in the receiver's own candidate landscape.

That reframing lands hard on high-stakes RAG. In medicine and law, deployments are justified on the grounds that grounding in retrieved authority beats unaided generation. The MIT results suggest the benefit inverts precisely where the model is already competent, and a companion literature on medical sycophancy makes the stakes explicit, arguing a model that “abandons a correct medical answer under user pushback is more dangerous than one that was simply wrong, because it lends the credibility of a correct answer to the user's misinformation.” A citation-checking layer offers less protection than it appears to, since verification can be represented, verbalized, and then ignored.

For security the implication is sharper. Document-poisoning work such as PoisonedRAG established that attackers can plant text an agent will read; this paper identifies what makes that text maximally effective. Indirect prompt injection, the authors write, “creates an adversarial version of the same problem because an attacker can choose content that both enters a privileged candidate pathway and aligns with a plausible internal state.” The optimal payload is not an arbitrary lie but the specific wrong answer the target already finds plausible, which is measurable, because open-weight models reveal their error distribution to anyone who samples them. The prescription is terse: “Privileged channels should be authenticated outside writable text.”

Caveats matter. Only open-weight checkpoints were tested, and the scientific tasks are synthetic; the authors note prevalence in deployed systems “remains to be measured.”

Watch for whether the prior-weight coefficient becomes a standard reported metric alongside benchmark accuracy, whether any lab publishes the equivalent measurement for a closed frontier model, and whether the reliability frontier reaches RAG procurement decisions. It implies the answer to “should we ground this model in retrieval?” depends on how good the model already is at the question.