LLMs Can Reason Deep. A New Benchmark Asks Whether They Can Reason Wide.

The last two years of language-model progress have been a story about going deeper. Chain-of-thought, test-time compute, "thinking" modes — nearly every headline benchmark rewards a model for extending one line of inference further than before. But there is a second kind of reasoning almost nobody scores: holding several unrelated fragments in mind at once and collapsing them into the single concept they all point at. That is what a crossword clue does, and what a doctor does with four unrelated symptoms. According to a new benchmark, frontier models are noticeably worse at it than their deep-reasoning scores suggest.

From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models (arXiv:2608.10444) was posted on August 11, 2026. Si'an Xie of Beijing University of Posts and Telecommunications and Jiaxun Liu of Peking University share first authorship, alongside Biao Yang, Wei Yuan, Fan Yang and Tingting Gao of Kuaishou Technology and Ming Wu of BUPT.

"Existing LLM benchmarks primarily emphasize reasoning 'depth,' evaluating step-by-step logical deduction and procedural reasoning," the authors write in their introduction. "In contrast, the evaluation of reasoning 'breadth' — the ability to aggregate dispersed semantic signals and perform abstract conceptual convergence — remains largely unexplored." Their one-line diagnosis: "A model may therefore reason deeply along one path while still failing to combine information available across several paths."

A party game, turned into a test

The benchmark, MPAR-Bench, borrows its format from the cooperative board game Just One, in which players each write a single-word clue for a hidden target and a guesser recovers it from the surviving hints. Each of the 1,000 items — 500 English, 500 Chinese — gives a model several deliberately dissimilar clues and asks for the word underneath.

The construction is designed to sidestep contamination. Target words come from public lists (RAT-derived vocabulary and Just One word cards), but every clue set is written from scratch by a multi-agent pipeline in which each LLM agent is assigned a distinct "association angle," and a judge agent strips out clues that are synonyms, homophones or near-duplicates of the answer. A Qwen3-Embedding-8B pass then filters for clue-to-clue diversity. Two NLP master's students hand-checked a random 250-item sample: 92.8 percent were judged to have a unique, unambiguous target (95 percent Wilson interval 88.6–95.4 percent).

Scoring goes beyond exact match: the team also reports ANLS (character-level edit similarity), fastText embedding cosine similarity, and a reasoning-trace check split into logical and factual verification. Every item comes in a perturbed variant too — clue masking, order shuffling, distractor injection, and multi-step clues.

Where the models land

In thinking mode, Gemini-3.1pro leads at 86.8 percent accuracy in English and 72.2 percent in Chinese, followed by Sonnet-4.5 (79.0/67.4), GPT-5.2 (77.6/64.4), Qwen3-max (73.2/65.0), Kimi-k2 (71.6/57.4), Seed-2-pro (71.4/64.6) and DeepSeek-v3.2 (69.8/61.2). Turn thinking off and the ordering scrambles: Sonnet-4.5 tops both languages (70.4/68.8) while GPT-5.2 drops to 59.6 in English.

The perturbations cost 9 to 18 percentage points in English and 5 to 12 in Chinese — but the damage is wildly uneven. Order shuffling is nearly free; sometimes accuracy goes up. Clue masking is brutal, costing an average of 20.0 percent relative to the shuffled setting in English. Distractor injection knocks Qwen3-max and Seed-2-pro down by more than 28 percent from their standard scores, evidence, the authors write, that "spurious semantic correlations introduced by irrelevant clue words can effectively derail the model's reasoning trajectory."

The trace analysis is the most uncomfortable result. Models frequently reach the right word by a bad route: in the English thinking setting, DeepSeek-v3.2 shows a 43.60 percent logical error rate in its traces, Kimi-k2 40.56 percent, Qwen3-max 38.21 percent — and even Gemini-3.1pro 20.61 percent. Factual error rates are far lower (11.52 percent for Gemini, 13.60 percent for DeepSeek). More thinking does not reliably help. Of Qwen3-max's 174 wrong Chinese answers, the correct target appeared in the reasoning trace 87.4 percent of the time before being talked away; in one cited case the answer was Philosophy and the model settled on Plato. Qwen3-max also burned 9,983 tokens per English item on average — seven times GPT-5.2's 1,355 — while scoring four points lower.

Why this is worth watching, and where it is thin

The attempted fixes should give labs pause. Scaling the Qwen3 family from 0.6B to 32B improved accuracy monotonically until 32B, which regressed from overthinking. A structured-reasoning prompt lifted Seed-2-pro by just 1.0 points in English and 3.2 in Chinese. "Improving breadth appears challenging," the conclusion states. "Scaling model size, adding reasoning strategies, and iterative feedback each bring only partial gains, suggesting that reasoning breadth may be a capability that current training paradigms do not naturally optimize for."

The caveats are substantial. This is a single-group result on a benchmark the same group built, with no external replication. The clues are LLM-generated and largely LLM-judged, which risks baking in the quirks of the generating models. A word-association party game is a clean probe but a narrow one, and the paper does not show that breadth scores predict anything downstream. The English–Chinese gap is confounded by design: the Chinese subset deliberately adds idioms, character-level properties and internet memes, so it is not a like-for-like comparison. And the models were queried through APIs at default settings, which makes the per-model rankings softer than the tables imply.

The paper's own limitations section is light on self-criticism, conceding only that there "remains some room to further broaden its coverage and ecological validity" and pointing to "more diverse and interactive settings" as future work.

Still, the framing is the contribution. If reasoning breadth really is orthogonal to the depth axis everyone optimizes, a lot of leaderboard movement over the past year has been measuring one dimension of a two-dimensional space. The authors say both subsets and the evaluation scripts will be released under an MIT license for non-commercial academic use on publication. Watch for two things: whether anyone independently reproduces the perturbation gaps, and whether any lab starts reporting a breadth number next to its math scores.