New Benchmark Finds Top Video Models Solve Barely Half of Visual Reasoning Tasks

A new benchmark from a team spanning the University of Illinois Urbana-Champaign, Tsinghua University, MIT, the University of Waterloo, and Microsoft Research finds that even the best commercial video-generation models solve barely half of a set of "visual reasoning" tasks — and that when they do get things wrong partway through a generated clip, they almost never fix the mistake. The benchmark, called VGI-Bench, was posted to arXiv on August 20 and revised to its current version on August 26, and it delivers one of the more sobering evaluations yet of the idea that video models are quietly becoming general-purpose visual reasoners.

VGI-Bench asks video-generation systems to do something more demanding than produce a pretty clip: given a starting image and a text prompt, the model has to generate a video that actually solves a task — untangling a knot, threading a maze, stacking objects, completing a puzzle — while obeying explicit rules along the way. The benchmark contains 27 tasks and 810 task instances, split across four domains (Visual Organization, Physical Manipulation, Structured Puzzles, and Spatiotemporal Dynamics) and tagged with seven underlying skills such as spatial reasoning, planning, physics, and topology. Crucially, success is graded not just on whether the final frame looks right, but on whether the video's intermediate steps stayed valid throughout — a "process-sensitive" design the authors say prior video-reasoning benchmarks largely lacked.

The headline number is blunt. "Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable," the authors write, "with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria." That top score came from a closed commercial model; open-source systems trailed badly, with HunyuanVideo-1.5 and Wan2.2 scoring 19.1% and 21.6% overall, versus 44.0% for Kling 3.0 and 44.4% for MiniMax-H3. Every model struggled most with the "Structured Puzzles" domain — tasks that require tracking multi-step rules and object states — and the two weakest skill dimensions across the board were topology (preserving connectivity) and temporal reasoning (tracking how state changes over time).

The most striking finding sits inside the denoising process itself, the step-by-step way diffusion-based video models build a clip from noise. The team decoded intermediate frames at multiple points during generation on four open-source models and tracked whether the emerging "solution state" shown in the video flipped from wrong to correct, correct to wrong, or wrong to a different wrong answer, as generation proceeded. The results undercut a popular assumption that later denoising steps let models "self-correct" earlier mistakes. "The solution state does change during denoising, but almost never toward a correct one," the authors write. Genuine self-correction — a wrong intermediate state flipping to the right one — stayed below 1% at every stage measured, while a wrong state flipping to a different wrong state occurred roughly 23% of the time between steps 4 and 10, and nearly 25% of the time between steps 10 and 20. By the second half of generation, the depicted solution state was stable 90.6% of the time — meaning models were mostly locking in and polishing an early hypothesis, correct or not, rather than reasoning their way out of an error. "Revision therefore moves between wrong solutions rather than toward the right one," the paper concludes, "later steps mostly lock in and refine the early hypothesis."

The researchers also tested whether giving models an "oracle prompt" — a fully explicit, step-by-step description of the intended solution — closed the gap. It helped some closed-source models modestly, but even with the entire solution spelled out in text, models frequently still failed to render the full correct trajectory, particularly HunyuanVideo-1.5. Separately, the team found that fine-tuning smaller open models on a million synthetic training examples (using a released dataset called VBVR) produced large gains on tasks structurally similar to the training data, but far smaller or even negative gains on tasks that didn't overlap with what the model had been fine-tuned on — evidence that current scaling recipes for video reasoning transfer unevenly rather than generalizing broadly.

Why It Matters

VGI-Bench adds hard numbers to a fast-growing but under-scrutinized claim in AI research: that video generators, trained purely to predict pixels, are picking up something like general visual reasoning as a side effect. The paper's contribution is less "video models can reason" and more "here is precisely where and how they can't." A ceiling of 51% on a benchmark the authors deliberately calibrated to be only partly solvable is not a failing grade by design, but it does show current systems are far from dependable visual problem-solvers.

The study has real limits the authors flag themselves. Tasks are capped at 5–10 seconds because that is roughly what current video models can reliably generate, so nothing here speaks to longer-horizon planning. The benchmark only covers image-to-video generation at a fixed 16:9 aspect ratio, is English-only, and — as with any benchmark built around 27 hand-designed task types — is "representative rather than exhaustive," not a full map of visual reasoning. It also has not been through peer review; as an arXiv preprint, its evaluation methodology (an automated Gemini-3-Flash judge, validated against human ratings) is the kind of design choice reviewers typically probe hard.

What to Watch

The authors say they will release VGI-Bench's code and data, which should let other labs check whether newer video models — or different training recipes for existing ones — actually close the "wrong-to-wrong" gap rather than just nudging up the headline score. Worth watching too is whether video-model developers respond by targeting the specific weak points this paper isolates, topology and multi-step temporal tracking, or whether progress stays concentrated in the domains that already score highest.