A New Benchmark Probes What Robots Actually Remember

Ask a household robot to return three books to their original spots on a shelf, wipe a table for exactly five passes, or fold laundry the way it just watched you do it, and you are asking for something today's most capable robot policies quietly fail at. Not because the arms are clumsy or the cameras are blind, but because the moment an object slips behind another one, or a count creeps past two, the machine forgets. A new benchmark accepted for oral presentation at ICML 2026 sets out to measure exactly how much robots remember, and the early numbers are humbling.

The paper, "RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies," comes from a team at the University of Michigan's SLED lab and Stanford, including Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Chelsea Finn, Nima Fazeli, and Joyce Chai. It was selected as an ICML 2026 Oral, a slot reserved for roughly the top 0.7 percent of submissions. The work lands in the middle of a broader wave of 2026 research on memory in agents and embodied AI, and it makes a pointed argument: the generalist robot policies everyone is excited about have been graded on tasks that never really tested whether they can hold a thought.

What generalist policies forget

The models in question are vision-language-action (VLA) systems, the robotics cousins of large multimodal models. They take in camera images and a natural-language instruction and output motor actions. The dominant ones, like the π0.5 backbone RoboMME builds on, are trained on enormous piles of demonstrations and can generalize across many manipulation tasks. But as the authors note, most existing manipulation benchmarks only "implicitly involve temporal memory but rarely require it." A policy that looks only at the current camera frame can already ace them, because the scene itself tells the robot what to do next. That is perception, not memory.

RoboMME is designed to break that shortcut. Drawing on classical cognitive theory of human memory, the benchmark carves memory into four dimensions: temporal (accumulating and ordering events), spatial (tracking where objects went, including under occlusion), object (recognizing that a thing referred to earlier is the same thing now), and procedural (reproducing a motion you were just shown). Those map to four task suites named Counting, Permanence, Reference, and Imitation, each with four tasks. Altogether there are 16 long-horizon tasks backed by 1,600 demonstrations and about 770,000 high-quality timesteps, a scale the authors argue prior efforts like MemoryBench (three near-solved tasks) and MIKASA-Robo (short-horizon, thin on demonstrations) never reached.

A leaderboard with a lot of empty space

To find out what actually helps, the team built a family of 14 memory-augmented variants on the same π0.5 backbone, so differences reflect the memory design rather than the underlying model. They sorted memory into three flavors: symbolic (history summarized as language subgoals or tracked points), perceptual (banks of visual features and multi-frame tokens), and recurrent (compressing the past into a fixed-size latent state). Each neural memory could be wired in three ways: as extra context, as a modulator on the action expert, or as a dedicated memory expert.

The headline result is that there is no headline winner. "No single memory representation or integration strategy consistently performs well across all tasks," the authors write, warning that "conclusions from prior work may not generalize." Effectiveness was sharply task-dependent: symbolic memory shone at counting and short-horizon reasoning, while perceptual memory mattered most for time-sensitive, motion-heavy behavior. The best all-around variant paired perceptual memory with a modulator design and landed at a 44.51 percent overall success rate, the strongest score in the suite and the best balance of performance and compute.

That number is worth sitting with. An oracle version fed ground-truth symbolic state reached 84 percent, showing the tasks are solvable and that language is a powerful vehicle for high-level reasoning when the information is handed over cleanly. Humans scored 90.5 percent. The gap between the best learned policy and human performance, roughly 46 points, is the size of the problem RoboMME is trying to make visible.

Why memory is the frontier

For years, embodied AI progress has been narrated through scaling: more data, bigger backbones, broader task coverage. RoboMME is part of an argument that the next bottleneck is architectural, not just bigger. A robot that cannot remember cannot do the mundane, extended chores that make home and warehouse robots economically interesting, the ones defined by counting, occlusion, reference, and imitation over minutes rather than seconds. The paper's most useful contribution may be diagnostic: by isolating four memory types and holding the backbone fixed, it turns a fuzzy complaint ("robots forget") into a measurable, comparable axis of progress.

What to watch: whether the RoboMME leaderboard, now public with code and demos, starts to climb as labs stress-test their own memory modules against it, and whether the sim results transfer to real hardware. Also watch for hybrid designs that combine symbolic counting with perceptual tracking, since the paper strongly implies no single mechanism will close the gap to human-level recall alone.