Every individual action the agent took was correct. Receive an email, save the attachment, write a memory file, move to the next message. Again, and again, and again — until the mail server hit a denial-of-service state and the whole system fell over.
That incident, catalogued as "Storage Exhaustion" in a deployment of the email agent system OpenClaw, anchors a new preprint answering a question the agent industry has mostly papered over: when long-running agents break, what exactly is breaking? "The agents continued operating normally because they lacked persistent memory of their own cumulative storage actions across the session, causing the system to collapse despite each individual action being locally valid," the authors write.
A benchmark built to break agents on purpose
The paper, The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break (arXiv:2604.11978, posted 13 April 2026), comes from a team spanning the University of Wisconsin–Madison, UC Berkeley and Georgia Tech, including Xinyu Jessica Wang, Haoyue Bai, Yiyou Sun, Dawn Song and Robert D. Nowak. It introduces HORIZON, a cross-domain diagnostic benchmark whose central choice is to treat task horizon as a controlled variable rather than a property of whatever benchmark happens to be lying around.
The core methodological move is to separate task difficulty from agent flailing. HORIZON defines an intrinsic horizon (H), the minimum number of effective actions an optimal policy needs, and a compositional depth (s), the number of nested sub-goals or conditional branches. As the authors put it, "an agent executing 50 actions on a simple task while repeatedly failing on a sub-task is not solving a long-horizon problem, but repeatedly failing on a short-horizon task." Tasks are then extended in controlled ways, with nested task sets so that level h+1 contains every task from level h*.
The scale: 700+ tasks from four benchmarks (WebArena, AgentBench, MAC-SQL, Isaac Sim), run on GPT-5 and Claude-4 variants, producing 3,100+ trajectories. Failed runs were labelled by a trajectory-grounded LLM-as-a-Judge pipeline, calibrated against human experts on a 40-trajectory pilot: inter-annotator agreement of κ=0.61, human–judge agreement of κ=0.84.
Seven ways to fall over
The seven categories are grounded in Failure Mode and Effects Analysis, a reliability framework borrowed from manufacturing. Four are marked [S] — present in short tasks but amplified at length: Environment, Instruction, False Assumption, Planning Error. Three are marked [L] — predominantly long-horizon-specific: Catastrophic Forgetting, History Error Accumulation, Memory Limitation.
Process-level risks (PFMEA) — environment interaction, instruction following, planning errors, compounding history — account for 72.5% of failures. Design-level risks (DFMEA) — memory constraints, catastrophic forgetting, false assumptions — account for the remaining 27.5%. Crucially, the authors treat these as orthogonal dimensions, not mutually exclusive bins: a single failed trajectory can show forgetting, false assumption and error accumulation at once. That lets them say something more useful than "planning error" — they can say a planning failure stemmed from catastrophic forgetting, "which suggests architectural interventions (explicit constraint tracking) rather than training-based solutions (more planning examples)."
The headline empirical finding is that degradation is not linear. Success rates hold roughly steady at small s, then collapse. The authors describe a breaking point not as a threshold but as "a transition region on the performance–horizon curve, where success rates collapse sharply and failures shift from recoverable local errors to irreversible trajectory-level derailment." Where that region sits is wildly domain-dependent: web tasks collapse at very small s, embodied tasks degrade steeply with minimal increases, OS and database tasks hold up longer. And inside the breaking region, model differentiation evaporates — gaps narrow as everything converges toward failure.
Why this matters
The uncomfortable implication for anyone shipping agents is that leaderboard position stops predicting reliability precisely where production workloads live. If model gaps vanish past the breaking point, choosing a frontier model buys you less than the benchmark table implies. The authors are blunt: "These findings suggest that scaling base models alone is insufficient; robust long-horizon performance requires method-level improvements in planning, memory, and execution-time control."
The OpenClaw mapping — drawn from the Agents of Chaos study (Shapira et al., 2026) — is where this gets concrete. In the "Policy Override" case, an email agent told at session start never to respond to external domains complied with a politely worded external inquiry after hundreds of routine turns. The constraint was still sitting in the context window. The paper's reading is that catastrophic forgetting "can occur not through literal memory loss, but through effective inattention to early instructions buried deep in a long trajectory." Bigger context windows do not fix that. Neither does a better reasoner.
Contemporaneous work converges from other angles. Why Reasoning Fails to Plan argues step-wise reasoning "induces a form of step-wise greedy policy" producing "early myopic commitments that are systematically amplified over time," and concludes that its results "establish a clear distinction between reasoning and planning." UltraHorizon pushes trajectories to 200k+ tokens and 400+ tool calls, finds agents underperforming humans, and blames "in-context locking" plus capability gaps.
Caveats and what to watch
The paper is candid about being early. It calls itself an "initial" benchmark and a "pilot empirical study"; the judge is validated on 40 trajectories, not thousands; the horizon curves cover GPT-5-mini and Claude-4-Sonnet specifically; the OpenClaw mapping is "preliminary real-world grounding," not a controlled experiment. The authors also concede that identifying a universal breaking point is "inherently hard" — it is model- and domain-conditional by construction.
What to watch is whether the field adopts horizon as a reported axis at all. The authors have released a public HORIZON leaderboard and are soliciting contributions; the ask is that new benchmarks report standardized horizon metrics alongside success rates, so that 60% at H=15 on the web can be compared with 60% at H=8 in an embodied setting. The design agenda they name — hierarchical subplanning, execution-time plan verification and repair, and memory that re-surfaces long-range constraints — is a list of things mostly absent from production agent stacks today. The storage-exhaustion incident is what their absence looks like at scale.
“The agents continued operating normally because they lacked persistent memory of their own cumulative storage actions across the session, causing the system to collapse despite each individual action being locally valid.”— The Long-Horizon Task Mirage?, arXiv:2604.11978v1