'Prefix Sliding' Makes Long-Reasoning Models 3x Faster by Forgetting Their Own Work

A reasoning model that thinks for an hour drags a growing anchor behind it: every new token it writes forces it to re-read every token that came before, so the cost of the next word climbs for as long as the model keeps talking. A new preprint from researchers at Stanford, the University of Washington, UC Santa Barbara, and Prime Intellect proposes a fix that sounds almost too simple to work — just let the model forget most of what it already said. The method, called Prefix Sliding, keeps only the original instructions and a moving window of recent tokens in memory, discarding everything in between. According to the paper, it makes existing models "3x faster while maintaining performance," with no retraining required.

The paper, titled "Prefix Sliding for efficient test-time scaling," was posted to arXiv on August 26, 2026 (arXiv:2608.26070) by a large author list led by Niklas Muennighoff, with Zhengyang Wang, Zeyi Chen, Weijia Shi, Binyuan Hui, John Yang, Dapeng Jiang, Mika Senghaas, Fares Obeid, Johannes Hagemann, Sami Jaghouar, Ludwig Schmidt, Percy Liang, Jason Wei, Andrew Y. Ng, Luke Zettlemoyer, Yejin Choi, and Mike Lewis credited as co-authors or advisors, spanning Stanford, UC Santa Barbara, the University of Washington, and the AI training startup Prime Intellect. Like all arXiv postings, it is a preprint that has not yet been through formal peer review.

The core idea rests on a straightforward observation about how reasoning models actually use their own output. "We find most intermediate reasoning tokens lose importance as the model continues reasoning," the authors write in the abstract. "This calls into question whether retaining them is worth the cost." The paper backs this up with an attention analysis on Qwen3-1.7B solving AIME25 problems: the first few tokens of a prompt function as "attention sinks" that soak up a disproportionate share of probability mass, the `<think>` delimiter that opens a reasoning trace draws unusually high attention, and then attention on the middle of long traces drops off — only spiking again for the tokens immediately preceding whatever the model is about to generate next.

Prefix Sliding turns that pattern into an inference strategy. The system retains two things: the "prefix" — the system instructions, tool definitions, and task prompt, which the paper says can be as small as 100 tokens for a simple task — and a sliding window of the most recent tokens, tested at sizes from 512 up to 16,384. Once the window fills, older intermediate reasoning tokens are simply dropped from memory, and the model keeps working from the prefix plus whatever fits in the window. Because the total context a token has to attend to never grows past prefix-size-plus-window-size, the cost of generating each new token stays constant no matter whether the model has already produced a thousand tokens or a hundred thousand. The authors compare this directly to full attention, where "the cost of each new token grows linearly with the number of already generated tokens," and argue that only a bounded-cost approach like theirs can support what they call "infinite test-time scaling" — models that might reason continuously for extended periods.

The tradeoffs are where the paper gets specific rather than promotional. Benchmarking on an Nvidia H100 GPU using vLLM and FlashAttention, the researchers found that Prefix Sliding's token-generation speed initially dips below full attention during a "warm-up phase" before the window fills, then stabilizes around 5,000 tokens per second — essentially matching a plain sliding-window kernel, while full attention keeps slowing down indefinitely as its context grows. Accuracy holds up on GPQA, MATH500, and AIME25, the three benchmarks used for evaluation, averaged across 64 runs per problem (avg@64) for statistical confidence. But it is not free everywhere: on LiveCodeBench, the paper reports that Prefix Sliding "requires a window of at least 16384 to match full attention," because models sometimes write code far ahead of explanatory comments, and a too-narrow window can push the start of a function outside the model's view before it finishes writing it. On short tasks like HealthBench, where the average response needs only about 2,086 tokens, the paper acknowledges "there is little room for any speed-up" because the window rarely fills enough to start evicting tokens.

The team also tested using Prefix Sliding during reinforcement learning training, via GRPO, rather than only at inference time. Because backpropagating through reasoning traces of hundreds of thousands of tokens can cause trainer out-of-memory errors, they used "truncated backpropagation" — computing gradients using roughly four times the sliding-window size of trailing context rather than the full trace. In one described run, a 100,000-token trace with a 2,048-token window meant only the last 8,192 tokens were sent to the trainer, with loss computed only on the final 2,048. This let the researchers train a 7B model with RL rollouts extending "beyond 100,000 tokens" without truncating and discarding oversized generations, a common workaround in current RL training pipelines.

In ablations against two commonly used alternatives — deleting everything except the last k tokens, and periodically summarizing the reasoning so far — the paper reports Prefix Sliding "provides the best performance efficiency trade-off." Both alternatives, the authors argue, waste compute reprocessing tokens twice and introduce irregular, spiky memory usage, whereas Prefix Sliding "adds only one hyperparameter: the size of the sliding window." The paper situates the approach against established KV-cache compression techniques: StreamingLLM, which keeps only a handful of fixed initial tokens as attention sinks, and H2O, which tracks and retains "heavy hitter" tokens based on cumulative attention scores. Prefix Sliding differs by preserving the entire prompt as a contiguous prefix rather than a handful of sink tokens, on the theory that task instructions — not just early tokens generically — are what the model needs to keep reasoning coherently.

Why It Matters

Inference cost, not training cost, is increasingly the bottleneck for deploying reasoning-heavy AI systems, and quadratic attention is the mathematical reason why. Every additional minute a model spends "thinking" through a hard problem multiplies the compute bill, which is part of why frontier labs have been racing to ship compaction and context-management features — the paper itself notes that summarization-style context management is "used by models like Opus 4.6, GPT 5.4, and Composer." If a training-free technique really delivers a 3x speedup with preserved accuracy on standard math and science benchmarks, it lowers the cost floor for every product built on long chains of thought, from coding agents to research assistants. It also matters for the emerging category of long-horizon agents that are meant to work autonomously for hours or days: those systems are currently bounded by how much context a model can afford to carry, and a bounded-cost-per-token method is a prerequisite for agents that don't get slower — or run out of memory — the longer they work.

What to Watch

The paper is explicit about its own limits: it compares only against methods that work on existing pretrained transformers without architecture changes, it hasn't been tested beyond 7B-parameter models, and coding tasks like LiveCodeBench show the technique needs a much larger window to avoid losing information the model wrote earlier and needs later. Worth watching next: whether independent teams reproduce the 3x figure on other model families and larger parameter counts, whether frontier labs adopt something like Prefix Sliding in production serving stacks, and how the approach handles messier agentic scenarios — reading large tool outputs or juggling multi-turn user instructions — that the authors flag as open problems rather than solved ones. The authors have released code at github.com/Muennighoff/prefix-sliding, which should make third-party verification straightforward.