On September 12, 2026, Yifan Zhang posted a technical report describing the Recurrent Looped Transformer, an architecture whose selling point was that its computational depth would grow without bound as a sequence got longer. The report specified state transitions, cache semantics, gradient paths and a reinforcement-learning replay contract. It contained no experiments. Three days later, on September 15, Leon Lehmann and Casie Nakamura of Empero AI published the experiments. Their conclusion, in the abstract's own four words: "The recurrence does not pay."
Per-token recurrence means the model's output state for one token is fed back in as part of the input for the next token, so computation accumulates across the sequence instead of resetting at each position. In RLT's version, a causal encoder builds a key–value memory and a sequential decoder carries both its final hidden state and its layerwise sliding-window cache forward across every prompt and response token. Zhang's framing was not modest. Announcing the architecture on X, he wrote: "We are at the dawn of Superintelligence. Introducing the Recurrent Looped Transformer (RLT), We now have Transformers with Infinite Reasoning depth."
Lehmann and Nakamura implemented RLT from the published equations, built an exact accelerated execution of the recurrence, and pretrained it on 500.1 million tokens drawn from five English subsets of ULTRAX-PREVIEW at two scales: a tiny configuration at roughly 50M parameters with width 512 and six encoder plus six decoder layers, and a main configuration at roughly 140M parameters with width 768 and twelve plus twelve. Every model saw the same 1024-token windows in the same order, under the same AdamW recipe, peak learning rate 6e-4, global batch of 262,144 tokens. The baselines were a parameter-matched Transformer, a block-matched Transformer, and — the control that does the real work here — an identical RLT with the feedback scale set to zero, which strips out the recurrent path while leaving the encoder, decoder, memory, parameter count and optimization schedule untouched.
The numbers are not close at the larger scale. At ~50M parameters, RLT reached a held-out loss of 3.8845 nats against 3.8841 for the parameter-matched Transformer — a 0.0004 nat gap, which is to say a tie. A conventional Transformer matched by blocks executed per token did better than both at 3.8312. At ~140M parameters RLT was the worst model in the study: 3.6884 nats, against 3.6664 for the feedback-off ablation and 3.6420 for the Transformer. Turning the recurrence on made the model 0.022 nats worse than turning it off.
The long-range claim fared worse. If carried state buys anything, the benefit should widen deeper into a document. At the main scale it inverted: the feedback penalty grew from about 0.006 nats in the first position bucket to roughly 0.025 nats past position 256. The paper's phrasing is blunt — the effect "does not grow with position in the document, contradicting the claim of useful long-range state, and downstream benchmarks show no advantage." LAMBADA accuracy came in at 0.147 for RLT against 0.163 for the Transformer and 0.166 for the feedback-off ablation, with perplexity at 668 versus 579 and 468. Context-length extrapolation at 1024, 2048 and 4096 tokens showed nothing.
Then there is the bill. Sequential decoding kills the parallelism that makes Transformer pretraining tractable. The authors wrote CUDA-graph capture, a custom reverse-traversal backward pass and Triton kernels for causal cross-attention, and verified equivalence with 27 CPU and 22 GPU tests. Even with kernels that sped the decoder up 1.9x, main-scale RLT trained at 8.4K tokens per second per RTX 5090 against 124.1K for the Transformer and 66.0K for the feedback-off variant. Processing the same 500M tokens took about 22 GPU-hours for RLT versus 1.14 for the Transformer. Data-parallel scaling was also poor: going from four devices to eight bought 1.38x aggregate throughput. Empero's summary on X was "Recurrence doesn't pay. Not at this scale."
Why it matters
That last clause is the honest one, and it should be quoted more often than the headline. This is a 50M-to-140M parameter study on 500M tokens, one data mixture, one feedback gate, one window size of 64. It does not establish what per-token recurrence does at 7B parameters or 10T tokens, and the authors do not claim it does. Negative results at small scale have a long history of being wrong about what happens later.
What the study does establish is narrower and more useful: the feedback-off ablation isolates the recurrence from the encoder–decoder memory architecture wrapped around it, and finds that the memory and attention are doing the work while the carried decoder state is a tax. That is a specific claim about a specific mechanism, and it is the kind of control that almost nobody runs.
The broader context is that recurrence keeps getting revived — Universal Transformers, Feedback Transformer in 2020, Geiping et al.'s 3.5B-parameter depth-recurrent model trained on 800B tokens in 2025, and a steady 2026 stream of looped-transformer work. Some of it wins. SMELT, posted to arXiv on September 1, loops the middle half of an MoE stack twice under matched per-token FLOPs, parameters and KV cache, and reports 6.8–18.0% training-FLOP savings on the compute-optimal frontier at up to 54B non-embedding parameters. The distinction that matters is not recurrence versus no recurrence. It is depth recurrence, which preserves parallel training, versus time recurrence, which does not.
What to watch
Whether anyone runs RLT at a scale where the sequential penalty could plausibly be amortized — and whether they publish the feedback-off control when they do. Whether Zhang responds with experiments of his own. Whether the ablation design here gets copied into the rest of the looped-transformer literature, most of which still compares against baselines that differ in more than one thing. And whether the field treats a three-day turnaround from architecture announcement to falsification as a healthy norm or an embarrassment. The code is public at github.com/empero-org/rlt-evaluation, which makes the next move cheap for whoever wants it.
“The recurrence does not pay.”— Leon Lehmann and Casie Nakamura, Authors, Empero AI