Backprop-Free Training Recovers 92% of RL's Gains for Tool-Using LLM Agents, on a Single 24GB GPU

The received wisdom in LLM post-training is settled: if you want an agent that plans across dozens of turns, calls tools, and recovers from its own mistakes, you reach for reinforcement learning. A paper posted to arXiv this month argues that the wisdom is partly an artifact of who can afford the hardware — and that an optimization technique which never computes a gradient can get most of the way to RL on less than an eighth of the memory.

The paper, Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training (arXiv:2608.02391), was submitted on August 3 by Zhiyuan Wang, Shengcai Liu, Jiahao Wu, Ning Lu, Hui Ouyang, Shaofeng Zhang, Haoze Lv and Ke Tang, with Liu and Tang listed as corresponding authors. It is marked as a submission to AAAI 2027, and the code is public.

Its diagnosis is blunt. “Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive,” the authors write. That is the whole problem in one sentence. A math agent that issues three search calls and five Python calls before answering can fill a 128,000-token context window, and backpropagating through that trajectory means holding activations, gradients and optimizer state for every one of those tokens. In the authors’ accounting, full-parameter GRPO on a 4-billion-parameter agent needs eight 48GB GPUs — 384GB of aggregate memory — before it will run at all at that context length. Even LoRA-based GRPO, the standard memory workaround, has a theoretical requirement “more than 30×” that of evolution strategies at 128K.

Evolution strategies sidestep this entirely. Instead of differentiating the model, ES perturbs its weights with random Gaussian noise, runs the perturbed copies forward on a batch of tasks, and nudges the weights toward whichever perturbations scored well. No backward pass, no optimizer state, no stored activations. As Xin Qiu and colleagues at Cognizant AI Lab put it in the ICML 2026 paper that revived the approach for language models, “ES is an inference-only fine-tuning mechanism, where the model weights are never differentiated, only evaluated.”

The catch, and the fix

ES has a well-known weakness: it is slow. Random search over four billion dimensions burns forward passes. The new paper quantifies the pain — standard ES needed an estimated 480.60 GPU-hours to reach its best checkpoint, “more than six times the estimated 78.61 GPU-hours for full-parameter GRPO.” Saving memory by spending a week of compute is not a trade most teams will take.

CoPES — Cooperative Parameter-subspace Evolution Strategy — attacks the time cost rather than the memory cost. At each training step it randomly partitions all model parameters into K disjoint subspaces (K=4 in the main experiments), allocates a slice of the perturbation budget to each, and perturbs only the parameters inside a given subspace while the rest of the model supplies unchanged context. Because the supports are disjoint, the resulting update directions are mutually orthogonal. Because the partition is resampled every step, no parameter gets permanently siloed. The perturbation scale is rescaled by the square root of K to keep the effective step size constant, and all rewards are standardized against a single shared pool rather than per-subspace statistics.

The results, on a Qwen3.5-4B tool-using agent trained on MATH and evaluated across five benchmarks: under a matched 16-step budget, CoPES hit 92.57% pass@1 on GSM8K where standard ES managed 46.88%, and 88.29% on MATH-500 against 78.88%. “Relative to Qwen3.5-4B without post-training, CoPES recovers 92% of the validation-accuracy gain achieved by full-parameter GRPO, compared with 67% for standard ES,” the authors report. On the hardest benchmarks — AIME 2024 and 2025 — they note that as k increases, CoPES actually surpasses full-parameter GRPO on pass@k despite the fixed budget. On multi-hop question answering, CoPES beat standard ES on all nine benchmark-metric combinations and edged out GRPO on MuSiQue.

The feasibility claim is the one worth remembering: CoPES trains on a single 24GB GPU. GRPO, in the same tested configurations, does not.

Why it matters

This is not the triumphalist result the ES revival has been trading on. CoPES does not beat RL; it recovers 92% of it, on a hardware footprint roughly one-sixteenth the size. That framing — accessibility rather than supremacy — is arguably the more consequential one. Agentic post-training has been effectively gated behind multi-node clusters, which means the set of people who can shape how tool-using agents behave has been small. A method that fits on one consumer-grade card widens that set considerably.

It also lands in the middle of a live argument. Qiu and his co-authors claimed ES beats RL on reward hacking and run-to-run stability, reporting that GRPO showed “15.5× higher standard deviation.” But William Hoy of the University of Miami, working with Binxu Wang and Xu Pan of Harvard, published a careful counterweight in April: ES weight updates are roughly 90 to 107 times larger in norm than GRPO’s, and ES lost 3.7 points of MMLU across sequential training where GRPO gained 0.8. Their conclusion is that “gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation.” Cheap agent training that quietly erodes general capability is not obviously a bargain.

What to watch

The CoPES ablations are a warning as much as a validation. Pushing to K=8 collapses performance — GSM8K pass@1 drops from 92.57% to 37.32% — because the per-subspace population falls to five. The authors recommend at least ten perturbations per subspace, which means the method’s scaling story depends on population budget, not just parameter count. Each configuration was also run once, on a single seed.

The open questions: whether CoPES holds at 30B or 100B parameters, whether the forgetting effect Hoy and colleagues documented compounds in agentic settings where trajectories are long and rewards sparse, and whether anyone reproduces the pass@k crossover on AIME. Given that the code is on GitHub and the hardware bar is one 24GB card, we should find out quickly. That, more than any single benchmark number, is the point.