To Train a Web Agent Cheaply, Mine Its Own Failures

Most of what a web agent does is boring. Scroll. Click the obvious link. Scroll again. On a typical Online-Mind2Web task, a well-trained 8B browsing model takes around sixteen actions; one or two of them decide the outcome. That asymmetry is a headache for anyone improving such an agent with reinforcement learning, because RL spends its budget uniformly across whatever you feed it. Feed it whole trajectories and most of that budget goes to re-teaching the model things it already does perfectly.

A preprint led by Chengbo Liu, filed as arXiv:2608.00335 and circulating through the August cs.AI listings, is built around that complaint. Their system, RMSWeb, trains vision-language web agents on Qwen3-VL-Instruct backbones at 8B and 32B, and its central move is to throw away almost all the training data before the RL stage begins.

RMSWeb has three parts. The first governs collection. A teacher model, GPT-5.1, attempts tasks derived from anonymized user queries sampled from a commercial search platform and rewritten by an LLM into executable browsing objectives. When the teacher fails, a reflection handler distills what went wrong and what a recovery would look like into a short note, which is injected into every browser state on the next attempt. The loop stops after two successes or five rounds; the notes are stripped before trajectories become training records, so the student never learns to lean on hints it will not have at test time. The filtered corpus flattens into more than 150,000 per-step supervised examples.

The second part is the mining. For each task, the authors pair a verified success with a run of the same task that failed or wandered, then walk backwards for the last point where the two agents stood in comparable states, judged by URL path, visible DOM elements, agent memory and progress, but chose different actions. An LLM judge tags the divergence using a failure taxonomy derived from watching the frozen fine-tuned model on a held-out split; the label acts only as a soft search prior. What survives is roughly 4,500 critical-state examples, and only the successful side of each pair enters RL.

The third part, Salvage-DS, is the optimizer. Group-relative RL of the GRPO family stalls in specific ways once the supervised policy is strong: exact string matching on web actions collapses different near-misses to the same score, leaving no contrast to learn from, while generic text similarity rewards clicking the wrong element because its label looked close enough. RMSWeb uses a polarized reward tuned to web-action semantics, then gates each sampled group twice, once for reward contrast and once for whether any rollout was actually competent. Groups failing either gate, normally discarded, are routed to a plain supervised salvage anchor on the correct action, with a KL brake against the frozen policy.

The numbers, under the authors' own fixed Browser Use harness and averaged over three pass@1 runs: the 8B base model scores 51.13 on WebVoyager, 36.56 on Online-Mind2Web and 23.31 on WebTailBench. Fine-tuning lifts that to 82.65 / 59.44 / 55.39. Adding mined RL reaches 86.39 / 66.44 / 57.74. At 32B the final figures are 87.64 / 68.11 / 59.28. The ablations do the real arguing: running the same RL over the full corpus rather than the mined one loses 2.0 / 5.7 / 4.8 points, removing the salvage anchor loses 1.7 / 5.9 / 1.0, and a DAPO baseline handed identical mined data and reward still trails by 1.5 / 3.8 / 3.0. Reflection, tested alone, leaves accuracy roughly flat on the two shorter benchmarks, adds 5.3 points on WebTailBench, and cuts actions on solved tasks by 12.9 to 19.7 percent.

Cross-system comparisons deserve care, and the authors say so: a scaffolded GPT-5 agent reports 90.6 on WebVoyager under a different harness. RMSWeb's claim is narrower, and it is a preprint with no peer review.

The team is unusually direct about the ceiling it accepted. "Our RL is offline: rewards come from step-level matching against ground-truth actions, not from executing the agent in a live environment," the paper states. "This is simpler and cheaper but caps the ceiling."

Why It Matters

Two costs dominate this field, and neither is the GPU bill for gradient steps. The first is trajectory acquisition: MolmoWeb, one of the open agents in RMSWeb's comparison table, was post-trained on 278,000 curated trajectories, each one a teacher model driving a real browser through a multi-step task. The second is online rollout and judging. OpenWebRL, a UIUC and Microsoft framework that takes the opposite approach and runs RL directly on live websites, reports that distilling its trajectory judge into an 8B model saved roughly $545 per experiment, and calls live browser interaction slow, brittle and non-stationary. Sites change under you; a benchmark run in April is not the same benchmark in August.

RMSWeb's bet is that most of that can be skipped. Its RL stage never opens a browser: 4,500 stored states scored against verified actions. Step efficiency compounds at deployment too, because every action means another screenshot and DOM dump through a vision-language model. Cutting steps by a fifth cuts the per-task inference bill by roughly as much.

What To Watch

Whether the artifacts land. The authors promise model weights and an inference framework; the training data is undergoing compliance review, which, given its provenance in commercial search logs, is not a small caveat. Watch also whether failure-mode mining survives contact with online RL. Liu's team names live-browser, outcome-rewarded training as the obvious next step, and if a mined curriculum still helps once real task rewards are available, the idea travels beyond browsers. And watch the harnesses: until this field agrees on a shared live-web protocol, every leaderboard claim here carries an asterisk.