A Tool That Compresses Tokens Made AI Coding More Expensive, Not Less

A Tool That Compresses Tokens Made AI Coding More Expensive, Not Less

The pitch is irresistible and the demo is genuinely satisfying: a small Rust binary sits between your coding agent and your shell, turns eleven lines of `git status` porcelain into three, and reports that it just saved you 90 percent of your tokens. What it does not report is your invoice. A Warsaw-based startup spent several days and more than $1,500 in API spend finding out, and the invoice went up.

Quesma published results on September 11 from a paired benchmark of RTK — Rust Token Killer, a CLI proxy with nearly 80,000 GitHub stars that intercepts shell commands and returns a terser version of the output. Across 1,740 attempts on Terminal-Bench 2.1, RTK made Claude Code running Fable 5.0 about 1 percent more expensive per task, a result statistically indistinguishable from zero, and made OpenCode running DeepSeek V4 Pro 0813 17 percent more expensive. Pass rates slipped in both arms: one point for Fable, two for DeepSeek.

Authors Bartosz Kotrys and Jacek Migdal scheduled every task five times without RTK and five times with it, on the same model route, platform and task-specific timeout. After dropping four Fable security tasks that drew refusals, the comparison covered 85 Fable tasks and 89 DeepSeek tasks. They stayed on Terminal-Bench 2.1 rather than the harder 3.0 or 4.0 releases, since cost only matters on tasks that pass.

The first chart looked promising. Summed across everything, the total bill fell 5 percent for Fable and rose 5 percent for DeepSeek. But totals are outlier-dominated, so Quesma reweighted to treat every task equally. Fable flipped to plus 1 percent, DeepSeek to plus 17. Almost all of Fable's apparent savings came from a single task, `winning-avg-corewars`, where the RTK arm happened to finish in roughly half as many turns. Across the other 84 tasks, savings were under 1 percent.

Meanwhile RTK's own scoreboard was reporting a triumph. Across 445 DeepSeek attempts it logged 349.2 million tokens saved, an 89 percent reduction, on tasks that were getting more expensive. The metric, `rtk gain`, is raw command output minus filtered output in bytes, divided by four. In one task the model ran `head -1 train.txt` twice; RTK credited itself 120.5 million tokens saved each time by measuring a one-line read against the entire file. Those two calls alone accounted for 69 percent of the savings counter for the whole comparison. As Quesma puts it, the metric “counts removed output, not money saved” — and it can make a more expensive attempt look optimized.

To RTK's credit, its own documentation says much of this. The README warns that cutting 90 percent of bash output “is not the same as cutting your bill by 90%”, and the savings guide calls its token counts “an order of magnitude, not an invoice line.” The gap is between RTK's docs and the discourse around them: the 313,000-view post claiming 60 percent Claude Code savings, the videos promising 90.

The metric you optimize versus the bill you pay

The structural finding is the one worth carrying beyond RTK. Terminal output was just 7 percent of Fable's input tokens — 11 percent counting all tool results — and 26 percent for DeepSeek. RTK only rewrites shell calls: Claude Code's `Read`, `Grep` and `Glob` tools bypass it entirely, and roughly half the Bash calls it did see had already limited their own output with `head`, `tail` or `wc`. In practice the hook fired on 31 percent of Claude Code's terminal calls and 51 percent of OpenCode's. Compression cannot beat a ceiling it never reaches.

Caching flattens what is left. Because context is cached after each turn, 94 percent of Fable's input tokens and 98 percent of DeepSeek's were cache reads, billed at a tenth and a thirtieth of regular input. The bytes RTK removes are largely the cheapest bytes in the session, while any behavioral side effect lands on the expensive part. The numbers show exactly that: DeepSeek's terminal output characters fell 9 percent, yet prompt tokens rose 9 percent. The average turn carried 7 percent less input — and there were 18 percent more turns. On 58 tasks the RTK arm took more turns, and 44 of those cost more.

JetBrains reached the same conclusion by a different route in July. Denis Shiryaev ran 425 billed trials, about $320, on SkillsBench with claude-sonnet-5 and measured a median 7.6 percent cost increase at low reasoning effort (p=0.004) on 13.8 percent more turns, with the penalty vanishing at high effort and task quality statistically tied throughout. RTK's dashboard claimed 96.2 million tokens saved over the same trials. His verdict was “Honest engineering, wrong counterfactual” — and the generalizable lesson that a tool's self-reported savings describe its own counterfactual, not your bill.

Two independent labs, two harnesses, one pattern. That is not a verdict, and the asterisks matter. Both tested pinned builds (0.45.0 and 0.43.0), and one pathological DeepSeek failure — a rewritten `find` that looped into 339 consecutive errors at roughly 9x the baseline cost — was fixed upstream in 0.46.0 after the runs. Neither lab found a quality collapse. RTK is competent software solving a problem that has largely migrated elsewhere.

Quesma’s summary is unambiguous: “We do not recommend RTK as a generic cost-saving tool.” Frontier models already reach for `head -n` and `tail -n` unprompted, so terminal verbosity is a shrinking share of the problem. RTK probably helped more a model generation ago.

Three things to watch. Whether RTK reframes `rtk gain` as a byte ratio in the interface rather than a caveat buried in docs. Whether the next wave of context-compression tools — Quesma says Headroom is next on its bench — arrives with paired-bill numbers instead of self-reported diffs. And whether anyone demonstrates savings that survive a full multi-turn trajectory, because that is the only place the money actually is. Until then, the default for any AI efficiency claim should be the one both labs reached independently: measure the invoice, not the diff.