Tim Dettmers Says a 125B Qwen Now Runs on One 24GB GPU, and a 550B DeepSeek on a MacBook

Tim Dettmers, the researcher behind bitsandbytes and QLoRA, says his lab has built an inference framework that runs Alibaba’s 125-billion-parameter Qwen 3.8 Flash Next on a single 24GB GPU, and DeepSeek V4.1, a 550B model, on a 128GB MacBook. The claim arrived Monday in a long blog post announcing a week of releases from dlab, the group Dettmers runs across Carnegie Mellon University and the University of Washington: two open-source projects and four papers, shipping together starting Tuesday.

“Qwen 3.8 at 27 billion parameters has been the popular local model,” Dettmers wrote. “Our framework lets you run its larger sibling, Qwen 3.8 Flash Next at 125 billion parameters, on a single 24 GB GPU — the card in a normal desktop machine.” For the larger DeepSeek model, he listed AMD Strix, an NVIDIA DGX Spark, or a MacBook with 128GB of unified memory as sufficient hardware, and said context compression and long-context handling are handled automatically by the framework.

The numbers rest on aggressive quantization, the field Dettmers has spent most of a decade defining. In one experiment described in the post, the lab pointed its agent harness at the framework’s Mac and Metal kernels and told it to optimize them without supervision. The result, per Dettmers, was quantized inference of Qwen 3.6 35B-A3B at 450 tokens per second at 1.5 bits per weight, roughly a tenth of the memory a half-precision model would need. He described the output at that precision as high quality, though the post offers no benchmark tables, and the code and papers that would let outsiders check the claims are not yet public.

The second headline release is CliffCompaction, an auto-compaction method for long agent sessions that the lab says it has used internally for months. Dettmers said sessions with it routinely run for millions of tokens, some of his own have passed 100 million, and it cuts overall cost by about 50 percent. “It is considerably more powerful than the auto-compaction in Claude Code or Codex,” he wrote, adding that one partner company measured a 45 percent reduction in its total AI budget after deploying it. On KernelBench, the lab claims state-of-the-art results that beat AlphaEvolve-style and hierarchical-memory approaches by a wide margin.

Dettmers also previewed a fully offline autonomous research system that he says outperforms deep-research products from frontier labs, as well as Sakana AI’s system and Google’s ScientistOne. His example: he asked the agent to find a fresh bioinformatics problem with cheap evaluation and recent literature, and within about two hours it had established a new lower bound on heuristic methods, reimplemented the best published heuristic, and surfaced flaws in the evaluation data the subfield relies on. It did not reach state of the art on the overall task, he acknowledged.

The post is as much manifesto as changelog. Dettmers opens with a classroom poll in which roughly 120 of 150 students said they feared not finding a job, and with emails from PhD students who see academic research as pointless next to frontier labs. His answer is that the paper is no longer the unit of research; the ecosystem is. “A couple of people with a couple of GPUs can build systems that compete with the frontier,” he wrote. “The renaissance does not require anyone’s permission. You can just do things.”

Early reaction was muted. The post reached Hacker News on Monday with only a handful of points and no discussion by the evening, a sign that the community is waiting for code rather than prose. That is consistent with how the open-source crowd has treated Dettmers before: bitsandbytes and QLoRA earned their reputation by being installable and reproducible, not by announcement.

Why it matters

If the memory figures hold up, this is the most consequential local-inference release in a while. A 24GB card is what enthusiasts and small labs already own, and a 550B mixture-of-experts model on a laptop would collapse the gap between what hobbyists can run and what API providers sell.

The CliffCompaction claim may matter more for day-to-day agent users than the quantization numbers. Compaction quality is the quiet bottleneck of long-running coding agents, and Anthropic and OpenAI both treat their implementations as internal details. An open method that claims 50 percent cost savings and 100M-token sessions, with a KernelBench result to point at, would be an immediate target for integration into open harnesses and a benchmark the closed vendors would have to answer.

There is also a personal arc here. In a 2024 interview with Interconnects, Nathan Lambert described Dettmers as “one of the people that everyone recognizes for quantization and other compute research,” and Dettmers replied that the field had mostly maxed out that lever and should look for other factors. Two years later his lab is pulling both levers at once: squeezing weights to 1.5 bits and squeezing context with compaction.

The caveats are real. The post has no benchmarks for output quality at 1.5 bits, no throughput numbers for the 125B and 550B configurations, and the comparisons to Sakana, Google, and the frontier deep-research tools are asserted rather than shown.

What to watch

The code lands starting Tuesday, and the first thing to check is whether the 125B-on-24GB result reproduces on a stock RTX card with usable tokens-per-second and acceptable perplexity, and whether the DeepSeek V4.1 configuration really fits on a 128GB MacBook without swapping. Watch for the four papers, particularly the CliffCompaction writeup and its KernelBench methodology, and for whether maintainers of open harnesses move to adopt it. Finally, watch for responses from Anthropic and OpenAI on compaction, and from Sakana and Google on the autonomous-research comparisons; if the claims survive a week of outside scrutiny, the argument that small academic labs can compete on systems, not just ideas, gets much harder to dismiss.