Notion's Knowledge Board Scores Models on Dollars per Completed Task, Not Benchmarks

The cheapest model on Notion's new Knowledge Board resolves 95.34% of real knowledge-work tasks for two cents. The most accurate one resolves 98.37% for eighty-seven cents. That is a 3-percentage-point quality gap priced at a 43x cost premium — the kind of number no capability benchmark was ever designed to surface.

Notion Labs published the Knowledge Board, last updated August 12, 2026, as a running measurement of how fifteen frontier and open-weight models perform inside Notion's own production agent. Rather than curating a fixed task set, Notion routes a randomized slice of live user traffic — roughly 7% of all weekly Notion agent tokens — evenly across the models under test, then scores each completed trace with three independent LLM judges: Anthropic's Claude Opus 4.8, Google's Gemini 3.5 Flash, and OpenAI's GPT-5.5. Two of three votes decide whether a task resolved. The output is a 1 or a 0, joined to anonymized product telemetry: dollars per task, resolution time, and undo rate.

What the board actually says

At the top, Claude Opus 5 posts a 98.37% (±0.19%) resolution rate at $0.87 per task. Kimi K3 sits statistically alongside it at 98.22% for $0.45 — and Notion's frontier analysis gives Kimi K3 a 100% probability of sitting on the quality-cost efficient frontier, versus 85% for Opus 5. Sonnet 5 resolves 96.67% at $0.34, GLM-5.2 96.48% for $0.17, and GPT-5.6-Luna 95.34% for two cents, also at 100% frontier probability. DeepSeek V4 Pro anchors the bottom at 93.67%.

The spread is the story. Across all fifteen models, resolution rates span 4.7 percentage points; costs span more than 40x. Undo rates — the share of agent responses users reversed — never exceed 1.0% and cluster between 0.3% and 0.8%, suggesting even weaker models rarely produce output so wrong users nuke it. Resolution times run from 48 seconds to 1:56.

Notion is forthright about the humility this demands. "Many of the differences we measure are smaller than their confidence intervals, so we've deliberately avoided creating a leaderboard," the company writes, adding that "overlapping confidence intervals mean models tie." Requests are dropped if a user opted out, switched models, hit a harness error, or if a judge flagged the trace as not fairly evaluable. No customer data was retained; only the binary resolution flag was stored. Models had to clear a private Notion baseline at a model-judged 0.8 or higher, and any model failing Notion's Zero Data Retention requirements — Claude Fable among them — was excluded outright.

The limits, stated plainly

This is a vendor-run evaluation, and that matters. Notion sells an AI product, negotiates with the labs whose models it ranks, and built the judge prompt "in collaboration with researchers at several frontier labs" — the same labs supplying the models. The task distribution is Notion-shaped: meeting follow-ups, inbox and calendar cleanup, support-ticket triage, sales-pipeline updates, knowledge-base Q&A. A model that excels at those is not thereby proven to excel at legal discovery or claims adjudication.

The judging layer has measurable slack. Notion's human labeling of 100 transcripts produced a Cohen's κ of 0.595 and 92.9% agreement with the ensemble — respectable, but Notion notes most disagreements were human-labeled failures the ensemble graded as passes, meaning the judge runs slightly lenient. When the whole board sits between 93.67% and 98.37%, a generous grader compresses that range further.

Independent analysts are blunt about this category. Digital Applied, in a May 2026 methodology guide, classifies vendor-controlled benchmarks as the least trustworthy governance type: "A vendor benchmark is a sales asset first and an evaluation second; read it that way." Their counterpoint is that static public benchmarks have their own terminal illnesses — contamination, where eval questions leak into training corpora, and saturation, where frontier models cluster so tightly near the ceiling that ranking becomes noise. One audit they cite found a median discriminative lifespan under two years for static evals.

Why It Matters

The Knowledge Board is best read not as a ranking but as evidence of a metric migration. For three years, model selection ran on capability scores — MMLU, GPQA, SWE-bench — that told buyers nothing about what a finished unit of work costs. Dollars per completed task collapses capability, latency, retries, tool calls, and token pricing into one procurement-legible number. Dylan Bristot, who runs the model-comparison site WhatLLM.org, framed the shift in a June analysis: "The right model is not the one that wins the most screenshots. It is the one that lets your system finish more work per dollar without making the expensive mistakes." He is equally clear that cheap is not automatically right — "Cheap failure is still failure" — which is why undo rate belongs beside cost.

Notion's AI engineering lead, Sarah Sachs, has argued a version of this for months. On the AI Engineer podcast in July, she described the trap facing companies built on rented inference: "You are basically buying tokens at a huge markup and selling them again. That is not value you can defend." Her prescription — model agnosticism, evaluation on cost-per-capability rather than per-token price — is what the Knowledge Board operationalizes in public. "We cannot do this without optionality," she said, "and we cannot do this without conviction that we understand what models are required for which tasks." Notion routes roughly 75% of its AI traffic through a router that picks the cheapest capable model; the Knowledge Board is the receipt.

What to Watch

The interesting question is whether anyone else publishes one. A single vendor's production board is a marketing artifact; five vendors publishing comparable cost-per-resolved-task figures across different task distributions would be a standard, and would let buyers triangulate the way evaluation researchers already recommend. Watch whether Notion keeps the board refreshed as new models ship, and whether the frontier-probability column becomes the number people actually cite. Kimi K3 and GPT-5.6-Luna both score 100% there while sitting second and twelfth on raw quality — the clearest sign yet that "best model" and "right model" have come apart.