Ornith-1.5 Writes Its Own Homework, and Lands on Claude Opus's Doorstep

An AI outfit that almost nobody had heard of two months ago published three open-weight models on August 19 and put a number on the board that frontier labs have spent the year defending. Ornith-1.5-397B scores 86.1 on Terminal-Bench 2.1, a hair above the 85.0 the same lab reports for Claude Opus 4.8. The weights are on Hugging Face under the MIT license. The most interesting thing about them is not the score — it is how the score was reached, through a training loop in which the model invents its own tasks, builds its own grader, and gets rewarded on all three at once.

Ornith shipped the family in three sizes: a 397-billion-parameter mixture-of-experts flagship, a 35B MoE that activates only 3B parameters per token, and a 9B dense model with a quantized Ornith-1.5-9B-Mobile build the team says deploys on iPhone and Android. GGUF, MLX, NVFP4 and FP8 conversions landed the same day. The 397B flagship carries a 262,144-token context window, extensible to roughly one million tokens via YaRN scaling, and weighs about 800GB in bf16 — realistically eight H200s to serve.

The technique is the headline. Ornith-1.0, released in late June, used what the lab calls self-scaffolding: let the model write its own agent harness rather than hand-coding one per benchmark, but still train it against a fixed, human-curated problem set. Ornith-1.5 removes the fixed set. As the release post puts it: “Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning.”

How the loop works

Each training cycle runs three stages. Given an environment or codebase and its own solving history, the model proposes a task harder than what it already handles. It then generates a task-specific scaffold — instructions, tools, decomposition strategy, orchestration. Finally the policy attempts the task under that scaffold. Reward propagates backward across all three: “Question generation, harness generation, and solution rollouts are all optimized with GRPO using their respective rewards,” the write-up states.

The obvious failure mode is a model that writes itself an easy exam. Ornith's answer is a multiplicative reward: validity times frontier difficulty times novelty. Validity is a hard gate — if the scaffold does not execute, if high-confidence solutions fail or clearly wrong ones pass, the whole task scores zero. “This prevents malformed tasks or unreliable scaffolds from receiving reward simply because they appear difficult,” the post says. Difficulty is measured from the model's own empirical pass rate across sampled rollouts, with reward peaking at a target success rate of 0.2 — hard enough that most attempts fail, easy enough that some succeed. As the model masters a task, that task's reward decays, pushing the generator toward harder problems with no hand-tuned curriculum. Novelty penalizes near-duplicates against a buffer of prior tasks. The generated harness is scored separately on task alignment, reward fidelity and resistance to hacking.

The deltas against Ornith-1.0 are the strongest evidence the loop does something. On DeepSWE, the flagship went from 8 to 56.0. On Frontier-Bench v0.1, from 2.7 to 13.5. At the 35B scale, DeepSWE moved from 0 to 22.0. Elsewhere the gains are ordinary: SWE-bench Verified rose from 82.4 to 86.0, GPQA Diamond from 88.1 to 92.8.

Why it matters, and what it does not prove

Read against Claude Opus 4.8, the picture is a split decision, not a win. Ornith-1.5-397B leads on Terminal-Bench 2.1 under both harnesses (86.1 and 85.2 against 85.0 and 78.9), on SWE-bench Multilingual (79.6 to 75.7), on WideSearch (80.8 to 72.9) and marginally on SWE-bench Verified (86.0 to 85.8). It loses on DeepSWE (56.0 to 59.0), SWE-bench Pro (65.1 to 68.0), NL2Repo (59.5 to 69.7), Humanity's Last Exam without tools (44.6 to 49.8) and GPQA Diamond (92.8 to 93.6). On Frontier-Bench, the hardest column in the table, Opus scores 21.1 to Ornith's 13.5. Moonshot's Kimi K3 — 2.8 trillion parameters, also open — beats both on Terminal-Bench (88.3), DeepSWE (67.5) and BrowseComp (91.2).

Three caveats deserve equal billing with the headline. First, every number is self-reported. Ornith averages five independent runs and documents its harnesses, temperatures, context windows and anti-hacking controls — git history stripped from repo images, network access disabled during SWE-bench solving — which is meaningfully more disciplined than a bare leaderboard row. It is still the vendor grading the vendor. Writing on explainx.ai, Yash Thakker noted that “leaderboard numbers and reproduced numbers diverge often enough that ‘self-reported’ should always carry an asterisk, regardless of how careful the reporting methodology looks on paper.”

Second, this is a post-training result, not a new foundation model. Ornith states plainly that 1.5 extends 1.0, “which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training.” The self-improvement loop is a recipe applied to somebody else's base weights — impressive as method, less so as a claim about independent frontier capability.

Third, self-improvement here means a training-time data flywheel. Nothing edits its own weights after you download it. The word invites a reading the write-up does not support.

What to watch

The reproduction question resolves fast or not at all. Terminal-Bench and SWE-bench Verified both have independent harnesses; within weeks, third-party runs will either land near 86.1 or well below it, and the gap will be the story. Watch also whether the loop generalizes: if a lab with more compute applies self-generated curricula to a frontier base and the DeepSWE-style jumps repeat, the fixed benchmark suite as a training target starts looking obsolete. And watch Ornith-1.5-9B-Mobile. A 9B model scoring 70.6 on SWE-bench Verified from a phone is a stranger fact than a 397B model tying Opus.