AMD Releases Instella-MoE, a Fully Open Mixture-of-Experts LLM Trained on Its Own GPUs

AMD has crossed a line that Nvidia's rivals have long talked about but rarely executed on: training a competitive large language model end to end, on its own silicon, and then handing over nearly everything — weights, data mixtures, training code, and intermediate checkpoints — for anyone to inspect and rebuild.

On August 1, 2026, AMD released Instella-MoE-16B-A3B, a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs using AMD's ROCm software stack. The model carries 16 billion total parameters but activates just 2.8 billion parameters per token, routing each token through 2 shared experts plus 6 of 64 available routed experts. According to AMD's ROCm engineering blog, the company is \"excited to introduce Instella-MoE, a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters,\" trained \"from scratch on AMD Instinct MI300X and MI325X GPUs with the AMD ROCm software stack.\"

The release is notable less for beating frontier models and more for what AMD chose to publish. Every stage of the training pipeline — pretraining, mid-training, long-context extension, supervised fine-tuning, DPO, and reinforcement learning — ships as a separate downloadable checkpoint, alongside the data mixtures and configs used to produce it. Pretraining ran across 7.1 trillion tokens drawn from open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. AMD paired that with two systems-level innovations it says squeezed real efficiency out of the hardware: Gated Multi-head Latent Attention, which adds a learned output gate to standard MLA, and FarSkip-Collective, a communication-overlap scheme for expert-parallel training. AMD reports the combination delivered a 12.7% pre-training speedup and up to a 39.2% reduction in time-to-first-token when the model is served with expert parallelism.

On benchmarks, the base checkpoint averages 76.7 across standard evaluations, which AMD and independent reporting describe as the strongest score among fully open models — ahead of Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9), though it trails the more restrictively licensed Qwen3.5-4B-Base (79.5). It leads all evaluated models on WinoGrande (86.5) and posts 65.7 on HumanEval+. After post-training, the reasoning-tuned \"Think\" checkpoint reaches an average of 73.22, edging out Olmo3-7B-Think (71.97), Gemma-4-E4B-think (70.47), and Qwen3.5-4B (69.73), with IFEval climbing from 77.08 after DPO to 83.70 following AMD's reinforcement learning stage. As MarkTechPost's Asif Razzaq summarized the release, the weights themselves \"ship under a ResearchRAIL license for academic and research purposes only, so this is not a drop-in commercial model\" — but the training codebase on GitHub is MIT licensed, which he called \"the more reusable asset here.\"

Why It Matters

Instella-MoE is a direct shot at two separate narratives at once. The first is Nvidia's training moat. For years, the assumption inside AI labs has been that serious pretraining runs — trillions of tokens, multi-week jobs, expert-parallel MoE architectures — require CUDA and Nvidia's software ecosystem to be practical. AMD's own account of a 7.1-trillion-token run completed entirely on MI300X and MI325X clusters, using AMD's Primus training framework and Miles RL framework instead of Nvidia's stack, is a concrete data point that ROCm has closed enough of the software gap to support frontier-adjacent training, not just inference. Every efficiency number AMD published — the 12.7% training speedup, the 39.2% TTFT reduction — is implicitly an argument that buyers no longer need to default to Nvidia hardware for MoE-scale work.

The second narrative is the fully-open-model movement, which has increasingly split into two camps: labs that release weights but withhold data and training code, and a smaller group — Allen Institute's OLMo project, LLM360, and now AMD — that publishes the entire pipeline. By shipping checkpoints from every training stage plus the data mixtures behind them, AMD is positioning Instella-MoE as a reproducibility benchmark rather than just another weights drop, explicitly inviting researchers to \"reproduce our results\" and \"study open MoE training recipes in detail.\" That openness has a catch: the ResearchRAIL license blocks commercial deployment, so companies hoping to build products on Instella-MoE weights directly are out of luck, even as the MIT-licensed training code remains fully reusable.

What to Watch

The near-term test is adoption: whether independent labs actually retrain or fine-tune on AMD's released data mixtures and checkpoints, and whether the claimed MI300X/MI325X efficiency gains hold up when reproduced outside AMD's own clusters. Watch for a formal Instella-MoE technical report, which AMD has said is forthcoming, to fill in evaluation and safety details the blog post glossed over. Also worth tracking is whether AMD or a third party releases a commercially licensed variant — the current ResearchRAIL restriction is the main obstacle between this release and real enterprise traction, and lifting it would turn Instella-MoE from a research artifact into a genuine alternative to Nvidia-trained open models.