# Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

arXiv: https://arxiv.org/abs/2606.15007 · HTML: https://mlsanity.com/paper/2606.15007/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 69 of its 116 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Mixture-of-experts / [MoE in reinforcement learning and robotics](https://mlsanity.com/moe/moe-in-reinforcement-learning-and-robotics/)
- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 69)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)

## Built on by (most cited first, 30 of 42)

- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluators](https://mlsanity.com/paper/2606.26294/) (2026)
- [Loop the Loopies!](https://mlsanity.com/paper/2607.16051/) (2026)
- [Agent Memory Is a Surface for Endogenous Authorization Laundering](https://mlsanity.com/paper/2609.01836/) (2026)
- [Intern-S2-Preview: Scientific Agentic Foundation Model](https://mlsanity.com/paper/2608.13505/) (2026)
- [A Practical Investigation of Training-free Relaxed Speculative Decoding](https://mlsanity.com/paper/2607.08690/) (2026)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [ReplaySSM: Cache SSM Inputs, Not State](https://mlsanity.com/paper/2610.13778/) (2026)
- [Scheduling Mixed RL Rollouts Beyond Prefix Locality](https://mlsanity.com/paper/2608.11152/) (2026)
- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)
- [NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation](https://mlsanity.com/paper/2609.17699/) (2026)
- [Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agents](https://mlsanity.com/paper/2609.31995/) (2026)
- [Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents](https://mlsanity.com/paper/2609.39982/) (2026)
- [EvalResearchBench: Can AI Agents Design Their Own Evaluations?](https://mlsanity.com/paper/2610.04184/) (2026)
- [Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains](https://mlsanity.com/paper/2607.18438/) (2026)
- [TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Trade](https://mlsanity.com/paper/2608.06549/) (2026)
- [Motif 3: Technical Report](https://mlsanity.com/paper/2608.09119/) (2026)
- [Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge](https://mlsanity.com/paper/2608.28478/) (2026)
- [A.X K2 Technical Report](https://mlsanity.com/paper/2608.30181/) (2026)
- [Post-Training Language Models for Gold-Medal Performance in Coding Competitions](https://mlsanity.com/paper/2609.02849/) (2026)
- [Rethinking Heterogeneous System Disaggregation for Subquadratic Attention](https://mlsanity.com/paper/2609.13134/) (2026)
- [Bellman Policy Optimization](https://mlsanity.com/paper/2609.15987/) (2026)
- [Shutdown Sabotage Propensities in Multi-Agent Systems](https://mlsanity.com/paper/2609.28274/) (2026)
- [MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.30837/) (2026)
- [Low-Bit Recurrent States in Hybrid Language Models](https://mlsanity.com/paper/2609.30950/) (2026)
- [On-Policy Attention Linearization](https://mlsanity.com/paper/2609.31947/) (2026)
- [No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.34745/) (2026)
- [Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2609.35347/) (2026)
- [ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.35532/) (2026)
- [SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents](https://mlsanity.com/paper/2609.35889/) (2026)

## Abstract (arXiv)

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.