mlsanityData to 14 Oct 2026

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

arXiv 2606.15007 ↗First posted 12 Jun 2026cs.CL cs.AI cs.LGcited by 42

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 3 more
Abstract (arXiv)

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the context length to 1M tokens, and post-trained using Supervised Fine Tuning (SFT), Reinforcement Learning (RL), and Multi-teacher On-Policy Distillation (MOPD). Nemotron 3 Ultra is our most capable model yet, employing multiple key technologies - LatentMoE, Multi Token Prediction (MTP), NVFP4 pre-training, multi-environment RLVR, MOPD, and reasoning budget control. Nemotron 3 Ultra achieves up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy. The state-of-the-art accuracy, high inference throughput, and 1M token context length make Nemotron 3 Ultra ideal for long-running autonomous agentic tasks. We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 69 of its 116 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

69 of 116 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning23 Dec
  2. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug

Other references · 67

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
38 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  5. 2024DeepSeek-V3 Technical Reportcited by 811
  6. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  9. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  10. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  11. 2025Kimi K2: Open Agentic Intelligencecited by 567
  12. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  13. 2025Humanity's Last Examcited by 491
  14. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  15. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  16. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  17. 2025Generalizing Verifiable Instruction Followingcited by 194
  18. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  19. 2026MiMo-V2-Flash Technical Reportcited by 191
  20. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  21. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  22. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 158
  23. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 156
  24. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  25. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  26. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  27. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  28. 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoningcited by 94
  29. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  30. 2025Pretraining Large Language Models with NVFP4cited by 76
  31. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  32. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 65
  33. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65
  34. 2026Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillationcited by 60
  35. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  36. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 49
  37. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  38. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 47

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 42.

42 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24
  2. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 19
  3. 2026Loop the Loopies!cited by 8
  4. 2026Agent Memory Is a Surface for Endogenous Authorization Launderingcited by 5
  5. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  6. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 3
  7. 2026Solar Open 2 Technical Reportcited by 3
  8. 2026ReplaySSM: Cache SSM Inputs, Not Statecited by 2
  9. 2026Scheduling Mixed RL Rollouts Beyond Prefix Localitycited by 1
  10. 2026Last Translation Benchmarkcited by 1
  11. 2026NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generationcited by 1
  12. 2026Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agentscited by 1
30 more
  1. 2026Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agentscited by 1
  2. 2026EvalResearchBench: Can AI Agents Design Their Own Evaluations?cited by 1
  3. 2026Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chainscited by 0
  4. 2026TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Tradecited by 0
  5. 2026Motif 3: Technical Reportcited by 0
  6. 2026Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledgecited by 0
  7. 2026A.X K2 Technical Reportcited by 0
  8. 2026Post-Training Language Models for Gold-Medal Performance in Coding Competitionscited by 0
  9. 2026Rethinking Heterogeneous System Disaggregation for Subquadratic Attentioncited by 0
  10. 2026Bellman Policy Optimizationcited by 0
  11. 2026Shutdown Sabotage Propensities in Multi-Agent Systemscited by 0
  12. 2026MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillationcited by 0
  13. 2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0
  14. 2026On-Policy Attention Linearizationcited by 0
  15. 2026No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillationcited by 0
  16. 2026Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillationcited by 0
  17. 2026ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learningcited by 0
  18. 2026SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agentscited by 0
  19. 2026Low-Discrepancy Dither for Quantized Recurrent State Cachescited by 0
  20. 2026Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretrainingcited by 0
  21. 2026From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillationcited by 0
  22. 2026Rethinking Self-Distillation for Multi-Teacher Capability Mergingcited by 0
  23. 2026Privileged Context as Drift in On-Policy Self-Distillationcited by 0
  24. 2026ExperienceIndex: Artifact-Grounded Memorycited by 0
  25. 2026Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Modelscited by 0
  26. 2026One Model to Loop Them All: Elastic Latent Recurrence for Language Modelscited by 0
  27. 2026When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMscited by 0
  28. 2026Double-Rollout Reinforcement Learning with Sampling-based Reward Estimationcited by 0
  29. 2026DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Losscited by 0
  30. 2026Diagnosing and Correcting Reward Hacking in Verifiable Instruction Followingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
  3. 2025Llama-Nemotron: Efficient Reasoning Models
  4. 2025Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
  5. 2026HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
  6. 2025Apriel-Nemotron-15B-Thinker
  7. 2026Jupiter-N Technical Report
  8. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning