Data to 14 Oct 2026

Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

arXiv 2604.12374 ↗First posted 14 Apr 2026cs.LG cs.AI cs.CLcited by 38

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

also in 3 more

Abstract (arXiv)

We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 74 of its 115 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

74 of 115 references.

Closest ancestors · 4

Same direction, newest first

Other references · 70

2021Training Verifiers to Solve Math Word Problemscited by 10,109
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2025Qwen3 Technical Reportcited by 8,820
2021Evaluating Large Language Models Trained on Codecited by 8,320
2020Measuring Massive Multitask Language Understandingcited by 7,963
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2024Qwen2.5 Technical Reportcited by 4,942
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2021Program Synthesis with Large Language Modelscited by 3,637
2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373

38 more

2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2017Decoupled Weight Decay Regularizationcited by 1,831
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
2024DeepSeek-V3 Technical Reportcited by 811
2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
2025Humanity's Last Examcited by 491
2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
2024ToolACE: Winning the Points of LLM Function Callingcited by 216
2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 38.

38 papers.

Sort:

By area:

2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 19
2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
2026Loop the Loopies!cited by 8
2026Mach-Mind-4-Flash Technical Reportcited by 4
2026Beyond the Black Box: Interpretability of Agentic AI Tool Usecited by 3
2026Forward-Free Diffusion Language Models with BPTT-Free Looped Refinementcited by 2
2026Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agentscited by 2
2026Multi-Harness Reinforcement Learning for Coding Agents: Training Gains and Harness Dependencecited by 2
2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditionscited by 1
2026SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPODcited by 1

26 more

2026NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generationcited by 1
2026Can a Single Message Paralyze the AI Infrastructure? The Rise of AbO-DDoS Attacks through Targeted Mobius Injectioncited by 0
2026SHIELDS: Automating OS Hardening with Iterative Multi-Agent Remediationcited by 0
2026SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variationscited by 0
2026Spokes: Optimizing for Diverse Pretraining Data Selectioncited by 0
2026Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackerscited by 0
2026Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versionscited by 0
2026Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configurationcited by 0
2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
2026Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stackcited by 0
2026Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering designcited by 0
2026GrowMTP: Can RL Grow Its Own Draft Head?cited by 0
2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivationcited by 0
2026Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Usecited by 0
2026Rufus-Air: An Open LLM Post-Training Recipecited by 0
2026Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scalecited by 0
2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
2026Low-Bit Recurrent States in Hybrid Language Modelscited by 0
2026SketchSSM: Write to the Full State, Read from a Compact Sketchcited by 0
2026ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modellingcited by 0
2026Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulationcited by 0
2026Low-Discrepancy Dither for Quantized Recurrent State Cachescited by 0
2026Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hardcited by 0
2026Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Modelscited by 0
2026Preserving Updates in Low-Bit Recurrent State Quantizationcited by 0
2026Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repaircited by 0

Similar, not cited

Close in wording, no citation either way