# Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision

arXiv: https://arxiv.org/abs/2512.15489 · HTML: https://mlsanity.com/paper/2512.15489/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning](https://mlsanity.com/paper/2504.11456/) (2025)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset](https://mlsanity.com/paper/2402.10176/) (2024)
- [1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training](https://mlsanity.com/paper/2503.19633/) (2025)
- [MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code](https://mlsanity.com/paper/2410.08196/) (2024)
- [MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning](https://mlsanity.com/paper/2402.17231/) (2024)
- [Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On](https://mlsanity.com/paper/2407.08348/) (2024)

## Built on by (most cited first, 26 of 26)

- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2604.12374/) (2026)
- [Are Full Rollouts Necessary for On-Policy Distillation?](https://mlsanity.com/paper/2605.31490/) (2026)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts](https://mlsanity.com/paper/2601.17111/) (2026)
- [Learning, Fast and Slow: Towards LLMs That Adapt Continually](https://mlsanity.com/paper/2605.12484/) (2026)
- [SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.14277/) (2026)
- [KV Admission: Learning What to Write for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2512.17452/) (2025)
- [Data Science and Technology Towards AGI Part I: Tiered Data Management](https://mlsanity.com/paper/2602.09003/) (2026)
- [Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.06326/) (2026)
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://mlsanity.com/paper/2608.05141/) (2026)
- [Thomson: Continual Learning of Frontier Models for SovereignAI](https://mlsanity.com/paper/2608.27147/) (2026)
- [JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency](https://mlsanity.com/paper/2604.03044/) (2026)
- [On Subquadratic Architectures: From Applications to Principles](https://mlsanity.com/paper/2606.12364/) (2026)
- [Building Multi-Task Agentic LLMs via Two-Phase Distillation](https://mlsanity.com/paper/2606.30044/) (2026)
- [On-Policy Delta Distillation for Multilingual Math Reasoning](https://mlsanity.com/paper/2608.05802/) (2026)
- [Leum-VL Technical Report](https://mlsanity.com/paper/2603.20354/) (2026)
- [Post-training is (Massive) Supervised Learning](https://mlsanity.com/paper/2606.07527/) (2026)
- [NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research](https://mlsanity.com/paper/2606.26671/) (2026)
- [Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization](https://mlsanity.com/paper/2607.05064/) (2026)
- [CantoneseLLM v2: Reasoning in a Low-Resource Language](https://mlsanity.com/paper/2609.06970/) (2026)
- [Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks](https://mlsanity.com/paper/2609.08404/) (2026)
- [Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training](https://mlsanity.com/paper/2609.14306/) (2026)
- [Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning](https://mlsanity.com/paper/2610.05872/) (2026)
- [Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training](https://mlsanity.com/paper/2610.12933/) (2026)

## Abstract (arXiv)

High-quality mathematical reasoning supervision requires diverse reasoning styles, long-form traces, and effective tool integration, capabilities that existing datasets provide only in limited form. Leveraging the multi-mode generation ability of gpt-oss-120b, we introduce Nemotron-Math, a large-scale mathematical reasoning dataset containing 7.5M solution traces across high, medium, and low reasoning modes, each available both with and without Python tool-integrated reasoning (TIR). The dataset integrates 85K curated AoPS problems with 262K community-sourced StackExchange-Math problems, combining structured competition tasks with diverse real-world mathematical queries. We conduct controlled evaluations to assess the dataset quality. Nemotron-Math consistently outperforms the original OpenMathReasoning on matched AoPS problems. Incorporating StackExchange-Math substantially improves robustness and generalization, especially on HLE-Math, while preserving accuracy on math competition benchmarks. To support efficient long-context training, we develop a sequential bucketed strategy that accelerates 128K context-length fine-tuning by 2--3× without significant accuracy loss. Overall, Nemotron-Math enables state-of-the-art performance, including 100% maj@16 accuracy on AIME 2024 and 2025 with Python TIR.