mlsanityData to 14 Oct 2026

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset

arXiv 2508.15096 ↗First posted 20 Aug 2025cs.CL cs.AI cs.LGcited by 38

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Pretraining large language models (LLMs) on high-quality, structured data such as mathematics and code substantially enhances reasoning capabilities. However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy HTML-to-text conversion, and the failure to reliably preserve mathematical structure. In this work, we introduce Nemotron-CC-Math, a large-scale, high-quality mathematical corpus constructed from Common Crawl using a novel, domain-agnostic pipeline specifically designed for robust scientific text extraction. Unlike previous efforts, our pipeline recovers math across various formats (e.g., MathJax, KaTeX, MathML) by leveraging layout-aware rendering with lynx and a targeted LLM-based cleaning stage. This approach preserves the structural integrity of equations and code blocks while removing boilerplate, standardizing notation into LaTeX representation, and correcting inconsistencies. We collected a large, high-quality math corpus, namely Nemotron-CC-Math-3+ (133B tokens) and Nemotron-CC-Math-4+ (52B tokens). Notably, Nemotron-CC-Math-4+ not only surpasses all prior open math datasets-including MegaMath, FineMath, and OpenWebMath-but also contains 5.5 times more tokens than FineMath-4+, which was previously the highest-quality math pretraining dataset. When used to pretrain a Nemotron-T 8B model, our corpus yields +4.8 to +12.6 gains on MATH and +4.6 to +14.3 gains on MBPP+ over strong baselines, while also improving general-domain performance on MMLU and MMLU-Stem. We present the first pipeline to reliably extract scientific content--including math--from noisy web-scale data, yielding measurable gains in math, code, and general reasoning, and setting a new state of the art among open math pretraining corpora. To support open-source efforts, we release our code and datasets.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 79 references that are in the map. First posted on arXiv 20 Aug 2025.

Builds on

36 of 79 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MegaMath: Pushing the Limits of Open Math Corpora3 Apr
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  3. 2023Llemma: An Open Language Model For Mathematics16 Oct
  4. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text10 Oct
  5. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun
  6. 2020Generative Language Modeling for Automated Theorem Proving7 Sep

Other references · 30

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
18 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  2. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  4. 2023StarCoder: may the source be with you!cited by 799
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  10. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  11. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  12. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  13. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  14. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
  15. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  16. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  17. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  18. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 34.

38 papers.

Sort:

By area:

  1. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  2. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  3. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  4. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  5. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  6. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 35
  7. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 13
  8. 2026STEM: Scaling Transformers with Embedding Modulescited by 12
  9. 2026From Growing to Looping: A Unified View of Iterative Computation in LLMscited by 7
  10. 2026OProver: A Unified Framework for Agentic Formal Theorem Provingcited by 7
  11. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 7
  12. 2026DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolutioncited by 6
26 more
  1. 2026Understanding Dynamic Compute Allocation in Recurrent Transformerscited by 6
  2. 2026Understanding Reasoning from Pretraining to Post-Trainingcited by 5
  3. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 4
  4. 2025MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sourcescited by 3
  5. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 3
  6. 2026LoopMTP: A looped transformer guided by latent multi-token predictioncited by 3
  7. 2026Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domaincited by 2
  8. 2026When Can LLMs Learn to Reason with Weak Supervision?cited by 2
  9. 2026Forward-Free Diffusion Language Models with BPTT-Free Looped Refinementcited by 2
  10. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  11. 2026Kelix Technical Reportcited by 1
  12. 2026The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errorscited by 1
  13. 2026A Sovereign, Open-Source Foundation Model for German and Englishcited by 1
  14. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 1
  15. 2026Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090cited by 1
  16. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigmcited by 0
  17. 2026WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvementcited by 0
  18. 2026daVinci-LLM:Towards the Science of Pretrainingcited by 0
  19. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Lawscited by 0
  20. 2026Hybrid Open-Ended Tri-Evolution Makes Better Deep Researchercited by 0
  21. 2026ShopX: A Foundation Model for Intent-to-Item Fulfillment in Agentic Shoppingcited by 0
  22. 2026Instella-MoE Technical Reportcited by 0
  23. 2026dQwen3.5: Hybrid-Attention Diffusion Language Modelscited by 0
  24. 2026Counting and Min-Cost Encoding for Tokenization in Large Language Modelscited by 0
  25. 2026NanoProof: Open and Efficient Automated Theorem Proving in Lean 4cited by 0
  26. 2026Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
  2. 2025SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain
  3. 2025MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
  4. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code
  5. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  6. 2025Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
  7. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning
  8. 2025TopoAlign: A Framework for Aligning Code to Math via Topological Decomposition