mlsanityData to 14 Oct 2026

MegaMath: Pushing the Limits of Open Math Corpora

arXiv 2504.02807 ↗First posted 3 Apr 2025cs.CL cs.AI cs.LGcited by 51

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Mathematical reasoning is a cornerstone of human intelligence and a key benchmark for advanced capabilities in large language models (LLMs). However, the research community still lacks an open, large-scale, high-quality corpus tailored to the demands of math-centric LLM pre-training. We present MegaMath, an open dataset curated from diverse, math-focused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all for acquiring higher-quality data on the Internet. (2) Recalling Math-related code data: We identified high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We synthesized QA-style text, math-related code, and interleaved text-code blocks from web data or code data. By integrating these strategies and validating their effectiveness through extensive ablations, MegaMath delivers 371B tokens with the largest quantity and top quality among existing open math pre-training datasets.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 56 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

46 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code10 Oct
  2. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data2 Oct
  3. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On11 Jul
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  5. 2023Llemma: An Open Language Model For Mathematics16 Oct
  6. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text10 Oct
  7. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  8. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning29 Sep
  9. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  12. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
25 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  9. 2022PAL: Program-aided Language Modelscited by 652
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2023Textbooks Are All You Needcited by 569
  12. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  13. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  14. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  15. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  16. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  17. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  18. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 106
  19. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  20. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  21. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  22. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  23. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  24. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 13
  25. 2024Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretrainingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 20.

51 papers.

Sort:

By area:

  1. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  2. 2025Scaling Latent Reasoning via Looped Language Modelscited by 165
  3. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  4. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  5. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  6. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 59
  7. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  8. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  9. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  10. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  11. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 27
  12. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
38 more
  1. 2025A Survey on LLM Mid-Trainingcited by 21
  2. 2025Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Spacecited by 17
  3. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  4. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  5. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 14
  6. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoningcited by 10
  7. 2025Essential-Web v1.0: 24T tokens of organized web datacited by 8
  8. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8
  9. 2026Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-trainingcited by 7
  10. 2025Meta-R1: Empowering Large Reasoning Models with Metacognitioncited by 6
  11. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  12. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 5
  13. 2025Thinking Augmented Pre-trainingcited by 4
  14. 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 4
  15. 2026Data Science and Technology Towards AGI Part I: Tiered Data Managementcited by 4
  16. 2025Large-Scale Diverse Synthesis for Mid-Trainingcited by 3
  17. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  18. 2026Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curationcited by 3
  19. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Spacecited by 3
  20. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 3
  21. 2026Sumi: Open Uniform Diffusion Language Model from Scratchcited by 3
  22. 2025JT-Math: A Multi-Stage Framework for Advanced Mathematical Reasoning in Large Language Modelscited by 2
  23. 2025LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Pointscited by 2
  24. 2025Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patternscited by 2
  25. 2025Arbitrage: Efficient Reasoning via Advantage-Aware Speculationcited by 2
  26. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  27. 2026Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversioncited by 2
  28. 2026TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?cited by 1
  29. 2026Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMscited by 1
  30. 2026DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Datacited by 1
  31. 2026Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within 5090cited by 1
  32. 2025KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networkscited by 0
  33. 2026Leum-VL Technical Reportcited by 0
  34. 2026daVinci-LLM:Towards the Science of Pretrainingcited by 0
  35. 2026TuringLLM: Efficiently Scaling Foundation Models Toward Physical AIcited by 0
  36. 2026Instella-MoE Technical Reportcited by 0
  37. 2026HERO-MoE: Historical Expert Routing with Scale-Preserving Fusioncited by 0
  38. 2026Scaling Laws for Multimodal Data Mixturescited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
  3. 2025Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
  4. 2024FineMath: A Fine-Grained Mathematical Evaluation Benchmark for Chinese Large Language Models
  5. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline
  6. 2025TopoAlign: A Framework for Aligning Code to Math via Topological Decomposition
  7. 2024CoinMath: Harnessing the Power of Coding Instruction for Math LLMs
  8. 2024PARAMANU-GANITA: Can Small Math Language Models Rival with Large Language Models on Mathematical Reasoning?