mlsanityData to 7 Oct 2026

MathPile: A Billion-Token-Scale Pretraining Corpus for Math

arXiv 2312.17120 ↗First posted 28 Dec 2023cs.CL cs.AI cs.LGcited by 67

Not on the map. Why?

Cited by 41 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of "less is more", firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 50 of its 75 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

50 of 75 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2018Datasheets for Datasetscited by 1,371
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  13. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  15. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  17. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  18. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  19. 2023Textbooks Are All You Needcited by 561
  20. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  21. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  22. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  23. 2023Llemma: An Open Language Model For Mathematicscited by 385
  24. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  25. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  26. 2018Learning Word Vectors for 157 Languagescited by 279
  27. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  28. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 180
  29. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  30. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  31. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  32. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  33. 2023What's In My Big Data?cited by 129
  34. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  35. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 80
  36. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  37. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  38. 2022Measuring Datacited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 31; 2026: 10.

67 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  5. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  6. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  7. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  8. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  9. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
  10. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  11. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  12. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
38 more
  1. 2024A Survey on Deep Learning for Theorem Provingcited by 55
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  4. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 47
  5. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  6. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  7. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  8. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  9. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  10. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  11. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  12. 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 26
  13. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  14. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  15. 2024MIND: Math Informed syNthetic Dialogues for Pretraining LLMscited by 14
  16. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  17. 2024Unearthing Large Scale Domain-Specific Knowledge from Public Corporacited by 12
  18. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  19. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  20. 2024ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Modelscited by 10
  21. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  22. 2025The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generationcited by 9
  23. 2024Great Memory, Shallow Reasoning: Limits of kNN-LMscited by 8
  24. 2024From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problemscited by 8
  25. 2025PretrainZero: Reinforcement Active Pretrainingcited by 8
  26. 2024SciDFM: A Large Language Model with Mixture-of-Experts for Sciencecited by 7
  27. 2024Task Oriented In-Domain Data Augmentationcited by 6
  28. 2025CritiQ: Mining Data Quality Criteria from Human Preferencescited by 6
  29. 2025AutoMathKG: The automated mathematical knowledge graph based on LLM and vector databasecited by 6
  30. 2026Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-trainingcited by 6
  31. 2025MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Modelscited by 4
  32. 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarkscited by 4
  33. 2024Mars-PO: Multi-Agent Reasoning System Preference Optimizationcited by 3
  34. 2025Function-Space Learning Ratescited by 3
  35. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  36. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  37. 2026LegalOne: A Family of Foundation Models for Reliable Legal Reasoningcited by 3
  38. 2026Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Modelscited by 3

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
  2. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes On
  3. 2025TopoAlign: A Framework for Aligning Code to Math via Topological Decomposition
  4. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  5. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline
  6. 2025MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Data
  7. 2025Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving
  8. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning