mlsanityData to 14 Oct 2026

OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

arXiv 2310.06786 ↗First posted 10 Oct 2023cs.AI cs.CL cs.LGcited by 155

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.

Key facts. Cited by 155 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 34 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

23 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun
  2. 2020Generative Language Modeling for Automated Theorem Proving7 Sep

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
9 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  4. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  5. 2016FastText.zip: Compressing text classification modelscited by 335
  6. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 192
  7. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
  8. 2023Evaluating Language Models for Mathematics through Interactionscited by 70
  9. 2021FastWARC: Optimizing Large-Scale Web Archive Analyticscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 52; 2025: 61; 2026: 37.

155 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  3. 2023Llemma: An Open Language Model For Mathematicscited by 388
  4. 2024LoRA Learns Less and Forgets Lesscited by 328
  5. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  7. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  8. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  9. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 125
  10. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  11. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  12. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
38 more
  1. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  2. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  3. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  4. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  5. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
  6. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 81
  7. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  8. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  9. 2024Augmenting Math Word Problems via Iterative Question Composingcited by 59
  10. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  11. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  12. 2024A Survey on Deep Learning for Theorem Provingcited by 57
  13. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  14. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  15. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 45
  16. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 41
  17. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  18. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  19. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  20. 2024InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoningcited by 36
  21. 2025LLM Pretraining with Continuous Conceptscited by 36
  22. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  23. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  24. 2025Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluationcited by 29
  25. 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 27
  26. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 27
  27. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 26
  28. 2025AutoData: A Multi-Agent System for Open Web Data Collectioncited by 25
  29. 2023Mathematical Language Models: A Surveycited by 24
  30. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
  31. 2024LangBridge: Multilingual Reasoning Without Multilingual Supervisioncited by 22
  32. 2025PromptCoT: Synthesizing Olympiad-level Problems for Mathematical Reasoning in Large Language Modelscited by 22
  33. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
  34. 2025Mid-Training of Large Language Models: A Surveycited by 21
  35. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  36. 2024Language Models as Science Tutorscited by 17
  37. 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 16
  38. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learningcited by 14

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  2. 2025RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
  3. 2026PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
  4. 2026CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions
  5. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
  6. 2025MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Data
  7. 2026MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
  8. 2026ResearchMath-14K: Scaling Research-Level Mathematics via Agents