mlsanityData to 7 Oct 2026

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

arXiv 2402.00159 ↗First posted 31 Jan 2024cs.CLcited by 475

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

Key facts. Cited by 475 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 160; 2025: 189; 2026: 126.

475 papers.

Sort:

By area:

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  7. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  8. 2024Generative Representational Instruction Tuningcited by 227
  9. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  10. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  11. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  12. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
38 more
  1. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  3. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  5. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  6. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  7. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  9. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  10. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  11. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  12. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  13. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  14. 2024Scaling Laws for Precisioncited by 92
  15. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  16. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  17. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  18. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  19. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  20. 2024Hyper-Connectionscited by 81
  21. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  22. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  23. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  24. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 65
  25. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  26. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  27. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  28. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  29. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  30. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 54
  31. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  32. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  33. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  34. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 49
  35. 2025Scaling Laws for Optimal Data Mixturescited by 49
  36. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  37. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 47
  38. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 46

50 most cited of 475 All 475 (JSON)