mlsanityData to 7 Oct 2026

A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

arXiv 2305.13169 ↗First posted 22 May 2023cs.CL cs.LGcited by 180

Not on the map. Why?

Abstract (arXiv)

Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guided by empirically unsupported intuitions. To address this, we pretrain 28 1.5B parameter decoder-only models, training on data curated (1) at different times, (2) with varying toxicity and quality filters, and (3) with different domain compositions. First, we quantify the effect of pretraining data age. A temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning. Second, we explore the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data. We also find that the effects of different types of filtering are not predictable from text domain characteristics. Lastly, we empirically validate that the inclusion of heterogeneous data sources, like books and web, is broadly beneficial and warrants greater prioritization. These findings constitute the largest set of experiments to validate, quantify, and expose many undocumented intuitions about text pretraining, which we hope will help support more informed data-centric decisions in LM development.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 85 of its 131 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

85 of 131 references.

References · 85

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  13. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  14. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  15. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  16. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  17. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  18. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  19. 2023PaLM 2 Technical Reportcited by 1,154
  20. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  21. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  22. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  23. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  24. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  25. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  26. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  27. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  28. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  29. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  30. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  31. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  32. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  33. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  34. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  35. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
  36. 2020The Pushshift Reddit Datasetcited by 289
  37. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  38. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 66; 2025: 60; 2026: 29.

180 papers.

Sort:

By area:

  1. 2023Textbooks Are All You Needcited by 561
  2. 2023Scaling Data-Constrained Language Modelscited by 391
  3. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  7. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  8. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  9. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  10. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
38 more
  1. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  2. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  3. 2023What's In My Big Data?cited by 129
  4. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  5. 2024How to Train Data-Efficient LLMscited by 91
  6. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  7. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  8. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  9. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  10. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
  11. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  12. 2024On the Societal Impact of Open Foundation Modelscited by 64
  13. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  14. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  15. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  16. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  17. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  18. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  19. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  20. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  21. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
  22. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 33
  23. 2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31
  24. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  25. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  26. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  27. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  28. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  29. 2024Set the Clock: Temporal Alignment of Pretrained Language Modelscited by 24
  30. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 24
  31. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  32. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  33. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 21
  34. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  35. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  36. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19
  37. 2025MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation Systemcited by 19
  38. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19

50 most cited of 180 All 180 (JSON)