mlsanityData to 14 Oct 2026

D4: Improving LLM Pretraining via Document De-Duplication and Diversification

arXiv 2308.12284 ↗First posted 23 Aug 2023cs.CL cs.AI cs.LGcited by 150

Not on the map. Why?

Abstract (arXiv)

Over recent years, an increasing amount of compute and data has been poured into training large language models (LLMs), usually by doing one-pass learning on as many tokens as possible randomly selected from large-scale web corpora. While training on ever-larger portions of the internet leads to consistent performance improvements, the size of these improvements diminishes with scale, and there has been little work exploring the effect of data selection on pre-training and downstream performance beyond simple de-duplication methods such as MinHash. Here, we show that careful data selection (on top of de-duplicated data) via pre-trained model embeddings can speed up training (20% efficiency gains) and improves average downstream accuracy on 16 NLP tasks (up to 2%) at the 6.7B model scale. Furthermore, we show that repeating data intelligently consistently outperforms baseline training (while repeating random data performs worse than baseline training). Our results indicate that clever data selection can significantly improve LLM pre-training, calls into question the common practice of training for a single epoch on as much data as possible, and demonstrates a path to keep improving our models past the limits of randomly sampling web data.

Key facts. Cited by 150 papers in the map (data through 14 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

50 of 65 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  15. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  17. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  18. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  19. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  20. 2020Dataset Condensation with Gradient Matchingcited by 516
  21. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  22. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  23. 2023Scaling Data-Constrained Language Modelscited by 403
  24. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  25. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  26. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  27. 2020The Pushshift Reddit Datasetcited by 292
  28. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  29. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  30. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  31. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  32. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  33. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  34. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  35. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  36. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  37. 2019Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Needcited by 43
  38. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 54; 2025: 58; 2026: 34.

150 papers.

Sort:

By area:

  1. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  2. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  3. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  4. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  5. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  6. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  7. 2024How to Train Data-Efficient LLMscited by 93
  8. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  9. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  10. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 87
  11. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  12. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
38 more
  1. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  2. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  4. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  5. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  6. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  7. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 47
  8. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
  9. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  10. 2024Data Selection via Optimal Control for Language Modelscited by 38
  11. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  12. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  13. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  14. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  15. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 33
  16. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  17. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  18. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 26
  19. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 25
  20. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 23
  21. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  22. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 23
  23. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  24. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  25. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  26. 2023Data Management For Training Large Language Models: A Surveycited by 17
  27. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  28. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  29. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  30. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
  31. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  32. 2025Efficient Data Selection at Scale via Influence Distillationcited by 15
  33. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  34. 2025Token Weighting for Long-Range Language Modelingcited by 14
  35. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  36. 2025Group-Level Data Selection for Efficient Pretrainingcited by 13
  37. 2023Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Datacited by 12
  38. 2024SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-trainingcited by 12

50 most cited of 150 All 150 (JSON)