mlsanityData to 7 Oct 2026

Deduplicating Training Data Makes Language Models Better

arXiv 2107.06499 ↗First posted 14 Jul 2021cs.CL cs.LGcited by 705

Not on the map. Why?

Cited by 50 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.

Key facts. Cited by 705 papers in the map (data through 7 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

27 of 53 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2018Datasheets for Datasetscited by 1,371
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
15 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2019Defending Against Neural Fake Newscited by 553
  5. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  7. 2018A Simple Method for Commonsense Reasoningcited by 223
  8. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  9. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  10. 2020Towards Controllable Biases in Language Generationcited by 112
  11. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  12. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  13. 2019Detecting Overfitting of Deep Generative Networks via Latent Recoverycited by 44
  14. 2021Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifierscited by 11
  15. 2018Connected Components at Scale via Local Contractionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 64; 2023: 131; 2024: 192; 2025: 166; 2026: 141.

705 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2023PaLM 2 Technical Reportcited by 1,154
38 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2022Galactica: A Large Language Model for Sciencecited by 736
  7. 2023Extracting Training Data from Diffusion Modelscited by 723
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  12. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  13. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  14. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  15. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  16. 2023Scaling Data-Constrained Language Modelscited by 391
  17. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  18. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  19. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  20. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  21. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  22. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  23. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  24. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  25. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  26. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  27. 2023Challenges and Applications of Large Language Modelscited by 251
  28. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  29. 2022Data Contamination: From Memorization to Exploitationcited by 234
  30. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  31. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  32. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  33. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  34. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  35. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  36. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  37. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  38. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187

50 most cited of 705 All 705 (JSON)