mlsanityData to 7 Oct 2026

What's In My Big Data?

arXiv 2310.20707 ↗First posted 31 Oct 2023cs.CL cs.LGcited by 129

Not on the map. Why?

Abstract (arXiv)

Large text corpora are the backbone of language models. However, we have a limited understanding of the content of these corpora, including general statistics, quality, social factors, and inclusion of evaluation data (contamination). In this work, we propose What's In My Big Data? (WIMBD), a platform and a set of sixteen analyses that allow us to reveal and compare the contents of large text corpora. WIMBD builds on two basic capabilities -- count and search -- at scale, which allows us to analyze more than 35 terabytes on a standard compute node. We apply WIMBD to ten different corpora used to train popular language models, including C4, The Pile, and RedPajama. Our analysis uncovers several surprising and previously undocumented findings about these corpora, including the high prevalence of duplicate, synthetic, and low-quality content, personally identifiable information, toxic language, and benchmark contamination. For instance, we find that about 50% of the documents in RedPajama and LAION-2B-en are duplicates. In addition, several datasets used for benchmarking models trained on such corpora are contaminated with respect to important benchmarks, including the Winograd Schema Challenge and parts of GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 15 of its 75 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

15 of 75 references.

References · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023StarCoder: may the source be with you!cited by 790
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  10. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  11. 2023SantaCoder: don't reach for the stars!cited by 169
  12. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
3 more
  1. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  2. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  3. 2022Measuring Datacited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 61; 2025: 44; 2026: 21.

129 papers.

Sort:

By area:

  1. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  3. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  4. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 112
  5. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  6. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  7. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  8. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  9. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  10. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  11. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  12. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
38 more
  1. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  2. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  3. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  4. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  5. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  6. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  7. 2024Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?cited by 30
  8. 2024On Leakage of Code Generation Evaluation Datasetscited by 29
  9. 2024TSDS: Data Selection for Task-Specific Model Finetuningcited by 28
  10. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  11. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  12. 2024Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gapscited by 24
  13. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  14. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 23
  15. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  16. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  17. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  18. 2024CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-trainingcited by 20
  19. 2024WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wildcited by 20
  20. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  21. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
  22. 2024Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?cited by 19
  23. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 19
  24. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  25. 2024Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasetscited by 18
  26. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  27. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  28. 2024On the Reliability of Large Language Models for Causal Discoverycited by 12
  29. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  30. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  31. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  32. 2025Tracing Multilingual Factual Knowledge Acquisition in Pretrainingcited by 10
  33. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Modelscited by 9
  34. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  35. 2024WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Casecited by 9
  36. 2024Automatic Histograms: Leveraging Language Models for Text Dataset Explorationcited by 8
  37. 2024Impact of Noisy Supervision in Foundation Model Learningcited by 8
  38. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8

50 most cited of 129 All 129 (JSON)