mlsanityData to 13 Oct 2026

Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

arXiv 2104.08758 ↗First posted 18 Apr 2021cs.CL cs.AIcited by 485

Not on the map. Why?

Cited by 26 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.

Key facts. Cited by 485 papers in the map (data through 13 Oct 2026). Builds on 48 of its 89 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

48 of 89 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2020Scaling Laws for Neural Language Modelscited by 6,494
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  11. 2020Extracting Training Data from Large Language Modelscited by 2,181
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
36 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,815
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,556
  4. 2018Datasheets for Datasetscited by 1,377
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  8. 2015Teaching Machines to Read and Comprehendcited by 1,045
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,042
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2018Neural Network Acceptability Judgmentscited by 957
  12. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  14. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 804
  15. 2019Defending Against Neural Fake Newscited by 559
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  17. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  18. 2020Scaling Laws for Autoregressive Generative Modelingcited by 514
  19. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  20. 2021Dynabench: Rethinking Benchmarking in NLPcited by 452
  21. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  22. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 242
  23. 2018A Simple Method for Commonsense Reasoningcited by 223
  24. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  25. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  26. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  27. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  28. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  29. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  30. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  31. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
  32. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  33. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  34. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  35. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  36. 2016Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corporacited by 19

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 9; 2022: 40; 2023: 102; 2024: 125; 2025: 110; 2026: 99.

485 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2021Ethical and social risks of harm from Language Modelscited by 964
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  9. 2023BloombergGPT: A Large Language Model for Financecited by 932
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 636
38 more
  1. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 603
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 461
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 347
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  7. 2023Challenges and Applications of Large Language Modelscited by 253
  8. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 244
  9. 2022Data Contamination: From Memorization to Exploitationcited by 236
  10. 2023Proving Test Set Contamination in Black Box Language Modelscited by 228
  11. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  12. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  13. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 194
  14. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  15. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  16. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 180
  17. 2022Language models show human-like content effects on reasoning taskscited by 172
  18. 2024Quantifying the Persona Effect in LLM Simulationscited by 166
  19. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 162
  20. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 156
  21. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 154
  22. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  23. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 149
  24. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  25. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 145
  26. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  27. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  28. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 141
  29. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 137
  30. 2023Auditing large language models: a three-layered approachcited by 136
  31. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  32. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  33. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  34. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 121
  35. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 117
  36. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  37. 2024Language models scale reliably with over-training and on downstream taskscited by 109
  38. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 105

50 most cited of 485 All 485 (JSON)