mlsanityData to 7 Oct 2026

A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training

arXiv 2407.07630 ↗First posted 10 Jul 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This article presents a comprehensive review of the challenges associated with using massive web-mined corpora for the pre-training of large language models (LLMs). This review identifies key challenges in this domain, including challenges such as noise (irrelevant or misleading information), duplication of content, the presence of low-quality or incorrect information, biases, and the inclusion of sensitive or personal information in web-mined corpora. Addressing these issues is crucial for the development of accurate, reliable, and ethically responsible language models. Through an examination of current methodologies for data cleaning, pre-processing, bias detection and mitigation, we highlight the gaps in existing approaches and suggest directions for future research. Our discussion aims to catalyze advancements in developing more sophisticated and ethically responsible LLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

30 of 48 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
18 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019Defending Against Neural Fake Newscited by 553
  4. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  7. 2018Learning Word Vectors for 157 Languagescited by 279
  8. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  9. 2023Challenges and Applications of Large Language Modelscited by 251
  10. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  11. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  12. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  13. 2023What's In My Big Data?cited by 129
  14. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  15. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  16. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  17. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  18. 2021Large Language Models are not Models of Natural Language: they are Corpus Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025On the Effectiveness of Large Language Models in Automating Categorization of Scientific Textscited by 3
  2. 2025Localizing Persona Representations in LLMscited by 3
  3. 2026Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testingcited by 2
  4. 2025ExplicitLM: Decoupling Knowledge from Parameters via Explicit Memory Bankscited by 1
  5. 2026An Alternative Trajectory for Generative AIcited by 1
  6. 2025A Comparative Study of Open-Source Libraries for Synthetic Tabular Data Generation: SDV vs. SynthCitycited by 0
  7. 2026Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Modelscited by 0