mlsanityData to 7 Oct 2026

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

arXiv 2103.12028 ↗First posted 22 Mar 2021cs.CL cs.AIcited by 241

Not on the map. Why?

Abstract (arXiv)

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

Key facts. Cited by 241 papers in the map (data through 7 Oct 2026). Builds on 39 of its 75 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

39 of 75 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2018Datasheets for Datasetscited by 1,371
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
27 more
  1. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  2. 2019CamemBERT: a Tasty French Language Modelcited by 368
  3. 2016FastText.zip: Compressing text classification modelscited by 331
  4. 2018Learning Word Vectors for 157 Languagescited by 279
  5. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  6. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  7. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  8. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  9. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  10. 2020German's Next Language Modelcited by 144
  11. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  12. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  13. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  14. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  15. 2018The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standardscited by 97
  16. 2020RobBERT: a Dutch RoBERTa-based Language Modelcited by 88
  17. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  18. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  19. 2019Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translationcited by 56
  20. 2020The birth of Romanian BERTcited by 44
  21. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  22. 2020AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understandingcited by 43
  23. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  24. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 37
  25. 2020Pitfalls in Machine Learning Research: Reexamining the Development Cyclecited by 17
  26. 2021HeBERT & HebEMO: a Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognitioncited by 14
  27. 2017Natural Language Processing with Small Feed-Forward Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 63; 2023: 49; 2024: 49; 2025: 29; 2026: 34.

241 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2021Ethical and social risks of harm from Language Modelscited by 957
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  11. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  12. 2022Data Contamination: From Memorization to Exploitationcited by 234
38 more
  1. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  3. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  4. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2022In-context Examples Selection for Machine Translationcited by 172
  7. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  8. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  9. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  10. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  11. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  12. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  13. 2023Auditing large language models: a three-layered approachcited by 135
  14. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  15. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  16. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  17. 2021Survey of Low-Resource Machine Translationcited by 92
  18. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  19. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  20. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  21. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  22. 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
  23. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  24. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  25. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  26. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  27. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  28. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  29. 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
  30. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 65
  31. 2021AraT5: Text-to-Text Transformers for Arabic Language Generationcited by 63
  32. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  33. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  34. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 57
  35. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  36. 2022Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP Worldcited by 55
  37. 2023An Open Dataset and Model for Language Identificationcited by 49
  38. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45

50 most cited of 241 All 241 (JSON)