mlsanityData to 7 Oct 2026

DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection

arXiv 2502.11546 ↗First posted 17 Feb 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The rapid development of multilingual large language models (LLMs) highlights the need for high-quality, diverse, and well-curated multilingual datasets. In this paper, we introduce DCAD-2000 (Data Cleaning as Anomaly Detection), a large-scale multilingual corpus constructed from newly extracted Common Crawl data and existing multilingual sources. DCAD-2000 covers 2,282 languages, 46.72TB of text, and 8.63 billion documents, spanning 155 high- and medium-resource languages and 159 writing scripts. To overcome the limitations of existing data cleaning approaches, which rely on manually designed heuristic thresholds, we reframe data cleaning as an anomaly detection problem. This dynamic filtering paradigm substantially improves data quality by automatically identifying and removing noisy or anomalous content. By fine-tuning LLMs on DCAD-2000, we demonstrate notable improvements in data quality, robustness of the cleaning pipeline, and downstream performance, particularly for low-resource languages across multiple multilingual benchmarks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 61 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

41 of 61 references.

References · 41

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2024Large Language Models: A Surveycited by 506
  11. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  12. 2024A Survey on Data Selection for Language Modelscited by 230
29 more
  1. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  2. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  3. 2022Can Foundation Models Wrangle Your Data?cited by 173
  4. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  5. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  6. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  7. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  8. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  9. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
  10. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  11. 2019On the Use of ArXiv as a Datasetcited by 101
  12. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
  13. 2021Mean-Shifted Contrastive Loss for Anomaly Detectioncited by 87
  14. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  15. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 69
  16. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  17. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  18. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 50
  19. 2024A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontierscited by 39
  20. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  21. 2024Large Language Models for Education: A Surveycited by 32
  22. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  23. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  24. 2024Sailor: Open Language Models for South-East Asiacited by 19
  25. 2024Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipediacited by 18
  26. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  27. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  28. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  29. 2023MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in Chinacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  2. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corporacited by 6
  3. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  4. 2026Scaling Model and Data for Multilingual Machine Translation with Open Large Language Modelscited by 4
  5. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1