mlsanityData to 7 Oct 2026

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

arXiv 2309.04662 ↗First posted 9 Sep 2023cs.CL cs.LGcited by 180

Not on the map. Why?

Abstract (arXiv)

We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 9 Sep 2023.

Builds on

49 of 66 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
37 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  3. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  7. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  10. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  13. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  14. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  15. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  16. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  17. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
  18. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  19. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  20. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  21. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  22. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  23. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  24. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  25. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  26. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  27. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  28. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  29. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 40
  30. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  31. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  32. 2020Complete Multilingual Neural Machine Translationcited by 30
  33. 2022Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Taskscited by 28
  34. 2022Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learningcited by 19
  35. 2017JESC: Japanese-English Subtitle Corpuscited by 16
  36. 2019A Baseline Neural Machine Translation System for Indian Languagescited by 3
  37. 2023When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scalecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 66; 2025: 67; 2026: 36.

180 papers.

Sort:

By area:

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  3. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  4. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  5. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  6. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  7. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  8. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  9. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  10. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  11. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  12. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
38 more
  1. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  2. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  3. 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
  4. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  5. 2023GlotLID: Language Identification for Low-Resource Languagescited by 37
  6. 2023User Inference Attacks on Large Language Modelscited by 36
  7. 2024Copyright Traps for Large Language Modelscited by 36
  8. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 35
  9. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  10. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  11. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  12. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  13. 2023AfriMTE and AfriCOMET: Enhancing COMET to Embrace Under-resourced African Languagescited by 24
  14. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  15. 2025Privacy Auditing of Large Language Modelscited by 22
  16. 2024Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphscited by 21
  17. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 21
  18. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  19. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 20
  20. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  21. 2024Sailor: Open Language Models for South-East Asiacited by 19
  22. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 19
  23. 2023TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processescited by 18
  24. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  25. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  26. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  27. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 15
  28. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  29. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  30. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 15
  31. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  32. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
  33. 2024Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translationcited by 13
  34. 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translationcited by 13
  35. 2024"It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologiescited by 12
  36. 2024MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructionscited by 12
  37. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  38. 2024Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Levelcited by 11

50 most cited of 180 All 180 (JSON)