mlsanityData to 14 Oct 2026

mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations

arXiv 2305.14224 ↗First posted 23 May 2023cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

50 of 62 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
38 more
  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2017Learning multiple visual domains with residual adapterscited by 557
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  7. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  9. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  11. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  12. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  13. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  14. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  15. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  16. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  17. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  18. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  19. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  20. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  21. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  22. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  23. 2020UDapter: Language Adaptation for Truly Universal Dependency Parsingcited by 77
  24. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
  25. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  26. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  27. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  28. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  29. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  30. 2021Efficient Test Time Adapter Ensembling for Low-resource Language Varietiescited by 26
  31. 2019Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slicescited by 23
  32. 2020Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfercited by 23
  33. 2021Multilingual Unsupervised Neural Machine Translation with Denoising Adapterscited by 22
  34. 2021Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapterscited by 21
  35. 2020MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scalecited by 20
  36. 2020Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMTcited by 18
  37. 2021Tricks for Training Sparse Translation Modelscited by 17
  38. 2020Parsing with Multilingual BERT, a Small Corpus, and a Small Treebankcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 4; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  2. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  3. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  4. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  5. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  6. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  7. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  8. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  9. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  10. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  11. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  12. 2024Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative taskscited by 11
12 more
  1. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  2. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  3. 2024Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysiscited by 7
  4. 2025Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Modelscited by 7
  5. 2023Insights into Classifying and Mitigating LLMs' Hallucinationscited by 6
  6. 2024Soft Language Prompts for Language Transfercited by 5
  7. 2024Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languagescited by 1
  8. 2025MALM: A Multi-Information Adapter for Large Language Models to Mitigate Hallucinationcited by 1
  9. 2024Language-Independent Representations Improve Zero-Shot Summarizationcited by 0
  10. 2024Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsingcited by 0
  11. 2024QUILL: Quotation Generation Enhancement of Large Language Modelscited by 0
  12. 2026Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Preventioncited by 0