mlsanityData to 7 Oct 2026

mT5: A massively multilingual pre-trained text-to-text transformer

arXiv 2010.11934 ↗First posted 22 Oct 2020cs.CLcited by 1,813

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

Key facts. Cited by 1,813 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 159; 2022: 301; 2023: 417; 2024: 469; 2025: 307; 2026: 160.

1,813 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
38 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2023A Comprehensive Overview of Large Language Modelscited by 639
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  11. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  12. 2024Large Language Models: A Surveycited by 506
  13. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  14. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  15. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  16. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  17. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  18. 2023Scaling Data-Constrained Language Modelscited by 391
  19. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  20. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  21. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  22. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  23. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  24. 2022UL2: Unifying Language Learning Paradigmscited by 292
  25. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  26. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  27. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  28. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  29. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  30. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  31. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  32. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  33. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  34. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  35. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  36. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  37. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  38. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233

50 most cited of 1,813 All 1,813 (JSON)