mlsanityData to 7 Oct 2026

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

arXiv 2104.08692 ↗First posted 18 Apr 2021cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on eight multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
25 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  10. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  12. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  13. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  14. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  15. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  16. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  17. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  18. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  19. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  20. 2017The IIT Bombay English-Hindi Parallel Corpuscited by 74
  21. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  22. 2020ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporacited by 63
  23. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  24. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  25. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 10; 2024: 6; 2025: 2; 2026: 2.

41 papers.

Sort:

  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  5. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  6. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  7. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  8. 2022A Survey on Cross-Lingual Summarizationcited by 38
  9. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  10. 2022ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarizationcited by 25
  11. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  12. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
29 more
  1. 2022GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminatorcited by 20
  2. 2023Zero-Shot Cross-Lingual Summarization via Large Language Modelscited by 16
  3. 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
  4. 2022Understanding Translationese in Cross-Lingual Summarizationcited by 14
  5. 2024Fairness Definitions in Language Models Explainedcited by 10
  6. 2021DOCmT5: Document-Level Pretraining of Multilingual Language Modelscited by 8
  7. 2023Cross-Lingual Supervision improves Large Language Models Pre-trainingcited by 8
  8. 2025Just Go Parallel: Improving the Multilingual Capabilities of Large Language Modelscited by 8
  9. 2023Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluationcited by 7
  10. 2022Polyglot Prompt: Multilingual Multitask PrompTrainingcited by 6
  11. 2023Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generationcited by 4
  12. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4
  13. 2022Cross-Lingual Query-Based Summarization of Crisis-Related Social Media: An Abstractive Approach Using Transformerscited by 3
  14. 2022Event Extraction: A Surveycited by 3
  15. 2026OpenSeal: Good, Fast, and Cheap Construction of an Open-Source Southeast Asian LLM via Parallel Datacited by 3
  16. 2022Cross-Lingual Phrase Retrievalcited by 2
  17. 2022Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Modelscited by 2
  18. 2024Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translationcited by 2
  19. 2026Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarizationcited by 2
  20. 2022XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languagescited by 1
  21. 2023Towards Zero-Shot Functional Compositionality of Language Modelscited by 1
  22. 2023XWikiGen: Cross-lingual Summarization for Encyclopedic Text Generation in Low Resource Languagescited by 1
  23. 2024SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarizationcited by 1
  24. 2024A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectivescited by 1
  25. 2021Cross-Lingual Language Model Meta-Pretrainingcited by 0
  26. 2023PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generationcited by 0
  27. 2023Fast Training of NMT Model with Data Sortingcited by 0
  28. 2024The Role of Handling Attributive Nouns in Improving Chinese-To-English Machine Translationcited by 0
  29. 2025An Empirical Study of Many-to-Many Summarization with Large Language Modelscited by 0