mlsanityData to 7 Oct 2026

Contrastive Learning for Many-to-many Multilingual Neural Machine Translation

arXiv 2105.09501 ↗First posted 20 May 2021cs.CL cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

Existing multilingual machine translation approaches mainly focus on English-centric directions, while the non-English directions still lag behind. In this work, we aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions. Our intuition is based on the hypothesis that a universal cross-language representation leads to better multilingual translation performance. To this end, we propose mRASP2, a training method to obtain a single unified multilingual translation model. mRASP2 is empowered by two techniques: a) a contrastive learning scheme to close the gap among representations of different languages, and b) data augmentation on both multiple parallel and monolingual data to further align token representations. For English-centric directions, mRASP2 outperforms existing best unified model and achieves competitive or even better performance than the pre-trained and fine-tuned model mBART on tens of WMT's translation directions. For non-English directions, mRASP2 achieves an improvement of average 10+ BLEU compared with the multilingual Transformer baseline. Code, data and trained models are available at https://github.com/PANXiao1994/mRASP2.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  12. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
28 more
  1. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  4. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  5. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  6. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2019Massively Multilingual Neural Machine Translationcited by 246
  9. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  10. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  13. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  14. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  15. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  16. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  17. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  18. 2018A neural interlingua for multilingual machine translationcited by 29
  19. 2019Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabulariescited by 27
  20. 2019Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translationcited by 23
  21. 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
  22. 2019Consistency by Agreement in Zero-shot Neural Machine Translationcited by 22
  23. 2017Enabling Multi-Source Neural Machine Translation By Concatenating Source Sentences In Multiple Languagescited by 10
  24. 2018Zero-Resource Neural Machine Translation with Multi-Agent Communication Gamecited by 8
  25. 2019Multilingual Multi-Domain Adaptation Approaches for Neural Machine Translationcited by 8
  26. 2020Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoderscited by 8
  27. 2017Effective Strategies in Zero-Shot Neural Machine Translationcited by 6
  28. 2017Improving a Multi-Source Neural Machine Translation Model with Corpus Extension for Low-Resource Languagescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 39; 2023: 25; 2024: 20; 2025: 10; 2026: 1.

108 papers.

Sort:

By area:

  1. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  2. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  3. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  4. 2021Survey of Low-Resource Machine Translationcited by 92
  5. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  6. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  7. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  8. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 47
  9. 2021Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Taskcited by 34
  10. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  11. 2023Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignmentcited by 33
  12. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
38 more
  1. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  2. 2024Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluationcited by 25
  3. 2021MTG: A Benchmark Suite for Multilingual Text Generationcited by 22
  4. 2021Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillationcited by 22
  5. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  6. 2021LightSeq2: Accelerated Training for Transformer-based Models on GPUscited by 19
  7. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  8. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  9. 2022CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translationcited by 17
  10. 2022CREATER: CTR-driven Advertising Text Generation with Controlled Pre-Training and Contrastive Fine-Tuningcited by 16
  11. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  12. 2023SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectivescited by 15
  13. 2023Multi-level Contrastive Learning for Script-based Character Understandingcited by 15
  14. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  15. 2022Universal Conditional Masked Language Pre-training for Neural Machine Translationcited by 12
  16. 2022Neural Machine Translation with Contrastive Translation Memoriescited by 12
  17. 2021Frequency-Aware Contrastive Learning for Neural Machine Translationcited by 11
  18. 2022Towards Better Chinese-centric Neural Machine Translation for Low-resource Languagescited by 11
  19. 2021Multilingual Translation via Grafting Pre-trained Language Modelscited by 10
  20. 2022A Token-level Contrastive Framework for Sign Language Translationcited by 10
  21. 2022GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translationcited by 10
  22. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  23. 2023Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularizationcited by 10
  24. 2023Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performancecited by 10
  25. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  26. 2021Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Informationcited by 9
  27. 2021CO2Sum:Contrastive Learning for Factual-Consistent Abstractive Summarizationcited by 9
  28. 2022Understanding and Mitigating the Uncertainty in Zero-Shot Translationcited by 9
  29. 2023Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformerscited by 9
  30. 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
  31. 2021Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translationcited by 8
  32. 2022R2D2: Robust Data-to-Text with Replacement Detectioncited by 8
  33. 2022Momentum Calibration for Text Generationcited by 8
  34. 2022MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generationcited by 7
  35. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 7
  36. 2024Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Losscited by 7
  37. 2022Adapting to Non-Centered Languages for Zero-shot Multilingual Translationcited by 6
  38. 2022Improving Multilingual Neural Machine Translation System for Indic Languagescited by 6

50 most cited of 108 All 108 (JSON)