mlsanityData to 7 Oct 2026

Learning Language Specific Sub-network for Multilingual Machine Translation

arXiv 2105.09259 ↗First posted 19 May 2021cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

Multilingual neural machine translation aims at learning a single translation model for multiple languages. These jointly trained models often suffer from performance degradation on rich-resource language pairs. We attribute this degeneration to parameter interference. In this paper, we propose LaSS to jointly train a single unified multilingual MT model. LaSS learns Language Specific Sub-network (LaSS) for each language pair to counter parameter interference. Comprehensive experiments on IWSLT and WMT datasets with various Transformer architectures show that LaSS obtains gains on 36 language pairs by up to 1.2 BLEU. Besides, LaSS shows its strong generalization performance at easy extension to new language pairs and zero-shot translation.LaSS boosts zero-shot translation with an average of 8.3 BLEU on 30 language pairs. Codes and trained models are available at https://github.com/NLP-Playground/LaSS.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 31 of its 35 references that are in the map. First posted on arXiv 19 May 2021.

Builds on

31 of 35 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2017Learning multiple visual domains with residual adapterscited by 553
19 more
  1. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  2. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  3. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  6. 2019Massively Multilingual Neural Machine Translationcited by 246
  7. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  8. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  9. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  10. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  11. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  12. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  13. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  14. 2018Parameter Sharing Methods for Multilingual Self-Attentional Translation Modelscited by 33
  15. 2019Learning Sparse Sharing Architectures for Multiple Taskscited by 31
  16. 2018Variational Recurrent Neural Machine Translationcited by 23
  17. 2020Deep Transformers with Latent Depthcited by 17
  18. 2020Revisiting Modularized Multilingual NMT to Meet Industrial Demandscited by 7
  19. 2018Towards Linear Time Neural Machine Translation with Capsule Networkscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 19; 2023: 16; 2024: 7; 2025: 3.

48 papers.

Sort:

By area:

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2023Modular Deep Learningcited by 82
  3. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  4. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  5. 2021Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Taskcited by 34
  6. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  7. 2021Learning Kernel-Smoothed Machine Translation with Retrieved Examplescited by 25
  8. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  9. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  10. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  11. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  12. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15
36 more
  1. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14
  2. 2023Examining Modularity in Multilingual LMs via Language-Specialized Subnetworkscited by 13
  3. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  4. 2024Conditional computation in neural networks: principles and research trendscited by 10
  5. 2022Understanding and Mitigating the Uncertainty in Zero-Shot Translationcited by 9
  6. 2023Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translationcited by 9
  7. 2023Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learningcited by 9
  8. 2022Adapting to Non-Centered Languages for Zero-shot Multilingual Translationcited by 6
  9. 2022Improving Multilingual Neural Machine Translation System for Indic Languagescited by 6
  10. 2023LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languagescited by 6
  11. 2021Parameter Differentiation based Multilingual Neural Machine Translationcited by 5
  12. 2022Is Encoder-Decoder Redundant for Neural Machine Translation?cited by 5
  13. 2022Improving Neural Machine Translation by Denoising Trainingcited by 4
  14. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 4
  15. 2022Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiencycited by 3
  16. 2023Condensing Multilingual Knowledge with Lightweight Language-Specific Modulescited by 3
  17. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3
  18. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  19. 2023Extending Multilingual Machine Translation through Imitation Learningcited by 3
  20. 2022Unifying the Convergences in Multilingual Neural Machine Translationcited by 2
  21. 2023Language-Aware Multilingual Machine Translation with Self-Supervised Learningcited by 2
  22. 2024Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applicationscited by 2
  23. 2024MAMMOTH: Massively Multilingual Modular Open Translation @ Helsinkicited by 2
  24. 2024More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routingcited by 2
  25. 2022Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translationcited by 1
  26. 2022Informative Language Representation Learning for Massively Multilingual Neural Machine Translationcited by 1
  27. 2022Searching for Effective Multilingual Fine-Tuning Methods: A Case Study in Summarizationcited by 1
  28. 2023Towards Faster k-Nearest-Neighbor Machine Translationcited by 1
  29. 2022Adversarial RAW: Image-Scaling Attack Against Imaging Pipelinecited by 0
  30. 2022Domain Specific Sub-network for Multi-Domain Neural Machine Translationcited by 0
  31. 2023One-stop Training of Multiple Capacity Modelscited by 0
  32. 2023Exploring Representational Disparities Between Multilingual and Bilingual Translation Modelscited by 0
  33. 2023Bridging the Domain Gaps in Context Representations for k-Nearest Neighbor Neural Machine Translationcited by 0
  34. 2024Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixturescited by 0
  35. 2024Exploring Intrinsic Language-specific Subspaces in Fine-tuning Multilingual Neural Machine Translationcited by 0
  36. 2025FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translationcited by 0