mlsanityData to 7 Oct 2026

Condensing Multilingual Knowledge with Lightweight Language-Specific Modules

arXiv 2305.13993 ↗First posted 23 May 2023cs.CLcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Incorporating language-specific (LS) modules is a proven method to boost performance in multilingual machine translation. This approach bears similarity to Mixture-of-Experts (MoE) because it does not inflate FLOPs. However, the scalability of this approach to hundreds of languages (experts) tends to be unmanageable due to the prohibitive number of parameters introduced by full-rank matrices in fully-connected layers. In this work, we introduce the Language-Specific Matrix Synthesis (LMS) method. This approach constructs LS modules by generating low-rank matrices from two significantly smaller matrices to approximate the full-rank matrix. Furthermore, we condense multilingual knowledge from multiple LS modules into a single shared module with the Fuse Distillation (FD) technique to improve the efficiency of inference and model serialization. We show that our LMS method significantly outperforms previous LS methods and MoE methods with the same amount of extra parameters, e.g., 1.73 BLEU points over the Switch Transformer on many-to-many multilingual machine translation. Importantly, LMS is able to have comparable translation performance with much fewer parameters.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 33 of its 43 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

33 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity3 May
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translation11 Jul
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
16 more
  1. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  2. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  3. 2019Massively Multilingual Neural Machine Translationcited by 246
  4. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  5. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  6. 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
  7. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  8. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  9. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  10. 2022Causes and Cures for Interference in Multilingual Translationcited by 23
  11. 2022Multilingual Machine Translation with Hyper-Adapterscited by 15
  12. 2022Language-Family Adapters for Low-Resource Multilingual Neural Machine Translationcited by 14
  13. 2023Learning Language-Specific Layers for Multilingual Machine Translationcited by 13
  14. 2022Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfercited by 6
  15. 2022The Importance of Being Parameters: An Intra-Distillation Method for Serious Gainscited by 6
  16. 2021Parameter Differentiation based Multilingual Neural Machine Translationcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  2. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  3. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languagescited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
  2. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model
  3. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
  4. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  5. 2024Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM
  6. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  7. 2026Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
  8. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators