mlsanityData to 7 Oct 2026

Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model

arXiv 2212.09811 ↗First posted 19 Dec 2022cs.CL cs.AI cs.LGcited by 36

Not on the map. Why?

Cited by 24 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that enables the removal of up to 80% of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

34 of 41 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
22 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  5. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  10. 2019Massively Multilingual Neural Machine Translationcited by 246
  11. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  12. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  13. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  14. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  15. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  16. 2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
  17. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  18. 2020Scalable Transfer Learning with Expert Modelscited by 48
  19. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  20. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  21. 2022Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformerscited by 9
  22. 2021Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?cited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 5; 2026: 12.

36 papers.

Sort:

By area:

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  3. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  4. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 57
  5. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  6. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  7. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  8. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  9. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 38
  10. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  11. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  12. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 19
24 more
  1. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  2. 2024Turn Waste into Worth: Rectifying Top-k Router of MoEcited by 11
  3. 2023Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasetscited by 9
  4. 2023ParameterNet: Parameters Are All You Needcited by 7
  5. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 7
  6. 2025UniSS: Unified Expressive Speech-to-Speech Translation with Your Voicecited by 6
  7. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  8. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  9. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  10. 2024Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languagescited by 3
  11. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  12. 2024Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translationcited by 1
  13. 2024Efficient Technical Term Translation: A Knowledge Distillation Approach for Parenthetical Terminology Translationcited by 1
  14. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstructioncited by 1
  15. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Modelscited by 1
  16. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 1
  17. 2025Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairscited by 0
  18. 2026Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guaranteescited by 0
  19. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translationcited by 0
  20. 2026Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generationcited by 0
  21. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  22. 2026UMoE:Unlocking Every Expert in Domain-Specific Trainingcited by 0
  23. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Modelscited by 0
  24. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pruning Laws for Large Language Models
  2. 2025ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
  3. 2024Pruning Multilingual Large Language Models for Multilingual Inference
  4. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  5. 2026Optimizing Korean-Centric LLMs via Token Pruning
  6. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  8. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models