mlsanityData to 7 Oct 2026

Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

arXiv 2110.03742 ↗First posted 24 Sep 2021cs.CL cs.LGcited by 96

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. However, MoE models are prohibitively large and practitioners often resort to methods such as distillation for serving. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from large sparse models. On WMT, our task-MoE with 32 experts (533M parameters) outperforms the best performing token-level MoE model (token-MoE) by +1.0 BLEU on average across 30 language pairs. The peak inference throughput is also improved by a factor of 1.9x when we route by tasks instead of tokens. While distilling a token-MoE to a smaller dense model preserves only 32% of the BLEU gains, our sub-network task-MoE, by design, preserves all the gains with the same inference cost as the distilled student model. Finally, when scaling up to 200 language pairs, our 128-expert task-MoE (13B parameters) performs competitively with a token-level counterpart, while improving the peak inference throughput by a factor of 2.6x.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 33 of its 50 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

33 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2016Sequence-Level Knowledge Distillationcited by 761
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  12. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
19 more
  1. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  2. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  3. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  4. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  5. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  6. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  7. 2019Depth-Adaptive Transformercited by 227
  8. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  9. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  10. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  11. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  12. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  13. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  14. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  15. 2019Domain, Translationese and Noise in Synthetic Data for Neural Machine Translationcited by 26
  16. 2016Continuous multilinguality with language vectorscited by 20
  17. 2020Deep Transformers with Latent Depthcited by 17
  18. 2018Emerging Language Spaces Learned From Massively Multilingual Corporacited by 9
  19. 2019Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 33; 2024: 26; 2025: 13; 2026: 7.

96 papers.

Sort:

By area:

  1. 2022Efficiently Scaling Transformer Inferencecited by 529
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  4. 2022Confident Adaptive Language Modelingcited by 297
  5. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  6. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  7. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  10. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  11. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
38 more
  1. 2023Modular Deep Learningcited by 82
  2. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  3. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  4. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  5. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  6. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  7. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  8. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  9. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  10. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  11. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  12. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  13. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  14. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  15. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  16. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  17. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  18. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  19. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  20. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  21. 2022Combining Modular Skills in Multitask Learningcited by 27
  22. 2023Dissociating language and thought in large language modelscited by 27
  23. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  24. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  25. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  26. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  27. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  28. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  29. 2023Improving Expert Specialization in Mixture of Expertscited by 16
  30. 2024Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Expertscited by 16
  31. 2023Learning Language-Specific Layers for Multilingual Machine Translationcited by 13
  32. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  33. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  34. 2023WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removalcited by 11
  35. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  36. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inferencecited by 11
  37. 2022SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understandingcited by 10
  38. 2023Towards More Unified In-context Visual Understandingcited by 10

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multilingual Routing in Mixture-of-Experts
  2. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  3. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
  4. 2025Mixture of Experts in Large Language Models
  5. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  6. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
  7. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models