mlsanityData to 7 Oct 2026

Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners

arXiv 2204.07689 ↗First posted 16 Apr 2022cs.LG cs.CLcited by 51

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Traditional multi-task learning (MTL) methods use dense networks that use the same set of shared weights across several different tasks. This often creates interference where two or more tasks compete to pull model parameters in different directions. In this work, we study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning by specializing some weights for learning shared representations and using the others for learning task-specific information. To this end, we devise task-aware gating functions to route examples from different tasks to specialized experts which share subsets of network weights conditioned on the task. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model. We demonstrate such sparse networks to improve multi-task learning along three key dimensions: (i) transfer to low-resource tasks from related tasks in the training mixture; (ii) sample-efficient generalization to tasks not seen during training by making use of task-aware routing from seen related tasks; (iii) robustness to the addition of unrelated tasks by avoiding catastrophic forgetting of existing tasks.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 46 references that are in the map. First posted on arXiv 16 Apr 2022.

Builds on

26 of 46 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
14 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  5. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  6. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  8. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  9. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  10. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  11. 2021Hash Layers For Large Sparse Modelscited by 236
  12. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  13. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  14. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 15; 2024: 18; 2025: 9; 2026: 7.

51 papers.

Sort:

By area:

  1. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  2. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  3. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  4. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  5. 2023Modular Deep Learningcited by 82
  6. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  7. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  8. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  9. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  10. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  11. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  12. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
38 more
  1. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  2. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  3. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  4. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  5. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  6. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  7. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  8. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  9. 2023Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNNcited by 15
  10. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  11. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  12. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  13. 2024PatentGPT: A Large Language Model for Intellectual Propertycited by 8
  14. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learningcited by 7
  15. 2024Flexible and Adaptable Summarization via Expertise Separationcited by 7
  16. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  17. 2025Tabby: A Language Model Architecture for Tabular and Structured Data Synthesiscited by 4
  18. 2023Provable Pathways: Learning Multiple Tasks over Multiple Pathscited by 3
  19. 2024SimSMoE: Solving Representational Collapse via Similarity Measurecited by 3
  20. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  21. 2023JiuZhang 2.0: A Unified Chinese Pre-trained Language Model for Multi-task Mathematical Problem Solvingcited by 2
  22. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  23. 2024Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddingscited by 1
  24. 2024Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translationcited by 1
  25. 2025NAT: Learning to Attack Neurons for Enhanced Adversarial Transferabilitycited by 1
  26. 2026FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learningcited by 1
  27. 2023SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skillscited by 0
  28. 2023G-SPEED: General SParse Efficient Editing MoDelcited by 0
  29. 2024Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)cited by 0
  30. 2025Optimizing Speech Multi-View Feature Fusion through Conditional Computationcited by 0
  31. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Expertscited by 0
  32. 2025SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discoverycited by 0
  33. 2025Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognitioncited by 0
  34. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  35. 2026HMR-Net: Hierarchical Modular Routing for Cross-Domain Object Detection in Aerial Imagescited by 0
  36. 2026MTL-MAD: Multi-Task Learners are Effective Medical Anomaly Detectorscited by 0
  37. 2026MoRE: A Mixture-of-Experts-Based Task-Adaptive End-to-End Network for Multimodal MRI Reconstructioncited by 0
  38. 2026SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcyclingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  2. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts
  3. 2026LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning
  4. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  5. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity
  6. 2026Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
  7. 2025Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts
  8. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners