mlsanityData to 7 Oct 2026

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

arXiv 2305.14705 ↗First posted 24 May 2023cs.CLcited by 65

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 61 of its 67 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

61 of 67 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  5. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference24 Sep
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2023PaLM 2 Technical Reportcited by 1,154
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  19. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  22. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  24. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  25. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  26. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  27. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  28. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  29. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  30. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  31. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  32. 2021Hash Layers For Large Sparse Modelscited by 236
  33. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  34. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  35. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  36. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  37. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  38. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 35; 2025: 16; 2026: 5.

65 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  4. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  5. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  6. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  7. 2024How to Train Data-Efficient LLMscited by 91
  8. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  9. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  10. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  11. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  12. 2025Advancing Expert Specialization for Better MoEcited by 37
38 more
  1. 2023Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Datacited by 33
  2. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  3. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  4. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  5. 2025SkySense V2: A Unified Foundation Model for Multi-modal Remote Sensingcited by 26
  6. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  7. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  8. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  9. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  10. 2024Identity-Driven Hierarchical Role-Playing Agentscited by 18
  11. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  12. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  13. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 13
  14. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12
  15. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
  16. 2024Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approachcited by 11
  17. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  18. 2024TradExpert: Revolutionizing Trading with Mixture of Expert LLMscited by 11
  19. 2024H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMscited by 11
  20. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  21. 2024VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuningcited by 9
  22. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  23. 2024MEMoE: Enhancing Model Editing with Mixture of Experts Adaptorscited by 8
  24. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruningcited by 8
  25. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 8
  26. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  27. 2023Enabling Language Models to Implicitly Learn Self-Improvementcited by 7
  28. 2024Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Modelscited by 7
  29. 2024textttMoE-RBench: Towards Building Reliable Language Models with Sparse Mixture-of-Expertscited by 7
  30. 2024Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecastingcited by 7
  31. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
  32. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  33. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networkscited by 5
  34. 2024On the Role of Discrete Representation in Sparse Mixture of Expertscited by 4
  35. 2024Investigating Mixture of Experts in Dense Retrievalcited by 4
  36. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  37. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  38. 2024SimSMoE: Solving Representational Collapse via Similarity Measurecited by 3

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  2. 2025Mixture of Experts in Large Language Models
  3. 2025Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities
  4. 2026Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  6. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
  7. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
  8. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks