mlsanityData to 7 Oct 2026

A Review of Sparse Expert Models in Deep Learning

arXiv 2209.01667 ↗First posted 4 Sep 2022cs.LG cs.CLcited by 147

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and others, all with the unifying idea that each example is acted on by a subset of the parameters. By doing so, the degree of sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models. The resulting models have demonstrated significant improvements across diverse domains such as natural language processing, computer vision, and speech recognition. We review the concept of sparse expert models, provide a basic description of the common algorithms, contextualize the advances in the deep learning era, and conclude by highlighting areas for future work.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 69 of its 90 references that are in the map. First posted on arXiv 4 Sep 2022.

Builds on

69 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Residual Mixture of Experts20 Apr
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  15. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  16. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  17. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  18. 2019The State of Sparsity in Deep Neural Networkscited by 471
  19. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  20. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  22. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  23. 2022Confident Adaptive Language Modelingcited by 297
  24. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  25. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  26. 2021Hash Layers For Large Sparse Modelscited by 236
  27. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  28. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  29. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  30. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  31. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  32. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  33. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  34. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  35. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  36. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 128
  37. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  38. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 33; 2024: 59; 2025: 33; 2026: 15.

147 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,785
  2. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  3. 2022Efficiently Scaling Transformer Inferencecited by 529
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  8. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  9. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  10. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  11. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
38 more
  1. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  2. 2023Modular Deep Learningcited by 82
  3. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  4. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 65
  5. 2023Explaining black box text modules in natural language with language modelscited by 61
  6. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  7. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  8. 2024RouteFinder: Towards Foundation Models for Vehicle Routing Problemscited by 54
  9. 2024Upcycling Large Language Models into Mixture of Expertscited by 48
  10. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  11. 2024MedCoT: Medical Chain of Thought via Hierarchical Expertcited by 46
  12. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  13. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  14. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  15. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  16. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  17. 2025Advancing Expert Specialization for Better MoEcited by 37
  18. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  19. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  20. 2024AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Expertscited by 33
  21. 2022Sparsity-Constrained Optimal Transportcited by 32
  22. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  23. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  24. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Expertscited by 27
  25. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  26. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  27. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  28. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  29. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  30. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  31. 2024Generalizing Motion Planners with Mixture of Experts for Autonomous Drivingcited by 22
  32. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  33. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  34. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20
  35. 2023GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Expertscited by 19
  36. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  37. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenescited by 19
  38. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability
  2. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
  3. 2018Deep Mixture of Experts via Shallow Embedding
  4. 2023From Sparse to Soft Mixtures of Experts
  5. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  6. 2026Sparsely gated tiny linear experts
  7. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  8. 2021Sparse MoEs meet Efficient Ensembles