mlsanityData to 7 Oct 2026

From Sparse to Soft Mixtures of Experts

arXiv 2308.00951 ↗First posted 2 Aug 2023cs.LG cs.AI cs.CVcited by 245

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40x more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 28 of its 33 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

28 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 27

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2021Scaling Vision Transformerscited by 993
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2020On Layer Normalization in the Transformer Architecturecited by 906
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
15 more
  1. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  2. 2017Self-Normalizing Neural Networkscited by 591
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  5. 2020Conditional Convolutions for Instance Segmentationcited by 314
  6. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  7. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  8. 2021Hash Layers For Large Sparse Modelscited by 236
  9. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  10. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  11. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  12. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  13. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  14. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  15. 2022Sparsity-Constrained Optimal Transportcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 66; 2025: 73; 2026: 95.

245 papers.

Sort:

By area:

  1. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  2. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  3. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  4. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  5. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  6. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  7. 2024Complexity Experts are Task-Discriminative Learners for Any Image Restorationcited by 72
  8. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  9. 2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
  10. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  11. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  12. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
38 more
  1. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 48
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  4. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  5. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  6. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  7. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  8. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  9. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  10. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 27
  11. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  12. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  13. 2024See More Details: Efficient Image Super-Resolution by Experts Miningcited by 23
  14. 2024Adapting to Distribution Shift by Visual Domain Prompt Generationcited by 23
  15. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  16. 2024Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shiftcited by 23
  17. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 23
  18. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  19. 2024Routers in Vision Mixture of Experts: An Empirical Studycited by 22
  20. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  21. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  22. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  23. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  24. 2024The Open Source Advantage in Large Language Models (LLMs)cited by 18
  25. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  26. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  27. 2025Hypergraph Vision Transformers: Images are More than Nodes, More than Edgescited by 16
  28. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  29. 2024MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restorationcited by 15
  30. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
  31. 2024MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Expertscited by 15
  32. 2025Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMscited by 15
  33. 2025Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injectioncited by 15
  34. 2024Mastering Chess with a Transformer Modelcited by 14
  35. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  36. 2024A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Expertscited by 13
  37. 2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
  38. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Sparse MoEs meet Efficient Ensembles
  2. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  3. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability
  6. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  7. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
  8. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks