mlsanityData to 7 Oct 2026

SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

arXiv 2606.17952 ↗First posted 16 Jun 2026cs.LG cs.AIcited by 3

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-k routing. While this preserves causality and suits autoregressive language models, the discrete top-k operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-k LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts. The source code is publicly available^dagger.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 35 of its 44 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

35 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
21 more
  1. 2023Scaling Data-Constrained Language Modelscited by 391
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  4. 2023From Sparse to Soft Mixtures of Expertscited by 245
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2023Learning A Sparse Transformer Network for Effective Image Derainingcited by 204
  7. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  8. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  9. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  10. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  11. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  12. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  13. 2018Smooth Loss Functions for Deep Top-k Classificationcited by 46
  14. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  15. 2015Loss Functions for Top-k Error: Analysis and Insightscited by 38
  16. 2022Sparsity-Constrained Optimal Transportcited by 32
  17. 2024Cardinality-Aware Set Prediction and Top-k Classificationcited by 23
  18. 2025LapSum -- One Method to Differentiate Them All: Ranking, Sorting and Top-k Selectioncited by 12
  19. 2022Stochastic smoothing of the top-K calibrated hinge loss for deep imbalanced classificationcited by 10
  20. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregationcited by 5
  21. 2024Top-k Classification and Cardinality-Aware Predictioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMscited by 1
  2. 2026Detecting a Route Flip Is Easier Than Knowing Whether to Fix It: Causal Route-Mediated Damage in Quantized Mixture-of-Expertscited by 1
  3. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DirMoE: Dirichlet-routed Mixture of Experts
  2. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  3. 2025ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
  4. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  5. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  6. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  7. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  8. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs