mlsanityData to 7 Oct 2026

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning

arXiv 2106.03760 ↗First posted 7 Jun 2021cs.LG math.OC stat.MLcited by 114

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture is showing promising results in improving parameter sharing in multi-task learning (MTL) and in scaling high-capacity neural networks. State-of-the-art MoE models use a trainable sparse gate to select a subset of the experts for each input example. While conceptually appealing, existing sparse gates, such as Top-k, are not smooth. The lack of smoothness can lead to convergence and statistical performance issues when training with gradient-based methods. In this paper, we develop DSelect-k: a continuously differentiable and sparse gate for MoE, based on a novel binary encoding formulation. The gate can be trained using first-order methods, such as stochastic gradient descent, and offers explicit control over the number of experts to select. We demonstrate the effectiveness of DSelect-k on both synthetic and real MTL datasets with up to 128 tasks. Our experiments indicate that DSelect-k can achieve statistically significant improvements in prediction and expert selection over popular MoE gates. Notably, on a real-world, large-scale recommender system, DSelect-k achieves over 22% improvement in predictive performance compared to Top-k. We provide an open-source implementation of DSelect-k.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 25 of its 45 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

25 of 45 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  9. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  12. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
13 more
  1. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  2. 2019Sparse Sequence-to-Sequence Modelscited by 167
  3. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  4. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  5. 2019Adaptively Sparse Transformerscited by 154
  6. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  7. 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 69
  8. 2020Gradient Estimation with Stochastic Softmax Trickscited by 66
  9. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  10. 2018Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithmscited by 27
  11. 2016Decision Forests, Convolutional Networks and the Models in-Betweencited by 15
  12. 2019Flexible Multi-task Networks by Learning Parameter Allocationcited by 7
  13. 2020Small Towers Make Big Differencescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 13; 2023: 30; 2024: 37; 2025: 18; 2026: 14.

114 papers.

Sort:

By area:

  1. 2024Mixtral of Expertscited by 1,785
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  4. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  5. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  6. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  7. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  8. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  9. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  10. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  11. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 88
38 more
  1. 2023Modular Deep Learningcited by 82
  2. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  3. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  5. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  6. 2024MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmeticcited by 53
  7. 2023ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detectioncited by 48
  8. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  9. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  10. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  11. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  12. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  13. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  14. 2024Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Expertscited by 31
  15. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gatecited by 30
  16. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  17. 2023Dual-Balancing for Multi-Task Learningcited by 27
  18. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  19. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  20. 2022LibMTL: A Python Library for Multi-Task Learningcited by 25
  21. 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 25
  22. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  23. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  24. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  25. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  26. 2024Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activitycited by 22
  27. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
  28. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  29. 2023STEM: Unleashing the Power of Embeddings for Multi-task Recommendationcited by 19
  30. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  31. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  32. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  33. 2024Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Modelscited by 16
  34. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16
  35. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  36. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  37. 2023Learning by Sorting: Self-supervised Learning with Group Ordering Constraintscited by 13
  38. 2023Alternating Updates for Efficient Transformerscited by 13

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Expert Specialization in Mixture of Experts
  2. 2026Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
  3. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  4. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  5. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
  6. 2023Revisiting Single-gated Mixtures of Experts
  7. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
  8. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms