mlsanityData to 7 Oct 2026

Deep Mixture of Experts via Shallow Embedding

arXiv 1806.01531 ↗First posted 5 Jun 2018cs.CVcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Larger networks generally have greater representational power at the cost of increased computational complexity. Sparsifying such networks has been an active area of research but has been generally limited to static regularization or dynamic approaches using reinforcement learning. We explore a mixture of experts (MoE) approach to deep dynamic routing, which activates certain experts in the network on a per-example basis. Our novel DeepMoE architecture increases the representational power of standard convolutional networks by adaptively sparsifying and recalibrating channel-wise features in each convolutional layer. We employ a multi-headed sparse gating network to determine the selection and scaling of channels for each input, leveraging exponential combinations of experts within a single convolutional network. Our proposed architecture is evaluated on four benchmark datasets and tasks, and we show that Deep-MoEs are able to achieve higher accuracy with lower computation than standard convolutional networks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 38 of its 53 references that are in the map. First posted on arXiv 5 Jun 2018.

Builds on

38 of 53 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  11. 2016Matching Networks for One Shot Learningcited by 2,246
  12. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
26 more
  1. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
  4. 2016HyperNetworkscited by 1,003
  5. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  6. 2014Neural Turing Machinescited by 630
  7. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  8. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  9. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  10. 2016Visual Relationship Detection with Language Priorscited by 333
  11. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  12. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  13. 2013Predicting Parameters in Deep Learningcited by 223
  14. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  15. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  16. 2018Low-Shot Learning from Imaginary Datacited by 181
  17. 2017Modulating early visual processing by languagecited by 167
  18. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  19. 2015Label-Embedding for Image Classificationcited by 164
  20. 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
  21. 2017Generalized Zero-Shot Learning via Synthesized Examplescited by 114
  22. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  23. 2016Learning feed-forward one-shot learnerscited by 93
  24. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  25. 2015Zero-Shot Learning via Joint Latent Similarity Embeddingcited by 39
  26. 2018Attributes as Operators: Factorizing Unseen Attribute-Object Compositionscited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  2. 2022Towards Understanding Mixture of Experts in Deep Learning
  3. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
  6. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
  7. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  8. 2026MoRE: Mixture of Reused Experts