Data to 7 Oct 2026

Learning Factored Representations in a Deep Mixture of Experts

arXiv 1312.4314 ↗First posted 16 Dec 2013cs.LGcited by 275

Not on the map. Why?

Cited by 147 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associating each input with a combination of experts at each layer, yet maintains a modest model size. On a randomly translated version of the MNIST dataset, we find that the Deep Mixture of Experts automatically learns to develop location-dependent ("where") experts at the first layer, and class-specific ("what") experts at the second layer. In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones. These demonstrate effective use of all expert combinations.

Key facts. Cited by 275 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 2; 2018: 0; 2019: 3; 2020: 3; 2021: 23; 2022: 27; 2023: 39; 2024: 89; 2025: 59; 2026: 30.

275 papers.

Sort:

By area:

2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2021Scaling Vision with Sparse Mixture of Expertscited by 740
2021Dynamic Neural Networks: A Surveycited by 405
2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
2022Unified Scaling Laws for Routed Language Modelscited by 249
2021Hash Layers For Large Sparse Modelscited by 236
2024A Survey on Mixture of Experts in Large Language Modelscited by 233
2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
2021Stochastic Scene-Aware Motion Predictioncited by 203

38 more

2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
2022A Review of Sparse Expert Models in Deep Learningcited by 147
2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
2023Synthesizing Physical Character-Scene Interactionscited by 102
2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
2023Modular Deep Learningcited by 82
2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
2024MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Expertscited by 69
2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
2024Theory on Mixture-of-Experts in Continual Learningcited by 50
2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
2019Convergence Rates for Gaussian Mixtures of Expertscited by 39
2021Modeling the Sequential Dependence among Audience Multi-step Conversions with Multi-task Learning in Targeted Display Advertisingcited by 39
2022Residual Mixture of Expertscited by 39

50 most cited of 275 All 275 (JSON)