mlsanityData to 14 Oct 2026

Routers in Vision Mixture of Experts: An Empirical Study

arXiv 2401.15969 ↗First posted 29 Jan 2024cs.CV cs.AI cs.LGcited by 23

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. A key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens). In this paper, we present a comprehensive study of routers in MoEs for computer vision tasks. We introduce a unified MoE formulation that subsumes different MoEs with two parametric routing tensors. This formulation covers both sparse MoE, which uses a binary or hard assignment between experts and tokens, and soft MoE, which uses a soft assignment between experts and weighted combinations of tokens. Routers for sparse MoEs can be further grouped into two variants: Token Choice, which matches experts to each token, and Expert Choice, which matches tokens to each expert. We conduct head-to-head experiments with 6 different routers, including existing routers from prior work and new ones we introduce. We show that (i) many routers originally developed for language modeling can be adapted to perform strongly in vision tasks, (ii) in sparse MoE, Expert Choice routers generally outperform Token Choice routers, and (iii) soft MoEs generally outperform sparse MoEs with a fixed compute budget. These results provide new insights regarding the crucial role of routers in vision MoE models.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 33 of its 43 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

33 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023From Sparse to Soft Mixtures of Experts2 Aug
  2. 2022On the Adversarial Robustness of Mixture of Experts19 Oct
  3. 2022A Review of Sparse Expert Models in Deep Learning4 Sep
  4. 2022Sparse Mixture-of-Experts are Domain Generalizable Learners8 Jun
  5. 2022Residual Mixture of Experts20 Apr
  6. 2021Scaling Vision with Sparse Mixture of Experts10 Jun
  7. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations21 Aug
  8. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Vision20 Apr

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  11. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  12. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
13 more
  1. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  2. 2021Hash Layers For Large Sparse Modelscited by 237
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  4. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  6. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  7. 2017Smooth and Sparse Optimal Transportcited by 83
  8. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  9. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  10. 2022Sparsity-Constrained Optimal Transportcited by 32
  11. 2023Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspectivecited by 26
  12. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  13. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 13.

23 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  3. 2025Mixture of Experts in Large Language Modelscited by 15
  4. 2024Conditional computation in neural networks: principles and research trendscited by 10
  5. 2024Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoderscited by 9
  6. 2026MeMix: Writing Less, Remembering More for Streaming 3D Reconstructioncited by 8
  7. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 7
  8. 2025FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuningcited by 7
  9. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 5
  10. 2026Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance Learningcited by 4
  11. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selectioncited by 3
  12. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
11 more
  1. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Expertscited by 2
  2. 2026EMoE: Eigenbasis-Guided Routing for Mixture-of-Expertscited by 2
  3. 2026Visual Text Compression as Measure Transportcited by 1
  4. 2026When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routingcited by 1
  5. 2025PERTINENCE: Input-based Opportunistic Neural Network Dynamic Executioncited by 0
  6. 2026MP-ISMoE: Mixed-Precision Interactive Side Mixture-of-Experts for Efficient Transfer Learningcited by 0
  7. 2026Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Predictioncited by 0
  8. 2026ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modelingcited by 0
  9. 2026PromptPath: Prompt-Adaptive Computational Pathways for In-Context Learningcited by 0
  10. 2026Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projectionscited by 0
  11. 2026Multi-View Mixture-of-Experts with Vision-Language Reranking for Cross-View Object Geo-Localizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  3. 2026L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
  4. 2026Towards a Statistical Understanding of Mixture-of-Experts
  5. 2026Self-Routing: Parameter-Free Expert Routing from Hidden States
  6. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers
  7. 2026Path-Constrained Mixture-of-Experts
  8. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition