mlsanityData to 14 Oct 2026

Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts

arXiv 2605.20610 ↗First posted 20 May 2026cs.CV cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 28 of its 63 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

28 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026MoE Lens -- An Expert Is All You Need6 Mar
  2. 2025Probing Semantic Routing in Large Mixture-of-Expert Models15 Feb
  3. 2024Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models22 Dec
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  6. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  7. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  8. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  9. 2022Residual Mixture of Expertscited by 39
  10. 2021Neural tuning and representational geometrycited by 31
  11. 2025Mixture of Experts Made Intrinsically Interpretablecited by 24
  12. 2024Investigating the Benefits of Projection Head for Representation Learningcited by 20
12 more
  1. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  2. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretabilitycited by 14
  3. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?cited by 11
  4. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 7
  5. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanismscited by 7
  6. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  7. 2025Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Expertscited by 4
  8. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  9. 2026HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detectioncited by 3
  10. 2019The nature of the animacy organization in human ventral temporal cortexcited by 1
  11. 2024Rethinking category-selectivity in human visual cortexcited by 1
  12. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  2. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts
  3. 2025Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing
  4. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  5. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
  6. 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
  7. 2025Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
  8. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing