# Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts

arXiv: https://arxiv.org/abs/2605.20610 · HTML: https://mlsanity.com/paper/2605.20610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [ST-MoE: Designing Stable and Transferable Sparse Expert Models](https://mlsanity.com/paper/2202.08906/) (2022)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Scaling Vision-Language Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2303.07226/) (2023)
- [A Closer Look into Mixture-of-Experts in Large Language Models](https://mlsanity.com/paper/2406.18219/) (2024)
- [Residual Mixture of Experts](https://mlsanity.com/paper/2204.09636/) (2022)
- [Neural tuning and representational geometry](https://mlsanity.com/paper/2104.09743/) (2021)
- [Mixture of Experts Made Intrinsically Interpretable](https://mlsanity.com/paper/2503.07639/) (2025)
- [Investigating the Benefits of Projection Head for Representation Learning](https://mlsanity.com/paper/2403.11391/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability](https://mlsanity.com/paper/2204.10598/) (2022)
- [Probing Semantic Routing in Large Mixture-of-Expert Models](https://mlsanity.com/paper/2502.10928/) (2025)
- [Mixture of Experts in Image Classification: What's the Sweet Spot?](https://mlsanity.com/paper/2411.18322/) (2024)
- [Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models](https://mlsanity.com/paper/2412.16971/) (2024)
- [Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model](https://mlsanity.com/paper/2507.01351/) (2025)
- [Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms](https://mlsanity.com/paper/2509.23933/) (2025)
- [MoE Lens -- An Expert Is All You Need](https://mlsanity.com/paper/2603.05806/) (2026)
- [MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models](https://mlsanity.com/paper/2508.09779/) (2025)
- [Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts](https://mlsanity.com/paper/2509.10025/) (2025)
- [Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities](https://mlsanity.com/paper/2503.22517/) (2025)
- [HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection](https://mlsanity.com/paper/2604.04908/) (2026)
- [The nature of the animacy organization in human ventral temporal cortex](https://mlsanity.com/paper/1904.02866/) (2019)
- [Rethinking category-selectivity in human visual cortex](https://mlsanity.com/paper/2411.08251/) (2024)
- [Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation](https://mlsanity.com/paper/2604.13761/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure.