The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 21 of its 28 references that are in the map. First posted on arXiv 6 Jan 2026.
Builds on
21 of 28 references.
Closest ancestors · 5
Same direction, newest first
- 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs27 Aug
- 2025Probing Semantic Routing in Large Mixture-of-Expert Models15 Feb
- 2024Towards an empirical understanding of MoE design choices20 Feb
- 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 16
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
- 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
- 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
- 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
- 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
- 2025Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Financecited by 35
- 2025Mixture of Experts Made Intrinsically Interpretablecited by 24
4 more
- 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
- 2025FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMscited by 14
- 2024Intrinsic User-Centric Interpretability through Global Mixture of Expertscited by 9
- 2025FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Informationcited by 3
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 13
- 2026Geometric Routing Enables Causal Expert Control in Mixture of Expertscited by 2
- 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Qualitycited by 1
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
- 2026SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specializationcited by 0
- 2026Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradientscited by 0
- 2026Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients29 Sep 2026
- 2026SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization4 Aug 2026
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing30 Jul 2026
- 2026Geometric Routing Enables Causal Expert Control in Mixture of Experts15 Apr 2026
- 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality15 Apr 2026
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise10 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
- 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
- 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
- 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
- 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
- 2026Path-Constrained Mixture-of-Experts
- 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
- 2026Towards a Statistical Understanding of Mixture-of-Experts