How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
On the map: Mixture-of-experts → MoE theory and gating functions
Abstract (arXiv)
Sparse Mixture-of-Experts (MoE) models route each token to a few of many experts, inviting the hypothesis that experts form functional modules tied to capabilities or languages. We test this causally on Command A+, a frontier open-weights MoE (218B total / 25B active; 128 experts, 8 active, +1 shared). We build a routing-mass atlas, pre-register six family-to-axis hypotheses before any intervention, and ablate each family at inference time against a size-matched random-expert null, measuring whether it selectively breaks its own axis (worst off-target effect at most one third of on-target). Crucially, we test the same families under four metrics and a held-out, independent-corpus run with bootstrap confidence intervals. Our finding is cautionary: robust functional modularity is rare and measurement-dependent. Of six pre-registered families, only one, the Arabic-language family, is a clean selective module that survives an independent corpus and a conservative statistical bar (1/6; a more permissive pre-registered point rule admits 3/6, but that count is threshold-sensitive). Every other family has a real causal effect yet fails selectivity, and its apparent modularity flips with the measurement: with the corpus, the metric, and the statistical bar. A positive control on Qwen3-30B-A3B recovers its published disjoint structure, confirming the method detects modularity when present. The verdict reproduces on the un-quantized BF16 model, ruling out a 4-bit quantization artifact. We conclude that ablation-based modularity verdicts are not safe unless the corpus, metric, and statistical bar are controlled. We release the atlas and ablation data.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 9 of its 9 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
9 of 9 references.
References · 9
- 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
- 2025Multilingual Routing in Mixture-of-Expertscited by 29
- 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
- 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 23
- 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
- 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
- 2026Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Modelscited by 4
- 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
- 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Robustness of Mixtures of Experts to Feature Noise
- 2026Geometric Routing Enables Causal Expert Control in Mixture of Experts
- 2026EMO: Pretraining Mixture of Experts for Emergent Modularity
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
- 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
- 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
- 2025Towards a Comprehensive Scaling Law of Mixture-of-Experts
- 2025Sparsity and Superposition in Mixture of Experts