Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation
On the map: Mixture-of-experts → Expert routing strategies
also in 1 more
Abstract (arXiv)
Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only 1.6× as a fraction of block output, while its residual-stream exposure spans 3.2×. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 37 of its 40 references that are in the map. First posted on arXiv 16 Aug 2026.
Builds on
37 of 40 references.
Closest ancestors · 6
Same direction, newest first
- 2026Stable Routing for Mixture-of-Experts in Class-Incremental Learning17 May
- 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models8 May
- 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs20 Apr
- 2025Load Balancing Mixture of Experts with Similarity Preserving Routers16 Jun
- 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 31
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2025Qwen3 Technical Reportcited by 8,820
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
19 more
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
- 2026Reinforcement Learning via Self-Distillationcited by 332
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2026Self-Distillation Enables Continual Learningcited by 247
- 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
- 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
- 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
- 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
- 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
- 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
- 2024Upcycling Large Language Models into Mixture of Expertscited by 49
- 2025Mixture of Experts Made Intrinsically Interpretablecited by 24
- 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
- 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
- 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
- 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learningcited by 4
- 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learningcited by 2
- 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learningcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
- 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
- 2022Revisiting Self-Distillation
- 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training
- 2026Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
- 2026Geometric Routing Enables Causal Expert Control in Mixture of Experts
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
- 2026Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing