Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such routing drift is often interpreted as routing failure, raising a fundamental question that remains unclear: does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair? We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as textittask loss recoverable under a specified routing intervention, with non-routing parameters fixed. These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose Selective Router Repair (SRR) as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that routing drift alone is insufficient evidence of routing failure: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 48 of its 55 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
48 of 55 references.
Closest ancestors · 4
Same direction, newest first
- 2026When Model Merging Breaks Routing: Training-Free Calibration for MoE2 Jun
- 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
- 2022Mixture-of-Experts with Expert Choice Routing18 Feb
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 44
- 2021Training Verifiers to Solve Math Word Problemscited by 9,916
- 2025Qwen3 Technical Reportcited by 8,510
- 2020Measuring Massive Multitask Language Understandingcited by 7,876
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
- 2024Mixtral of Expertscited by 1,811
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
- 2022Editing Models with Task Arithmeticcited by 1,101
32 more
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 961
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 808
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 687
- 2021Merging Models with Fisher-Weighted Averagingcited by 617
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 396
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
- 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 251
- 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 249
- 2023From Sparse to Soft Mixtures of Expertscited by 247
- 2024Evolutionary Optimization of Model Merging Recipescited by 204
- 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
- 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 145
- 2024Model merging with SVD to tie the Knotscited by 120
- 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 115
- 2024Representation Surgery for Multi-Task Model Mergingcited by 95
- 2023Deep Model Fusion: A Surveycited by 90
- 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 61
- 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 46
- 2025Model Merging in Pre-training of Large Language Modelscited by 37
- 2021Tricks for Training Sparse Translation Modelscited by 17
- 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 16
- 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
- 2025Model Merging Scaling Laws in Large Language Modelscited by 11
- 2026Behavior Knowledge Merge in Reinforced Agentic Modelscited by 11
- 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunkingcited by 8
- 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Modelscited by 7
- 2026Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Trainingcited by 5
- 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
- 2026E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoringcited by 4
- 2026Discovering Physical Directions in Weight Space: Composing Neural PDE Expertscited by 3
- 2026Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Mergingcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
- 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
- 2026Mixture of Experts for Low-Resource LLMs
- 2025Multilingual Routing in Mixture-of-Experts
- 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
- 2026Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
- 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
- 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing