mlsanityData to 13 Oct 2026

Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

arXiv 2609.32821 ↗First posted 26 Sep 2026cs.AI cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such routing drift is often interpreted as routing failure, raising a fundamental question that remains unclear: does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair? We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as textittask loss recoverable under a specified routing intervention, with non-routing parameters fixed. These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose Selective Router Repair (SRR) as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that routing drift alone is insufficient evidence of routing failure: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 48 of its 55 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

48 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026When Model Merging Breaks Routing: Training-Free Calibration for MoE2 Jun
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 44

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  2. 2025Qwen3 Technical Reportcited by 8,510
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,163
  9. 2024Mixtral of Expertscited by 1,811
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
  12. 2022Editing Models with Task Arithmeticcited by 1,101
32 more
  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 961
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 853
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 808
  4. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 687
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 617
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 396
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 298
  8. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 251
  9. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 249
  10. 2023From Sparse to Soft Mixtures of Expertscited by 247
  11. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  12. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  13. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 145
  14. 2024Model merging with SVD to tie the Knotscited by 120
  15. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 115
  16. 2024Representation Surgery for Multi-Task Model Mergingcited by 95
  17. 2023Deep Model Fusion: A Surveycited by 90
  18. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 61
  19. 2026Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillationcited by 46
  20. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  21. 2021Tricks for Training Sparse Translation Modelscited by 17
  22. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 16
  23. 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
  24. 2025Model Merging Scaling Laws in Large Language Modelscited by 11
  25. 2026Behavior Knowledge Merge in Reinforced Agentic Modelscited by 11
  26. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunkingcited by 8
  27. 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Modelscited by 7
  28. 2026Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Trainingcited by 5
  29. 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
  30. 2026E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoringcited by 4
  31. 2026Discovering Physical Directions in Weight Space: Composing Neural PDE Expertscited by 3
  32. 2026Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Mergingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
  2. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  3. 2026Mixture of Experts for Low-Resource LLMs
  4. 2025Multilingual Routing in Mixture-of-Experts
  5. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  6. 2026Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
  7. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
  8. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing