mlsanityData to 14 Oct 2026

Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?

arXiv 2606.29613 ↗First posted 28 Jun 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have recently been extended with role-based mechanisms for interpretability. This is typically done by assigning semantic roles to individual expert components, for example roles like synergy, redundancy, and uniqueness in multimodal settings. However, whether such structural role decomposition preserves explanation faithfulness of the overall architecture remains largely underexplored. We hypothesize that inter-expert representation overlap weakens effective role separation and degrades attribution-based faithfulness, even when semantic roles are explicitly defined. To address this limitation, we introduce representation-level decorrelation regularization to explicitly reduce inter-expert similarity in latent space. Using representation decorrelation objectives, we encourage clearer specialization among experts by minimizing representation overlap. Our experiments show that across multiple multimodal benchmarks, this separation consistently improves explanation faithfulness, as measured by comprehensiveness, sufficiency, and their Area Over the Perturbation Curve (AOPC) summaries, while preserving task performance. We further show that these improvements are not limited to role-based architectures such as Interpretable Multimodal Interaction-aware MoE (I2MoE). Similar trends are observed in a standard sparse MoE baseline, suggesting that representation-level separation may provide a more general mechanism for enhancing explanation faithfulness in MoE systems. Overall, our findings suggest that structural role decomposition alone may be insufficient to guarantee faithful explanations and that representation-level separation helps improve explanation faithfulness. To support reproducibility, the source code and supplementary material are publicly available at https://github.com/dut0817/FL-I2MoE_Decor.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 26 of its 33 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

26 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning23 May
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 23

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2020Quantifying Attention Flow in Transformerscited by 775
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  8. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  9. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  10. 2018On the Robustness of Interpretability Methodscited by 314
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  12. 2017Gated Multimodal Units for Information Fusioncited by 248
11 more
  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  4. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  5. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  6. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  7. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 30
  8. 2021A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decompositioncited by 26
  9. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  10. 2024Measuring Cross-Modal Interactions in Multimodal Modelscited by 16
  11. 2026Feature-level Interaction Explanations in Multimodal Transformerscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
  2. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  3. 2026Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
  4. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts
  5. 2025Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
  6. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
  7. 2026Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
  8. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing