mlsanityData to 14 Oct 2026

I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts

arXiv 2505.19190 ↗First posted 25 May 2025cs.LG cs.AI cs.CVcited by 30

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 56 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

30 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts10 Oct
  2. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts31 Jul
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun

Other references · 27

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  4. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  5. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  6. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  7. 2017Gated Multimodal Units for Information Fusioncited by 248
  8. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
  9. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  10. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  11. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  12. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
15 more
  1. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 56
  2. 2024What to align in multimodal contrastive learning?cited by 47
  3. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  4. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  5. 2020On the Benefits of Early Fusion in Multimodal Representation Learningcited by 27
  6. 2021A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decompositioncited by 26
  7. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  8. 2024Measuring Cross-Modal Interactions in Multimodal Modelscited by 16
  9. 2023MultiModN- Multimodal, Multi-Task, Interpretable Modular Networkscited by 14
  10. 2015Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functionscited by 12
  11. 2018End-To-End Alzheimer's Disease Diagnosis and Biomarker Identificationcited by 12
  12. 2023Learning Unseen Modality Interactioncited by 12
  13. 2023Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeatcited by 10
  14. 2024Intrinsic User-Centric Interpretability through Global Mixture of Expertscited by 9
  15. 2023SHARCS: Shared Concept Space for Explainable Multimodal Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 23.

30 papers.

Sort:

By area:

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
  3. 2026How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoningcited by 8
  4. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  5. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 5
  6. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  7. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  8. 2025RecruitView: A Multimodal Dataset for Predicting Personality and Interview Performance for Human Resources Applicationscited by 2
  9. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  10. 2026PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classificationcited by 1
  11. 2026Modeling Cross-vision Synergy for Unified Large Vision Modelcited by 1
  12. 2026Feature-level Interaction Explanations in Multimodal Transformerscited by 1
18 more
  1. 2026Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purificationcited by 1
  2. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  3. 2026BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructurecited by 1
  4. 2025Interpretable Alzheimer's Diagnosis via Multimodal Fusion of Regional Brain Expertscited by 0
  5. 2026M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Modelcited by 0
  6. 2026Are Independently Estimated View Uncertainties Comparable? Unified Routing for Trusted Multi-View Classificationcited by 0
  7. 2026Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Modelscited by 0
  8. 2026PromptDx: Differentiable Prompt Tuning for Multimodal In-Context Alzheimer's Diagnosiscited by 0
  9. 2026ConceptM3oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathologycited by 0
  10. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  11. 2026SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learningcited by 0
  12. 2026Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?cited by 0
  13. 2026iLENS: Interpretable LLM-Guided Mixture-of-Experts for Neuroimaging Survival Analysiscited by 0
  14. 2026Understanding Structured Health Data through Interaction-Aware Mixture-of-Expertscited by 0
  15. 2026CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognitioncited by 0
  16. 2026DoGMA: A Central-Dogma-Guided Foundation Model for Multi-Omics Alignment and Multi-Task Learning in Oncologycited by 0
  17. 2026M2PFN: End-to-End Disentangled Alignment for Generalizable Multimodal In-Context Learning in Alzheimer's Diseasecited by 0
  18. 2026Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Balanced Multimodal Learning via Mutual Information
  2. 2025Unimodal-driven Distillation in Multimodal Emotion Recognition with Dynamic Fusion
  3. 2025MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment
  4. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  5. 2026C2MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning
  6. 2025BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
  7. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  8. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding