# I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts

arXiv: https://arxiv.org/abs/2505.19190 · HTML: https://mlsanity.com/paper/2505.19190/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 25 May 2025.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 30 of 30)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](https://mlsanity.com/paper/2206.02770/) (2022)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts](https://mlsanity.com/paper/2407.21770/) (2024)
- [MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts](https://mlsanity.com/paper/2410.07348/) (2024)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations](https://mlsanity.com/paper/2203.02013/) (2022)
- [On the Benefits of Early Fusion in Multimodal Representation Learning](https://mlsanity.com/paper/2011.07191/) (2020)
- [A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition](https://mlsanity.com/paper/2105.04187/) (2021)
- [MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts](https://mlsanity.com/paper/2311.09580/) (2023)
- [Measuring Cross-Modal Interactions in Multimodal Models](https://mlsanity.com/paper/2412.15828/) (2024)
- [MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks](https://mlsanity.com/paper/2309.14118/) (2023)
- [Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions](https://mlsanity.com/paper/1510.00831/) (2015)
- [End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification](https://mlsanity.com/paper/1810.00523/) (2018)
- [Learning Unseen Modality Interaction](https://mlsanity.com/paper/2306.12795/) (2023)
- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [Intrinsic User-Centric Interpretability through Global Mixture of Experts](https://mlsanity.com/paper/2402.02933/) (2024)
- [SHARCS: Shared Concept Space for Explainable Multimodal Learning](https://mlsanity.com/paper/2307.00316/) (2023)

## Built on by (most cited first, 30 of 30)

- [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025)
- [MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation](https://mlsanity.com/paper/2508.15281/) (2025)
- [How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning](https://mlsanity.com/paper/2602.15580/) (2026)
- [Multimodal Model for Computational Pathology:Representation Learning and Image Compression](https://mlsanity.com/paper/2603.18660/) (2026)
- [MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free](https://mlsanity.com/paper/2601.02967/) (2026)
- [Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Experts](https://mlsanity.com/paper/2509.25678/) (2025)
- [Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Survey](https://mlsanity.com/paper/2510.00078/) (2025)
- [RecruitView: A Multimodal Dataset for Predicting Personality and Interview Performance for Human Resources Applications](https://mlsanity.com/paper/2512.00450/) (2025)
- [Explainable Multimodal Regression via Information Decomposition](https://mlsanity.com/paper/2512.22102/) (2025)
- [PathMoE: Interpretable Multimodal Interaction Experts for Pediatric Brain Tumor Classification](https://mlsanity.com/paper/2603.01547/) (2026)
- [Modeling Cross-vision Synergy for Unified Large Vision Model](https://mlsanity.com/paper/2603.03564/) (2026)
- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)
- [Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification](https://mlsanity.com/paper/2604.10695/) (2026)
- [CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering](https://mlsanity.com/paper/2604.16930/) (2026)
- [BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure](https://mlsanity.com/paper/2607.00573/) (2026)
- [Interpretable Alzheimer's Diagnosis via Multimodal Fusion of Regional Brain Experts](https://mlsanity.com/paper/2512.10966/) (2025)
- [M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model](https://mlsanity.com/paper/2604.08936/) (2026)
- [Are Independently Estimated View Uncertainties Comparable? Unified Routing for Trusted Multi-View Classification](https://mlsanity.com/paper/2604.09288/) (2026)
- [Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models](https://mlsanity.com/paper/2605.08145/) (2026)
- [PromptDx: Differentiable Prompt Tuning for Multimodal In-Context Alzheimer's Diagnosis](https://mlsanity.com/paper/2605.08585/) (2026)
- [ConceptM3oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology](https://mlsanity.com/paper/2605.24399/) (2026)
- [Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey](https://mlsanity.com/paper/2605.27431/) (2026)
- [SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning](https://mlsanity.com/paper/2606.09853/) (2026)
- [Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?](https://mlsanity.com/paper/2606.29613/) (2026)
- [iLENS: Interpretable LLM-Guided Mixture-of-Experts for Neuroimaging Survival Analysis](https://mlsanity.com/paper/2607.08778/) (2026)
- [Understanding Structured Health Data through Interaction-Aware Mixture-of-Experts](https://mlsanity.com/paper/2607.12255/) (2026)
- [CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition](https://mlsanity.com/paper/2608.07867/) (2026)
- [DoGMA: A Central-Dogma-Guided Foundation Model for Multi-Omics Alignment and Multi-Task Learning in Oncology](https://mlsanity.com/paper/2608.08148/) (2026)
- [M2PFN: End-to-End Disentangled Alignment for Generalizable Multimodal In-Context Learning in Alzheimer's Disease](https://mlsanity.com/paper/2609.28836/) (2026)
- [Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback](https://mlsanity.com/paper/2609.39456/) (2026)

## Abstract (arXiv)

Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.