# MultiSHAP: A Shapley-Based Framework for Explaining Cross-Modal Interactions in Multimodal AI Models

arXiv: https://arxiv.org/abs/2508.00576 · HTML: https://mlsanity.com/paper/2508.00576/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 1 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Faith-Shap: The Faithful Shapley Interaction Index](https://mlsanity.com/paper/2203.00870/) (2022)
- [MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks](https://mlsanity.com/paper/2212.08158/) (2022)
- [TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation](https://mlsanity.com/paper/2407.10114/) (2024)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach](https://mlsanity.com/paper/2505.01997/) (2025)
- [Measuring Cross-Modal Interactions in Multimodal Models](https://mlsanity.com/paper/2412.15828/) (2024)
- [Attention, Please! PixelSHAP Reveals What Vision-Language Models Actually Focus On](https://mlsanity.com/paper/2503.06670/) (2025)
- [Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes](https://mlsanity.com/paper/2503.12286/) (2025)
- [GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text](https://mlsanity.com/paper/2312.15320/) (2023)
- [Fair CCA for Fair Representation Learning: An ADNI Study](https://mlsanity.com/paper/2507.09382/) (2025)

## Built on by (most cited first, 7 of 7)

- [Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition](https://mlsanity.com/paper/2606.00959/) (2026)
- [llmSHAP: A Principled Approach to LLM Explainability](https://mlsanity.com/paper/2511.01311/) (2025)
- [Quantifying Modality Contributions via Disentangling Multimodal Representations](https://mlsanity.com/paper/2511.19470/) (2025)
- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)
- [Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability](https://mlsanity.com/paper/2605.22168/) (2026)
- [LEMON: Local Explanations via Modality-aware OptimizatioN](https://mlsanity.com/paper/2602.02786/) (2026)
- [GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods](https://mlsanity.com/paper/2606.14740/) (2026)

## Abstract (arXiv)

Multimodal AI models have achieved impressive performance in tasks that require integrating information from multiple modalities, such as vision and language. However, their "black-box" nature poses a major barrier to deployment in high-stakes applications where interpretability and trustworthiness are essential. How to explain cross-modal interactions in multimodal AI models remains a major challenge. While existing model explanation methods, such as attention map and Grad-CAM, offer coarse insights into cross-modal relationships, they cannot precisely quantify the synergistic effects between modalities, and are limited to open-source models with accessible internal weights. Here we introduce MultiSHAP, a model-agnostic interpretability framework that leverages the Shapley Interaction Index to attribute multimodal predictions to pairwise interactions between fine-grained visual and textual elements (such as image patches and text tokens), while being applicable to both open- and closed-source models. Our approach provides: (1) instance-level explanations that reveal synergistic and suppressive cross-modal effects for individual samples - "why the model makes a specific prediction on this input", and (2) dataset-level explanation that uncovers generalizable interaction patterns across samples - "how the model integrates information across modalities". Experiments on public multimodal benchmarks confirm that MultiSHAP faithfully captures cross-modal reasoning mechanisms, while real-world case studies demonstrate its practical utility. Our framework is extensible beyond two modalities, offering a general solution for interpreting complex multimodal AI models.