# DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

arXiv: https://arxiv.org/abs/2203.02013 · HTML: https://mlsanity.com/paper/2203.02013/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 68 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 30 of 46)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Analyzing the Behavior of Visual Question Answering Models](https://mlsanity.com/paper/1606.07356/) (2016)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [Leveraging Sparse Linear Layers for Debuggable Deep Networks](https://mlsanity.com/paper/2105.04857/) (2021)
- [Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers](https://mlsanity.com/paper/2109.04448/) (2021)

## Built on by (most cited first, 30 of 34)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models](https://mlsanity.com/paper/2405.20985/) (2024)
- [Cross-modal Information Flow in Multimodal Large Language Models](https://mlsanity.com/paper/2411.18620/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks](https://mlsanity.com/paper/2212.08158/) (2022)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts](https://mlsanity.com/paper/2505.19190/) (2025)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [Multimodal Fusion Interactions: A Study of Human and Automatic Quantification](https://mlsanity.com/paper/2306.04125/) (2023)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [Probabilistic Stability Guarantees for Feature Attributions](https://mlsanity.com/paper/2504.13787/) (2025)
- [Which LIME should I trust? Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2503.24365/) (2025)
- [POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generation](https://mlsanity.com/paper/2504.13392/) (2025)
- [Rethinking Explainability in the Era of Multimodal AI](https://mlsanity.com/paper/2506.13060/) (2025)
- [Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions](https://mlsanity.com/paper/2508.05430/) (2025)
- [An Empirical Study on How Video-LLMs Answer Video Questions](https://mlsanity.com/paper/2508.15360/) (2025)
- [Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention](https://mlsanity.com/paper/2601.08151/) (2026)
- [Generic Attention-model Explainability by Weighted Relevance Accumulation](https://mlsanity.com/paper/2308.10240/) (2023)
- [A Survey on Interpretable Cross-modal Reasoning](https://mlsanity.com/paper/2309.01955/) (2023)
- [Knowledge Transfer from Interaction Learning](https://mlsanity.com/paper/2509.18733/) (2025)
- [Investigating Modality Contribution in Audio LLMs for Music](https://mlsanity.com/paper/2509.20641/) (2025)
- [Towards Fine-Grained Interpretability: Counterfactual Explanations for Misclassification with Saliency Partition](https://mlsanity.com/paper/2511.07974/) (2025)
- [Explainable Multimodal Regression via Information Decomposition](https://mlsanity.com/paper/2512.22102/) (2025)
- [Quantifying Modality Contributions via Disentangling Multimodal Representations](https://mlsanity.com/paper/2511.19470/) (2025)
- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)

## Abstract (arXiv)

The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making. As a result, the research fields of interpretable and explainable AI have gained traction within AI communities as well as interdisciplinary scientists seeking to apply AI in their subject areas. In this paper, we focus on advancing the state-of-the-art in interpreting multimodal models - a class of machine learning methods that tackle core challenges in representing and capturing interactions between heterogeneous data sources such as images, text, audio, and time-series data. Multimodal models have proliferated numerous real-world applications across healthcare, robotics, multimedia, affective computing, and human-computer interaction. By performing model disentanglement into unimodal contributions (UC) and multimodal interactions (MI), our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models while maintaining generality across arbitrary modalities, model architectures, and tasks. Through a comprehensive suite of experiments on both synthetic and real-world multimodal tasks, we show that DIME generates accurate disentangled explanations, helps users of multimodal models gain a deeper understanding of model behavior, and presents a step towards debugging and improving these models for real-world deployment. Code for our experiments can be found at https://github.com/lvyiwei1/DIME.