# Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution

arXiv: https://arxiv.org/abs/2312.17174 · HTML: https://mlsanity.com/paper/2312.17174/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [Restricting the Flow: Information Bottlenecks for Attribution](https://mlsanity.com/paper/2001.00396/) (2020)
- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [Vision-and-Language Pretrained Models: A Survey](https://mlsanity.com/paper/2204.07356/) (2022)

## Built on by (most cited first, 30 of 34)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation](https://mlsanity.com/paper/2409.19483/) (2024)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP](https://mlsanity.com/paper/2502.18816/) (2025)
- [Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions](https://mlsanity.com/paper/2508.05430/) (2025)
- [Extract Free Dense Misalignment from CLIP](https://mlsanity.com/paper/2412.18404/) (2024)
- [Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2505.15217/) (2025)
- [Improving Concept Alignment in Vision-Language Concept Bottleneck Models](https://mlsanity.com/paper/2405.01825/) (2024)
- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability](https://mlsanity.com/paper/2502.14889/) (2025)
- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)
- [CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoning](https://mlsanity.com/paper/2605.29602/) (2026)
- [Disentangling Textual and Acoustic Features of Neural Speech Representations](https://mlsanity.com/paper/2410.03037/) (2024)
- [Enhancing Interpretability for Vision Models via Shapley Value Optimization](https://mlsanity.com/paper/2512.14354/) (2025)
- [Explainable Multimodal Regression via Information Decomposition](https://mlsanity.com/paper/2512.22102/) (2025)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Models](https://mlsanity.com/paper/2601.11666/) (2026)
- [Noise-Free Explanation for Driving Action Prediction](https://mlsanity.com/paper/2407.06339/) (2024)
- [Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability](https://mlsanity.com/paper/2411.04008/) (2024)
- [If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptions](https://mlsanity.com/paper/2411.15611/) (2024)
- [ABE: A Unified Framework for Robust and Faithful Attribution-Based Explainability](https://mlsanity.com/paper/2505.06258/) (2025)
- [CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities](https://mlsanity.com/paper/2506.08690/) (2025)
- [Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Model](https://mlsanity.com/paper/2509.14664/) (2025)
- [Mask What Matters: Controllable Text-Guided Masking for Self-Supervised Medical Image Analysis](https://mlsanity.com/paper/2509.23054/) (2025)
- [Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media](https://mlsanity.com/paper/2603.18611/) (2026)
- [RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding](https://mlsanity.com/paper/2605.15561/) (2026)
- [Understanding How MLLMs Describe Artworks Using Token Activation Maps](https://mlsanity.com/paper/2606.27947/) (2026)
- [Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation](https://mlsanity.com/paper/2607.17693/) (2026)

## Abstract (arXiv)

Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multi-modal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.