mlsanityData to 14 Oct 2026

Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution

arXiv 2312.17174 ↗First posted 28 Dec 2023cs.CV cs.AI cs.LGcited by 34

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language pretrained models have seen remarkable success, but their application to safety-critical settings is limited by their lack of interpretability. To improve the interpretability of vision-language models such as CLIP, we propose a multi-modal information bottleneck (M2IB) approach that learns latent representations that compress irrelevant information while preserving relevant visual and textual features. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models, increasing attribution accuracy and improving the interpretability of such models when applied to safety-critical domains such as healthcare. Crucially, unlike commonly used unimodal attribution methods, M2IB does not require ground truth labels, making it possible to audit representations of vision-language pretrained models when multiple modalities but no ground-truth data is available. Using CLIP as an example, we demonstrate the effectiveness of M2IB attribution and show that it outperforms gradient-based, perturbation-based, and attention-based attribution methods both qualitatively and quantitatively.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 33 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

23 of 33 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2018Sanity Checks for Saliency Mapscited by 1,156
  8. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  9. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  10. 2016Deep Variational Information Bottleneckcited by 930
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  12. 2015Deep Learning and the Information Bottleneck Principlecited by 802
11 more
  1. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  2. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  3. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  4. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  5. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  6. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  7. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  8. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 197
  9. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  10. 2022A Consistent and Efficient Evaluation Strategy for Attribution Methodscited by 102
  11. 2022Vision-and-Language Pretrained Models: A Surveycited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 11.

34 papers.

Sort:

By area:

  1. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 39
  4. 2024An Information Criterion for Controlled Disentanglement of Multimodal Datacited by 14
  5. 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 8
  6. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  7. 2024Extract Free Dense Misalignment from CLIPcited by 5
  8. 2025Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detectioncited by 5
  9. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Modelscited by 4
  10. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 4
  11. 2024Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributionscited by 3
  12. 2025Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretabilitycited by 3
22 more
  1. 2025Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approachcited by 3
  2. 2026CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoningcited by 3
  3. 2024Disentangling Textual and Acoustic Features of Neural Speech Representationscited by 2
  4. 2025Enhancing Interpretability for Vision Models via Shapley Value Optimizationcited by 2
  5. 2025Explainable Multimodal Regression via Information Decompositioncited by 2
  6. 2025Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questionscited by 1
  7. 2026MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Modelscited by 1
  8. 2024Noise-Free Explanation for Driving Action Predictioncited by 0
  9. 2024Aligning Characteristic Descriptors with Images for Human-Expert-like Explainabilitycited by 0
  10. 2024If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptionscited by 0
  11. 2025ABE: A Unified Framework for Robust and Faithful Attribution-Based Explainabilitycited by 0
  12. 2025CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalitiescited by 0
  13. 2025Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Modelcited by 0
  14. 2025Mask What Matters: Controllable Text-Guided Masking for Self-Supervised Medical Image Analysiscited by 0
  15. 2026Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Mediacited by 0
  16. 2026RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understandingcited by 0
  17. 2026Understanding How MLLMs Describe Artworks Using Token Activation Mapscited by 0
  18. 2026Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentationcited by 0
  19. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0
  20. 2026SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosiscited by 0
  21. 2026Beyond Mean Foils: Auditing Worst-Foil Specificity in Frozen CLIP Region Explanationscited by 0
  22. 2026Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning
  2. 2025Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models
  3. 2026Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models
  4. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  5. 2025From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance
  6. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  7. 2024Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?