mlsanityData to 7 Oct 2026

Envisioning MedCLIP: A Deep Dive into Explainability for Medical Vision-Language Models

arXiv 2403.18996 ↗First posted 27 Mar 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Explaining Deep Learning models is becoming increasingly important in the face of daily emerging multimodal models, particularly in safety-critical domains like medical imaging. However, the lack of detailed investigations into the performance of explainability methods on these models is widening the gap between their development and safe deployment. In this work, we analyze the performance of various explainable AI methods on a vision-language model, MedCLIP, to demystify its inner workings. We also provide a simple methodology to overcome the shortcomings of these methods. Our work offers a different new perspective on the explainability of a recent well-known VLM in the medical domain and our assessment method is generalizable to other current and possible future VLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 29 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

16 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generation23 Nov

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  11. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
3 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  2. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  3. 2021Explainable artificial intelligence (XAI) in deep learning-based medical image analysiscited by 147

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025XDR-LVLM: An Explainable Vision-Language Large Model for Diabetic Retinopathy Diagnosiscited by 1
  2. 2024Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025An Explainable Biomedical Foundation Model via Large-Scale Concept-Enhanced Vision-Language Pre-training
  2. 2025MedSAE: Dissecting MedCLIP Representations with Sparse Autoencoders
  3. 2025Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive Survey
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
  6. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  7. 2026MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning
  8. 2024XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization