# Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis

arXiv: https://arxiv.org/abs/2303.13391 · HTML: https://mlsanity.com/paper/2303.13391/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Potential, Challenges and Future Directions for Deep Learning in Prognostics and Health Management Applications](https://mlsanity.com/paper/2005.02144/) (2020)
- [Explaining Chest X-ray Pathologies in Natural Language](https://mlsanity.com/paper/2207.04343/) (2022)
- [Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Training](https://mlsanity.com/paper/2205.07139/) (2022)
- [CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models](https://mlsanity.com/paper/2302.12343/) (2023)

## Built on by (most cited first, 23 of 23)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPT](https://mlsanity.com/paper/2304.04920/) (2023)
- [MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise](https://mlsanity.com/paper/2405.11793/) (2024)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis](https://mlsanity.com/paper/2501.03565/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answering](https://mlsanity.com/paper/2407.21368/) (2024)
- [MAGDA: Multi-agent guideline-driven diagnostic assistance](https://mlsanity.com/paper/2409.06351/) (2024)
- [Few-shot medical image classification with simple shape and texture text descriptors using vision-language models](https://mlsanity.com/paper/2308.04005/) (2023)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Position-Guided Prompt Learning for Anomaly Detection in Chest X-Rays](https://mlsanity.com/paper/2405.11976/) (2024)
- [A Progressive Single-Modality to Multi-Modality Classification Framework for Alzheimer's Disease Sub-type Diagnosis](https://mlsanity.com/paper/2407.18466/) (2024)
- [Language Model as Visual Explainer](https://mlsanity.com/paper/2412.07802/) (2024)
- [Focus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classification](https://mlsanity.com/paper/2507.10869/) (2025)
- [Interpretable Multimodal Zero-Shot ECG Diagnosis via Structured Clinical Knowledge Alignment](https://mlsanity.com/paper/2510.21551/) (2025)
- [How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?](https://mlsanity.com/paper/2409.00543/) (2024)
- [Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment](https://mlsanity.com/paper/2510.11112/) (2025)
- [Hierarchically Robust Zero-shot Vision-language Models](https://mlsanity.com/paper/2604.18867/) (2026)
- [Leveraging Vision-Language Foundation Models for Fine-Grained Downstream Tasks](https://mlsanity.com/paper/2307.06795/) (2023)
- [ECAMP: Entity-centered Context-aware Medical Vision Language Pre-training](https://mlsanity.com/paper/2312.13316/) (2023)
- [BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models](https://mlsanity.com/paper/2510.15866/) (2025)

## Abstract (arXiv)

Automated diagnosis prediction from medical images is a valuable resource to support clinical decision-making. However, such systems usually need to be trained on large amounts of annotated data, which often is scarce in the medical domain. Zero-shot methods address this challenge by allowing a flexible adaption to new settings with different clinical findings without relying on labeled data. Further, to integrate automated diagnosis in the clinical workflow, methods should be transparent and explainable, increasing medical professionals' trust and facilitating correctness verification. In this work, we introduce Xplainer, a novel framework for explainable zero-shot diagnosis in the clinical setting. Xplainer adapts the classification-by-description approach of contrastive vision-language models to the multi-label medical diagnosis task. Specifically, instead of directly predicting a diagnosis, we prompt the model to classify the existence of descriptive observations, which a radiologist would look for on an X-Ray scan, and use the descriptor probabilities to estimate the likelihood of a diagnosis. Our model is explainable by design, as the final diagnosis prediction is directly based on the prediction of the underlying descriptors. We evaluate Xplainer on two chest X-ray datasets, CheXpert and ChestX-ray14, and demonstrate its effectiveness in improving the performance and explainability of zero-shot diagnosis. Our results suggest that Xplainer provides a more detailed understanding of the decision-making process and can be a valuable tool for clinical diagnosis.