# Explaining Chest X-ray Pathologies in Natural Language

arXiv: https://arxiv.org/abs/2207.04343 · HTML: https://mlsanity.com/paper/2207.04343/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 9 Jul 2022.

## Builds on (most cited first, 26 of 26)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [TorchXRayVision: A library of chest X-ray datasets and models](https://mlsanity.com/paper/2111.00595/) (2021)
- [e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks](https://mlsanity.com/paper/2105.03761/) (2021)
- [XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations](https://mlsanity.com/paper/2103.10663/) (2021)
- [Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs](https://mlsanity.com/paper/2010.07526/) (2020)
- [Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Images](https://mlsanity.com/paper/2101.07563/) (2021)
- [RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting](https://mlsanity.com/paper/2107.02104/) (2021)
- [Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations](https://mlsanity.com/paper/2106.13876/) (2021)
- [Generation of Multimodal Justification Using Visual Word Constraint Model for Explainable Computer-Aided Diagnosis](https://mlsanity.com/paper/1906.03922/) (2019)
- [Interpretable Deep Models for Cardiac Resynchronisation Therapy Response Prediction](https://mlsanity.com/paper/2006.13811/) (2020)
- [Saliency Methods for Explaining Adversarial Attacks](https://mlsanity.com/paper/1908.08413/) (2019)
- [Producing radiologist-quality reports for interpretable artificial intelligence](https://mlsanity.com/paper/1806.00340/) (2018)

## Built on by (most cited first, 27 of 27)

- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- [Rigorously Assessing Natural Language Explanations of Neurons](https://mlsanity.com/paper/2309.10312/) (2023)
- [The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models](https://mlsanity.com/paper/2404.03189/) (2024)
- [Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis](https://mlsanity.com/paper/2303.13391/) (2023)
- [The Impact of Imperfect XAI on Human-AI Decision-Making](https://mlsanity.com/paper/2307.13566/) (2023)
- [Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06728/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime](https://mlsanity.com/paper/2303.17644/) (2023)
- [LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologies](https://mlsanity.com/paper/2410.04749/) (2024)
- [CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes](https://mlsanity.com/paper/2408.11965/) (2024)
- [Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approach](https://mlsanity.com/paper/2407.20899/) (2024)
- [Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting](https://mlsanity.com/paper/2410.12284/) (2024)
- [A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation](https://mlsanity.com/paper/2604.00493/) (2026)
- [KNOW How to Make Up Your Mind! Adversarially Detecting and Alleviating Inconsistencies in Natural Language Explanations](https://mlsanity.com/paper/2306.02980/) (2023)
- [Learning To Guide Human Decision Makers With Vision-Language Models](https://mlsanity.com/paper/2403.16501/) (2024)
- [TiBiX: Leveraging Temporal Information for Bidirectional X-ray and Report Generation](https://mlsanity.com/paper/2403.13343/) (2024)
- [SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations](https://mlsanity.com/paper/2305.13235/) (2023)
- [Using Natural Language Explanations to Improve Robustness of In-context Learning](https://mlsanity.com/paper/2311.07556/) (2023)
- [Towards Efficient and Robust VQA-NLE Data Generation with Large Vision-Language Models](https://mlsanity.com/paper/2409.14785/) (2024)
- [EchoNarrator: Generating natural text explanations for ejection fraction predictions](https://mlsanity.com/paper/2410.23744/) (2024)
- [Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2507.07148/) (2025)
- [RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance](https://mlsanity.com/paper/2311.18681/) (2023)
- [Explaining Chest X-ray Pathology Models using Textual Concepts](https://mlsanity.com/paper/2407.00557/) (2024)
- [MedVision: Benchmarking Quantitative Medical Image Analysis](https://mlsanity.com/paper/2511.18676/) (2025)
- [WeNLEX: Weakly Supervised Natural Language Explanations for Multilabel Chest X-ray Classification](https://mlsanity.com/paper/2603.18752/) (2026)
- [Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language](https://mlsanity.com/paper/2607.03882/) (2026)

## Abstract (arXiv)

Most deep learning algorithms lack explanations for their predictions, which limits their deployment in clinical practice. Approaches to improve explainability, especially in medical imaging, have often been shown to convey limited information, be overly reassuring, or lack robustness. In this work, we introduce the task of generating natural language explanations (NLEs) to justify predictions made on medical images. NLEs are human-friendly and comprehensive, and enable the training of intrinsically explainable models. To this goal, we introduce MIMIC-NLE, the first, large-scale, medical imaging dataset with NLEs. It contains over 38,000 NLEs, which explain the presence of various thoracic pathologies and chest X-ray findings. We propose a general approach to solve the task and evaluate several architectures on this dataset, including via clinician assessment.