# VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels

arXiv: https://arxiv.org/abs/2102.11467 · HTML: https://mlsanity.com/paper/2102.11467/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 23 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [NegBio: a high-performance tool for negation and uncertainty detection in radiology reports](https://mlsanity.com/paper/1712.05898/) (2017)
- [Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels](https://mlsanity.com/paper/2005.12734/) (2020)
- [Exploring large scale public medical image datasets](https://mlsanity.com/paper/1907.12720/) (2019)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)
- [Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Dataset](https://mlsanity.com/paper/2010.08006/) (2020)
- [Cross-Modal Data Programming Enables Rapid Medical Machine Learning](https://mlsanity.com/paper/1903.11101/) (2019)
- [CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output](https://mlsanity.com/paper/2006.15229/) (2020)
- [CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting](https://mlsanity.com/paper/2002.11379/) (2020)
- [Caveats in Generating Medical Imaging Labels from Radiology Reports](https://mlsanity.com/paper/1905.02283/) (2019)
- [Labelling imaging datasets on the basis of neuroradiology reports: a validation study](https://mlsanity.com/paper/2007.04226/) (2020)
- [CheXphotogenic: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays](https://mlsanity.com/paper/2011.06129/) (2020)

## Built on by (most cited first, 27 of 27)

- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays](https://mlsanity.com/paper/2206.12704/) (2022)
- [BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis](https://mlsanity.com/paper/2108.04938/) (2021)
- [A Systematic Review of Deep Learning-based Research on Radiology Report Generation](https://mlsanity.com/paper/2311.14199/) (2023)
- [RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction](https://mlsanity.com/paper/2308.05046/) (2023)
- [Structured dataset documentation: a datasheet for CheXpert](https://mlsanity.com/paper/2105.03020/) (2021)
- [Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese](https://mlsanity.com/paper/2209.04794/) (2022)
- [Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation](https://mlsanity.com/paper/2407.01948/) (2024)
- [Leveraging an Alignment Set in Tackling Instance-Dependent Label Noise](https://mlsanity.com/paper/2307.04868/) (2023)
- [Deep learning for understanding multilabel imbalanced Chest X-ray datasets](https://mlsanity.com/paper/2207.14408/) (2022)
- [Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodology](https://mlsanity.com/paper/2408.04121/) (2024)
- [The Disparate Benefits of Deep Ensembles](https://mlsanity.com/paper/2410.13831/) (2024)
- [Informed Deep Abstaining Classifier: Investigating noise-robust training for diagnostic decision support systems](https://mlsanity.com/paper/2410.21014/) (2024)
- [Exploring the interplay of label bias with subgroup size and separability: A case study in mammographic density classification](https://mlsanity.com/paper/2507.17996/) (2025)
- [Limitations of Public Chest Radiography Datasets for Artificial Intelligence: Label Quality, Domain Shift, Bias and Evaluation Challenges](https://mlsanity.com/paper/2509.15107/) (2025)
- [DAug: Diffusion-based Channel Augmentation for Radiology Image Retrieval and Classification](https://mlsanity.com/paper/2412.04828/) (2024)
- [VLM-KG: Multimodal Radiology Knowledge Graph Generation](https://mlsanity.com/paper/2505.17042/) (2025)
- [Privacy-Preserving Chest X-ray Classification in Latent Space with Homomorphically Encrypted Neural Inference](https://mlsanity.com/paper/2506.15258/) (2025)
- [Radiologist-Guided Causal Concept Bottleneck Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2605.07785/) (2026)
- [Direct Preference Optimization for Adaptive Concept-based Explanations](https://mlsanity.com/paper/2505.15626/) (2025)
- [MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification](https://mlsanity.com/paper/2509.07477/) (2025)
- [OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classification](https://mlsanity.com/paper/2606.19838/) (2026)
- [Prospective clinical indication, post-hoc report leakage, and fusion design in multi-image chest radiograph classification: a patient-clustered evaluation](https://mlsanity.com/paper/2607.13800/) (2026)
- [SLT: Robust Quantum Neural Networks for Noisy-Label Medical Image Classification via Supermartingale-based Label Transition](https://mlsanity.com/paper/2607.16293/) (2026)
- [Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation](https://mlsanity.com/paper/2608.05341/) (2026)
- [When Repository Labels Are Not Image-Level Truth: A Supervision Auditing Framework for Chest Radiograph AI](https://mlsanity.com/paper/2608.10084/) (2026)
- [RadFusion: Towards Threshold-Controllable Radiology Report Generation](https://mlsanity.com/paper/2608.10505/) (2026)

## Abstract (arXiv)

Automatic extraction of medical conditions from free-text radiology reports is critical for supervising computer vision models to interpret medical images. In this work, we show that radiologists labeling reports significantly disagree with radiologists labeling corresponding chest X-ray images, which reduces the quality of report labels as proxies for image labels. We develop and evaluate methods to produce labels from radiology reports that have better agreement with radiologists labeling images. Our best performing method, called VisualCheXbert, uses a biomedically-pretrained BERT model to directly map from a radiology report to the image labels, with a supervisory signal determined by a computer vision model trained to detect medical conditions from chest X-ray images. We find that VisualCheXbert outperforms an approach using an existing radiology report labeler by an average F1 score of 0.14 (95% CI 0.12, 0.17). We also find that VisualCheXbert better agrees with radiologists labeling chest X-ray images than do radiologists labeling the corresponding radiology reports by an average F1 score across several medical conditions of between 0.12 (95% CI 0.09, 0.15) and 0.21 (95% CI 0.18, 0.24).