# Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies

arXiv: https://arxiv.org/abs/2007.15778 · HTML: https://mlsanity.com/paper/2007.15778/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays](https://mlsanity.com/paper/2206.12704/) (2022)
- [GREN: Graph-Regularized Embedding Network for Weakly-Supervised Disease Localization in X-ray Images](https://mlsanity.com/paper/2107.06442/) (2021)
- [You've Got Two Teachers: Co-evolutionary Image and Report Distillation for Semi-supervised Anatomical Abnormality Detection in Chest X-ray](https://mlsanity.com/paper/2307.09184/) (2023)
- [Using Multi-modal Data for Improving Generalizability and Explainability of Disease Classification in Radiology](https://mlsanity.com/paper/2207.14781/) (2022)
- [Weighing Features of Lung and Heart Regions for Thoracic Disease Classification](https://mlsanity.com/paper/2105.12430/) (2021)
- [Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation](https://mlsanity.com/paper/2412.13599/) (2024)
- [Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations](https://mlsanity.com/paper/2511.00456/) (2025)
- [Anatomy-Structured Hierarchical MIL for Weakly-Supervised Thoracic Disease Detection in Chest X-rays](https://mlsanity.com/paper/2609.33520/) (2026)

## Abstract (arXiv)

Detecting clinically relevant objects in medical images is a challenge despite large datasets due to the lack of detailed labels. To address the label issue, we utilize the scene-level labels with a detection architecture that incorporates natural language information. We present a challenging new set of radiologist paired bounding box and natural language annotations on the publicly available MIMIC-CXR dataset especially focussed on pneumonia and pneumothorax. Along with the dataset, we present a joint vision language weakly supervised transformer layer-selected one-stage dual head detection architecture (LITERATI) alongside strong baseline comparisons with class activation mapping (CAM), gradient CAM, and relevant implementations on the NIH ChestXray-14 and MIMIC-CXR dataset. Borrowing from advances in vision language architectures, the LITERATI method demonstrates joint image and referring expression (objects localized in the image using natural language) input for detection that scales in a purely weakly supervised fashion. The architectural modifications address three obstacles -- implementing a supervised vision and language detection method in a weakly supervised fashion, incorporating clinical referring expression natural language information, and generating high fidelity detections with map probabilities. Nevertheless, the challenging clinical nature of the radiologist annotations including subtle references, multi-instance specifications, and relatively verbose underlying medical reports, ensures the vision language detection task at scale remains stimulating for future investigation.