# Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports

arXiv: https://arxiv.org/abs/2110.03094 · HTML: https://mlsanity.com/paper/2110.03094/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization](https://mlsanity.com/paper/1704.04232/) (2017)
- [Attention-based Dropout Layer for Weakly Supervised Object Localization](https://mlsanity.com/paper/1908.10028/) (2019)
- [Contrastive Learning for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/2006.09920/) (2020)
- [COVID-19 Image Data Collection](https://mlsanity.com/paper/2003.11597/) (2020)
- [Thoracic Disease Identification and Localization with Limited Supervision](https://mlsanity.com/paper/1711.06373/) (2017)
- [Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment](https://mlsanity.com/paper/1903.11649/) (2019)
- [Weakly-supervised Visual Grounding of Phrases with Linguistic Structures](https://mlsanity.com/paper/1705.01371/) (2017)
- [Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/1908.10568/) (2019)
- [Knowledge Aided Consistency for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/1803.03879/) (2018)
- [Predicting COVID-19 Pneumonia Severity on Chest X-ray with Deep Learning](https://mlsanity.com/paper/2005.11856/) (2020)
- [Bimodal network architectures for automatic generation of image annotation from text](https://mlsanity.com/paper/1809.01610/) (2018)

## Built on by (most cited first, 21 of 21)

- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray images](https://mlsanity.com/paper/2307.03942/) (2023)
- [Detailed Annotations of Chest X-Rays via CT Projection for Report Understanding](https://mlsanity.com/paper/2210.03416/) (2022)
- [Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays](https://mlsanity.com/paper/2206.12704/) (2022)
- [Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting](https://mlsanity.com/paper/2307.05766/) (2023)
- [Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects](https://mlsanity.com/paper/2311.02332/) (2023)
- [Visual Grounding of Whole Radiology Reports for 3D CT Images](https://mlsanity.com/paper/2312.04794/) (2023)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [ConTEXTual Net: A Multimodal Vision-Language Model for Segmentation of Pneumothorax](https://mlsanity.com/paper/2303.01615/) (2023)
- [Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models](https://mlsanity.com/paper/2404.12920/) (2024)
- [FlexR: Few-shot Classification with Language Embeddings for Structured Reporting of Chest X-rays](https://mlsanity.com/paper/2203.15723/) (2022)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)
- [You've Got Two Teachers: Co-evolutionary Image and Report Distillation for Semi-supervised Anatomical Abnormality Detection in Chest X-ray](https://mlsanity.com/paper/2307.09184/) (2023)
- [Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models](https://mlsanity.com/paper/2507.12236/) (2025)
- [Optimizing Universal Lesion Segmentation: State Space Model-Guided Hierarchical Networks with Feature Importance Adjustment](https://mlsanity.com/paper/2404.17235/) (2024)
- [Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion](https://mlsanity.com/paper/2504.03135/) (2025)
- [Localizing Anatomical Landmarks in Ocular Images using Zoom-In Attentive Networks](https://mlsanity.com/paper/2210.02445/) (2022)
- [Graph-based multimodal multi-lesion DLBCL treatment response prediction from PET images](https://mlsanity.com/paper/2310.16863/) (2023)
- [Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation](https://mlsanity.com/paper/2412.13599/) (2024)
- [Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images](https://mlsanity.com/paper/2510.21839/) (2025)
- [Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting](https://mlsanity.com/paper/2603.11938/) (2026)

## Abstract (arXiv)

Localization and characterization of diseases like pneumonia are primary steps in a clinical pipeline, facilitating detailed clinical diagnosis and subsequent treatment planning. Additionally, such location annotated datasets can provide a pathway for deep learning models to be used for downstream tasks. However, acquiring quality annotations is expensive on human resources and usually requires domain expertise. On the other hand, medical reports contain a plethora of information both about pneumonia characteristics and its location. In this paper, we propose a novel weakly-supervised attention-driven deep learning model that leverages encoded information in medical reports during training to facilitate better localization. Our model also performs classification of attributes that are associated to pneumonia and extracted from medical reports for supervision. Both the classification and localization are trained in conjunction and once trained, the model can be utilized for both the localization and characterization of pneumonia using only the input image. In this paper, we explore and analyze the model using chest X-ray datasets and demonstrate qualitatively and quantitatively that the introduction of textual information improves pneumonia localization. We showcase quantitative results on two datasets, MIMIC-CXR and Chest X-ray-8, and we also showcase severity characterization on the COVID-19 dataset.