# Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays

arXiv: https://arxiv.org/abs/2206.12704 · HTML: https://mlsanity.com/paper/2206.12704/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 25 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [NegBio: a high-performance tool for negation and uncertainty detection in radiology reports](https://mlsanity.com/paper/1712.05898/) (2017)
- [Thoracic Disease Identification and Localization with Limited Supervision](https://mlsanity.com/paper/1711.06373/) (2017)
- [Mixture Proportion Estimation and PU Learning: A Modern Approach](https://mlsanity.com/paper/2111.00980/) (2021)
- [VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels](https://mlsanity.com/paper/2102.11467/) (2021)
- [Improving Pneumonia Localization via Cross-Attention on Medical Images and Reports](https://mlsanity.com/paper/2110.03094/) (2021)
- [Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies](https://mlsanity.com/paper/2007.15778/) (2020)
- [Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop](https://mlsanity.com/paper/2104.04968/) (2021)

## Built on by (most cited first, 21 of 21)

- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography](https://mlsanity.com/paper/2405.12255/) (2024)
- [Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework](https://mlsanity.com/paper/2403.07636/) (2024)
- [CardiacNet: Learning to Reconstruct Abnormalities for Cardiac Disease Assessment from Echocardiogram Videos](https://mlsanity.com/paper/2410.20769/) (2024)
- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [Anatomy-Driven Pathology Detection on Chest X-rays](https://mlsanity.com/paper/2309.02578/) (2023)
- [Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays](https://mlsanity.com/paper/2207.04394/) (2022)
- [Finding-Aware Anatomical Tokens for Chest X-Ray Automated Reporting](https://mlsanity.com/paper/2308.15961/) (2023)
- [Towards Universal Unsupervised Anomaly Detection in Medical Imaging](https://mlsanity.com/paper/2401.10637/) (2024)
- [Weakly Supervised Object Detection in Chest X-Rays with Differentiable ROI Proposal Networks and Soft ROI Pooling](https://mlsanity.com/paper/2402.11985/) (2024)
- [Distilling BlackBox to Interpretable models for Efficient Transfer Learning](https://mlsanity.com/paper/2305.17303/) (2023)
- [Enhanced Knowledge Injection for Radiology Report Generation](https://mlsanity.com/paper/2311.00399/) (2023)
- [Tackling Shortcut Learning in Deep Neural Networks: An Iterative Approach with Interpretable Models](https://mlsanity.com/paper/2302.10289/) (2023)
- [You've Got Two Teachers: Co-evolutionary Image and Report Distillation for Semi-supervised Anatomical Abnormality Detection in Chest X-ray](https://mlsanity.com/paper/2307.09184/) (2023)
- [Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models](https://mlsanity.com/paper/2506.10633/) (2025)
- [Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation](https://mlsanity.com/paper/2412.13599/) (2024)
- [Medical Report Generation: A Hierarchical Task Structure-Based Cross-Modal Causal Intervention Framework](https://mlsanity.com/paper/2511.02271/) (2025)
- [OrthoAI: A Neurosymbolic Framework for Evidence-Grounded Biomechanical Reasoning in Clear Aligner Orthodontics](https://mlsanity.com/paper/2603.00124/) (2026)
- [Clinically-Grounded Hierarchical Classification for Consistent Chest X-ray Interpretation](https://mlsanity.com/paper/2608.03016/) (2026)
- [Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation](https://mlsanity.com/paper/2608.05341/) (2026)
- [Anatomy-Structured Hierarchical MIL for Weakly-Supervised Thoracic Disease Detection in Chest X-rays](https://mlsanity.com/paper/2609.33520/) (2026)

## Abstract (arXiv)

Creating a large-scale dataset of abnormality annotation on medical images is a labor-intensive and costly task. Leveraging weak supervision from readily available data such as radiology reports can compensate lack of large-scale data for anomaly detection methods. However, most of the current methods only use image-level pathological observations, failing to utilize the relevant anatomy mentions in reports. Furthermore, Natural Language Processing (NLP)-mined weak labels are noisy due to label sparsity and linguistic ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to address these issues of weak annotation. Our framework consists of a cascade of two networks, one responsible for identifying anatomical abnormalities and the second responsible for pathological observations. The critical component in our framework is an anatomy-guided attention module that aids the downstream observation network in focusing on the relevant anatomical regions generated by the anatomy network. We use Positive Unlabeled (PU) learning to account for the fact that lack of mention does not necessarily mean a negative label. Our quantitative and qualitative results on the MIMIC-CXR dataset demonstrate the effectiveness of AGXNet in disease and anatomical abnormality localization. Experiments on the NIH Chest X-ray dataset show that the learned feature representations are transferable and can achieve the state-of-the-art performances in disease classification and competitive disease localization results. Our code is available at https://github.com/batmanlab/AGXNet