# A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation

arXiv: https://arxiv.org/abs/2104.10326 · HTML: https://mlsanity.com/paper/2104.10326/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Bottom-up Object Detection by Grouping Extreme and Center Points](https://mlsanity.com/paper/1901.08043/) (2019)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Feature Selective Anchor-Free Module for Single-Shot Object Detection](https://mlsanity.com/paper/1903.00621/) (2019)
- [Thoracic Disease Identification and Localization with Limited Supervision](https://mlsanity.com/paper/1711.06373/) (2017)
- [Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification](https://mlsanity.com/paper/1801.09927/) (2018)
- [Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays](https://mlsanity.com/paper/1807.06067/) (2018)
- [Deep Hiearchical Multi-Label Classification Applied to Chest X-Ray Abnormality Taxonomies](https://mlsanity.com/paper/2009.05609/) (2020)
- [Learning what to look in chest X-rays with a recurrent visual attention model](https://mlsanity.com/paper/1701.06452/) (2017)

## Built on by (most cited first, 30 of 34)

- [MedRAX: Medical Reasoning Agent for Chest X-ray](https://mlsanity.com/paper/2502.02673/) (2025)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation](https://mlsanity.com/paper/2308.03726/) (2023)
- [BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys](https://mlsanity.com/paper/2310.10765/) (2023)
- [Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning](https://mlsanity.com/paper/2302.09636/) (2023)
- [MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation](https://mlsanity.com/paper/2409.19684/) (2024)
- [Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation](https://mlsanity.com/paper/2311.01092/) (2023)
- [Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision](https://mlsanity.com/paper/2404.15672/) (2024)
- [A foundation model for generalizable disease diagnosis in chest X-ray images](https://mlsanity.com/paper/2410.08861/) (2024)
- [Towards Foundation Models Learned from Anatomy in Medical Imaging via Self-Supervision](https://mlsanity.com/paper/2309.15358/) (2023)
- [Self-supervised visual learning in the low-data regime: a comparative evaluation](https://mlsanity.com/paper/2404.17202/) (2024)
- [S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2408.06447/) (2024)
- [DeViDe: Faceted medical knowledge for improved medical vision-language pre-training](https://mlsanity.com/paper/2404.03618/) (2024)
- [CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation](https://mlsanity.com/paper/2510.21324/) (2025)
- [A unified multi-task framework enables interpretable chest radiograph analysis](https://mlsanity.com/paper/2606.03417/) (2026)
- [DWARF: Disease-weighted network for attention map refinement](https://mlsanity.com/paper/2406.17032/) (2024)
- [The FIX Benchmark: Extracting Features Interpretable to eXperts](https://mlsanity.com/paper/2409.13684/) (2024)
- [GREN: Graph-Regularized Embedding Network for Weakly-Supervised Disease Localization in X-ray Images](https://mlsanity.com/paper/2107.06442/) (2021)
- [Foundation X: Integrating Classification, Localization, and Segmentation through Lock-Release Pretraining Strategy for Chest X-ray Analysis](https://mlsanity.com/paper/2503.09860/) (2025)
- [Risk Controlled Image Retrieval](https://mlsanity.com/paper/2307.07336/) (2023)
- [Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identification](https://mlsanity.com/paper/2507.21703/) (2025)
- [Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks](https://mlsanity.com/paper/2511.06665/) (2025)
- [Clinical Domain Knowledge-Derived Template Improves Post Hoc AI Explanations in Pneumothorax Classification](https://mlsanity.com/paper/2403.18871/) (2024)
- [Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging](https://mlsanity.com/paper/2412.20070/) (2024)
- [ACE: Anatomically Consistent Embeddings in Composition and Decomposition](https://mlsanity.com/paper/2501.10131/) (2025)
- [Understanding Dataset Bias in Medical Imaging: A Case Study on Chest X-rays](https://mlsanity.com/paper/2507.07722/) (2025)
- [Uncertainty-Aware Compositional Localization and Placement Assessment of Catheters and Tubes in Chest X-Rays](https://mlsanity.com/paper/2608.11288/) (2026)
- [Foundation Model Makes Clustering A Better Initialization For Cold-Start Active Learning](https://mlsanity.com/paper/2402.02561/) (2024)
- [Automatic Fine-grained Segmentation-assisted Report Generation](https://mlsanity.com/paper/2507.16623/) (2025)
- [AURAD: Anatomy-Pathology Unified Radiology Synthesis with Progressive Representations](https://mlsanity.com/paper/2509.04819/) (2025)

## Abstract (arXiv)

Instance level detection and segmentation of thoracic diseases or abnormalities are crucial for automatic diagnosis in chest X-ray images. Leveraging on constant structure and disease relations extracted from domain knowledge, we propose a structure-aware relation network (SAR-Net) extending Mask R-CNN. The SAR-Net consists of three relation modules: 1. the anatomical structure relation module encoding spatial relations between diseases and anatomical parts. 2. the contextual relation module aggregating clues based on query-key pair of disease RoI and lung fields. 3. the disease relation module propagating co-occurrence and causal relations into disease proposals. Towards making a practical system, we also provide ChestX-Det, a chest X-Ray dataset with instance-level annotations (boxes and masks). ChestX-Det is a subset of the public dataset NIH ChestX-ray14. It contains ~3500 images of 13 common disease categories labeled by three board-certified radiologists. We evaluate our SAR-Net on it and another dataset DR-Private. Experimental results show that it can enhance the strong baseline of Mask R-CNN with significant improvements. The ChestX-Det is released at https://github.com/Deepwise-AILab/ChestX-Det-Dataset.