# Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

arXiv: https://arxiv.org/abs/1603.08486 · HTML: https://mlsanity.com/paper/1603.08486/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2016.

## Built on by (most cited first, 30 of 59)

- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment](https://mlsanity.com/paper/2112.15011/) (2021)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022)
- [Progressive Transformer-Based Generation of Radiology Reports](https://mlsanity.com/paper/2102.09777/) (2021)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports](https://mlsanity.com/paper/2306.08749/) (2023)
- [Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting](https://mlsanity.com/paper/2310.17811/) (2023)
- [Inference of captions from histopathological patches](https://mlsanity.com/paper/2202.03432/) (2022)
- [GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification](https://mlsanity.com/paper/2305.18221/) (2023)
- [Medical Image Captioning via Generative Pretrained Transformers](https://mlsanity.com/paper/2209.13983/) (2022)
- [Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"](https://mlsanity.com/paper/2409.16603/) (2024)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation](https://mlsanity.com/paper/2408.16213/) (2024)
- [Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images](https://mlsanity.com/paper/2308.07688/) (2023)
- [Replace and Report: NLP Assisted Radiology Report Generation](https://mlsanity.com/paper/2306.17180/) (2023)
- [Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation](https://mlsanity.com/paper/2209.01604/) (2022)
- [Diagnostic Captioning: A Survey](https://mlsanity.com/paper/2101.07299/) (2021)
- [Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self-supervised representations](https://mlsanity.com/paper/2504.08584/) (2025)
- [Out-of-Distribution Generalized Dynamic Graph Neural Network for Human Albumin Prediction](https://mlsanity.com/paper/2311.15545/) (2023)
- [Breast Ultrasound Report Generation using LangChain](https://mlsanity.com/paper/2312.03013/) (2023)
- [BB-GCN: A Bi-modal Bridged Graph Convolutional Network for Multi-label Chest X-Ray Recognition](https://mlsanity.com/paper/2302.11082/) (2023)
- [Cross-dimensional transfer learning in medical image segmentation with deep learning](https://mlsanity.com/paper/2307.15872/) (2023)
- [DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching](https://mlsanity.com/paper/2412.14535/) (2024)
- [Multi-label Ranking: Mining Multi-label and Label Ranking Data](https://mlsanity.com/paper/2101.00583/) (2021)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [Medical Report Generation based on Segment-Enhanced Contrastive Representation Learning](https://mlsanity.com/paper/2312.15869/) (2023)

## Abstract (arXiv)

Despite the recent advances in automatically describing image contents, their applications have been mostly limited to image caption datasets containing natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep learning model to efficiently detect a disease from an image and annotate its contexts (e.g., location, severity and the affected organs). We employ a publicly available radiology dataset of chest x-rays and their reports, and use its image annotations to mine disease names to train convolutional neural networks (CNNs). In doing so, we adopt various regularization techniques to circumvent the large normal-vs-diseased cases bias. Recurrent neural networks (RNNs) are then trained to describe the contexts of a detected disease, based on the deep CNN features. Moreover, we introduce a novel approach to use the weights of the already trained pair of CNN/RNN on the domain-specific image/text dataset, to infer the joint image/text contexts for composite image labeling. Significantly improved image annotation results are demonstrated using the recurrent neural cascade model by taking the joint image/text contexts into account.