# Finding-Aware Anatomical Tokens for Chest X-Ray Automated Reporting

arXiv: https://arxiv.org/abs/2308.15961 · HTML: https://mlsanity.com/paper/2308.15961/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Benchmarking Detection Transfer Learning with Vision Transformers](https://mlsanity.com/paper/2111.11429/) (2021)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [Chest X-ray Report Generation through Fine-Grained Label Learning](https://mlsanity.com/paper/2007.13831/) (2020)
- [Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays](https://mlsanity.com/paper/2206.12704/) (2022)
- [AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-ray](https://mlsanity.com/paper/2105.09937/) (2021)
- [Self adaptive global-local feature enhancement for radiology report generation](https://mlsanity.com/paper/2211.11380/) (2022)

## Built on by (most cited first, 6 of 6)

- [Controllable Chest X-Ray Report Generation from Longitudinal Representations](https://mlsanity.com/paper/2310.05881/) (2023)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports](https://mlsanity.com/paper/2505.16624/) (2025)
- [Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components](https://mlsanity.com/paper/2407.21638/) (2024)
- [Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts](https://mlsanity.com/paper/2411.10789/) (2024)

## Abstract (arXiv)

The task of radiology reporting comprises describing and interpreting the medical findings in radiographic images, including description of their location and appearance. Automated approaches to radiology reporting require the image to be encoded into a suitable token representation for input to the language model. Previous methods commonly use convolutional neural networks to encode an image into a series of image-level feature map representations. However, the generated reports often exhibit realistic style but imperfect accuracy. Inspired by recent works for image captioning in the general domain in which each visual token corresponds to an object detected in an image, we investigate whether using local tokens corresponding to anatomical structures can improve the quality of the generated reports. We introduce a novel adaptation of Faster R-CNN in which finding detection is performed for the candidate bounding boxes extracted during anatomical structure localisation. We use the resulting bounding box feature representations as our set of finding-aware anatomical tokens. This encourages the extracted anatomical tokens to be informative about the findings they contain (required for the final task of radiology reporting). Evaluating on the MIMIC-CXR dataset of chest X-Ray images, we show that task-aware anatomical tokens give state-of-the-art performance when integrated into an automated reporting pipeline, yielding generated reports with improved clinical accuracy.