# Contrastive Attention for Automatic Chest X-ray Report Generation

arXiv: https://arxiv.org/abs/2106.06965 · HTML: https://mlsanity.com/paper/2106.06965/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 69 references that are in the map.
- First posted on arXiv 13 Jun 2021.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://mlsanity.com/paper/1611.06607/) (2016)
- [Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation](https://mlsanity.com/paper/1903.10122/) (2019)
- [Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://mlsanity.com/paper/1911.02541/) (2019)

## Built on by (most cited first, 30 of 71)

- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment](https://mlsanity.com/paper/2112.15011/) (2021)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations](https://mlsanity.com/paper/2211.11427/) (2022)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [HERGen: Elevating Radiology Report Generation with Longitudinal Data](https://mlsanity.com/paper/2407.15158/) (2024)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation](https://mlsanity.com/paper/2403.04945/) (2024)
- [CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generation](https://mlsanity.com/paper/2406.11451/) (2024)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning](https://mlsanity.com/paper/2310.13864/) (2023)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation](https://mlsanity.com/paper/2207.01208/) (2022)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [Radiology Report Generation via Multi-objective Preference Optimization](https://mlsanity.com/paper/2412.08901/) (2024)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation](https://mlsanity.com/paper/2311.01092/) (2023)
- [A Systematic Review of Deep Learning-based Research on Radiology Report Generation](https://mlsanity.com/paper/2311.14199/) (2023)
- [Large Language Models in the Clinic: A Comprehensive Benchmark](https://mlsanity.com/paper/2405.00716/) (2024)
- [R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation](https://mlsanity.com/paper/2408.09743/) (2024)
- [FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks](https://mlsanity.com/paper/2410.01089/) (2024)

## Abstract (arXiv)

Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge of chest X-ray report generation is to accurately capture and describe the abnormal regions. In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. Due to such data bias, learning-based models may fail to attend to abnormal regions. In this work, to effectively capture and describe abnormal regions, we propose the Contrastive Attention (CA) model. Instead of solely focusing on the current input image, the CA model compares the current input image with normal images to distill the contrastive information. The acquired contrastive information can better represent the visual features of abnormal regions. According to the experiments on the public IU-X-ray and MIMIC-CXR datasets, incorporating our CA into several existing models can boost their performance across most metrics. In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis. Specifically, we achieve the state-of-the-art results on the two public datasets.