# Rethinking Radiology Report Generation via Causal Inspired Counterfactual Augmentation

arXiv: https://arxiv.org/abs/2311.13307 · HTML: https://mlsanity.com/paper/2311.13307/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://mlsanity.com/paper/1611.06607/) (2016)
- [Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation](https://mlsanity.com/paper/1903.10122/) (2019)
- [Counterfactual Data Augmentation using Locally Factored Dynamics](https://mlsanity.com/paper/2007.02863/) (2020)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning](https://mlsanity.com/paper/2310.13864/) (2023)
- [Masked Images Are Counterfactual Samples for Robust Fine-tuning](https://mlsanity.com/paper/2303.03052/) (2023)
- [Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition](https://mlsanity.com/paper/2101.03287/) (2021)

## Built on by (most cited first, 3 of 3)

- [MAMBO-NET: Multi-Causal Aware Modeling Backdoor-Intervention Optimization for Medical Image Segmentation Network](https://mlsanity.com/paper/2505.21874/) (2025)
- [RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture](https://mlsanity.com/paper/2601.15891/) (2026)
- [Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty](https://mlsanity.com/paper/2606.28520/) (2026)

## Abstract (arXiv)

Radiology Report Generation (RRG) draws attention as a vision-and-language interaction of biomedical fields. Previous works inherited the ideology of traditional language generation tasks, aiming to generate paragraphs with high readability as reports. Despite significant progress, the independence between diseases-a specific property of RRG-was neglected, yielding the models being confused by the co-occurrence of diseases brought on by the biased data distribution, thus generating inaccurate reports. In this paper, to rethink this issue, we first model the causal effects between the variables from a causal perspective, through which we prove that the co-occurrence relationships between diseases on the biased distribution function as confounders, confusing the accuracy through two backdoor paths, i.e. the Joint Vision Coupling and the Conditional Sequential Coupling. Then, we proposed a novel model-agnostic counterfactual augmentation method that contains two strategies, i.e. the Prototype-based Counterfactual Sample Synthesis (P-CSS) and the Magic-Cube-like Counterfactual Report Reconstruction (Cube), to intervene the backdoor paths, thus enhancing the accuracy and generalization of RRG models. Experimental results on the widely used MIMIC-CXR dataset demonstrate the effectiveness of our proposed method. Additionally, a generalization performance is evaluated on IU X-Ray dataset, which verifies our work can effectively reduce the impact of co-occurrences caused by different distributions on the results.