# Medical Report Generation based on Segment-Enhanced Contrastive Representation Learning

arXiv: https://arxiv.org/abs/2312.15869 · HTML: https://mlsanity.com/paper/2312.15869/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation](https://mlsanity.com/paper/1603.08486/) (2016)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [Automated Generation of Accurate & Fluent Medical X-ray Reports](https://mlsanity.com/paper/2108.12126/) (2021)
- [Representative Image Feature Extraction via Contrastive Learning Pretraining for Chest X-ray Report Generation](https://mlsanity.com/paper/2209.01604/) (2022)
- [CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation](https://mlsanity.com/paper/2303.01094/) (2023)

## Built on by (most cited first, 3 of 3)

- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models](https://mlsanity.com/paper/2411.11362/) (2024)
- [MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generation](https://mlsanity.com/paper/2512.16145/) (2025)

## Abstract (arXiv)

Automated radiology report generation has the potential to improve radiology reporting and alleviate the workload of radiologists. However, the medical report generation task poses unique challenges due to the limited availability of medical data and the presence of data bias. To maximize the utility of available data and reduce data bias, we propose MSCL (Medical image Segmentation with Contrastive Learning), a framework that utilizes the Segment Anything Model (SAM) to segment organs, abnormalities, bones, etc., and can pay more attention to the meaningful ROIs in the image to get better visual representations. Then we introduce a supervised contrastive loss that assigns more weight to reports that are semantically similar to the target while training. The design of this loss function aims to mitigate the impact of data bias and encourage the model to capture the essential features of a medical image and generate high-quality reports. Experimental results demonstrate the effectiveness of our proposed model, where we achieve state-of-the-art performance on the IU X-Ray public dataset.