# CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding

arXiv: https://arxiv.org/abs/2502.20509 · HTML: https://mlsanity.com/paper/2502.20509/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [HERGen: Elevating Radiology Report Generation with Longitudinal Data](https://mlsanity.com/paper/2407.15158/) (2024)
- [CheXRelNet: An Anatomy-Aware Model for Tracking Longitudinal Relationships between Chest X-Rays](https://mlsanity.com/paper/2208.03873/) (2022)
- [Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training](https://mlsanity.com/paper/2405.19654/) (2024)
- [Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation](https://mlsanity.com/paper/2311.18260/) (2023)
- [Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays](https://mlsanity.com/paper/2402.08966/) (2024)
- [MetaLR: Meta-tuning of Learning Rates for Transfer Learning in Medical Imaging](https://mlsanity.com/paper/2206.01408/) (2022)

## Built on by (most cited first, 11 of 11)

- [MedGemma 1.5 Technical Report](https://mlsanity.com/paper/2604.05081/) (2026)
- [Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering](https://mlsanity.com/paper/2505.18915/) (2025)
- [XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosis](https://mlsanity.com/paper/2604.02695/) (2026)
- [Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning](https://mlsanity.com/paper/2505.13081/) (2025)
- [Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning](https://mlsanity.com/paper/2604.15705/) (2026)
- [Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models](https://mlsanity.com/paper/2512.13072/) (2025)
- [Location-Aware Pretraining for Medical Difference Visual Question Answering](https://mlsanity.com/paper/2603.04950/) (2026)
- [Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation](https://mlsanity.com/paper/2601.16753/) (2026)
- [CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs](https://mlsanity.com/paper/2605.05810/) (2026)
- [Learning Directional Semantic Transitions for Longitudinal Chest X-ray Analysis](https://mlsanity.com/paper/2606.15938/) (2026)
- [ALTER: Modeling Longitudinal Changes via Regional Differencing for 3D CT Report Generation](https://mlsanity.com/paper/2608.05615/) (2026)

## Abstract (arXiv)

Vision-language models have proven to be of great benefit for medical image analysis since they learn rich semantics from both images and reports. Prior efforts have focused on better alignment of image and text representations to enhance image understanding. However, though explicit reference to a prior image is common in Chest X-Ray (CXR) reports, aligning progression descriptions with the semantics differences in image pairs remains under-explored. In this work, we propose two components to address this issue. (1) A CXR report processing pipeline to extract temporal structure. It processes reports with a large language model (LLM) to separate the description and comparison contexts, and extracts fine-grained annotations from reports. (2) A contrastive captioner model for CXR, namely CoCa-CXR, to learn how to both describe images and their temporal progressions. CoCa-CXR incorporates a novel regional cross-attention module to identify local differences between paired CXR images. Extensive experiments show the superiority of CoCa-CXR on both progression analysis and report generation compared to previous methods. Notably, on MS-CXR-T progression classification, CoCa-CXR obtains 65.0% average testing accuracy on five pulmonary conditions, outperforming the previous state-of-the-art (SOTA) model BioViL-T by 4.8%. It also achieves a RadGraph F1 of 24.2% on MIMIC-CXR, which is comparable to the Med-Gemini foundation model.