# Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays

arXiv: https://arxiv.org/abs/2402.08966 · HTML: https://mlsanity.com/paper/2402.08966/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [Multiple Meta-model Quantifying for Medical Visual Question Answering](https://mlsanity.com/paper/2105.08913/) (2021)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)

## Built on by (most cited first, 9 of 9)

- [CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding](https://mlsanity.com/paper/2502.20509/) (2025)
- [Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking](https://mlsanity.com/paper/2412.20750/) (2024)
- [Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports](https://mlsanity.com/paper/2505.16624/) (2025)
- [See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis](https://mlsanity.com/paper/2506.18140/) (2025)
- [LUMEN: Longitudinal Multi-Modal Radiology Model for Prognosis and Diagnosis](https://mlsanity.com/paper/2602.21142/) (2026)
- [MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays](https://mlsanity.com/paper/2605.15574/) (2026)
- [Saliency Guided Longitudinal Medical Visual Question Answering](https://mlsanity.com/paper/2509.25374/) (2025)
- [MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence](https://mlsanity.com/paper/2603.27176/) (2026)
- [Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models](https://mlsanity.com/paper/2606.06534/) (2026)

## Abstract (arXiv)

Difference visual question answering (diff-VQA) is a challenging task that requires answering complex questions based on differences between a pair of images. This task is particularly important in reading chest X-ray images because radiologists often compare multiple images of the same patient taken at different times to track disease progression and changes in its severity in their clinical practice. However, previous works focused on designing specific network architectures for the diff-VQA task, missing opportunities to enhance the model's performance using a pretrained vision-language model (VLM). Here, we introduce a novel VLM called PLURAL, which is pretrained on natural and longitudinal chest X-ray data for the diff-VQA task. The model is developed using a step-by-step approach, starting with being pretrained on natural images and texts, followed by being trained using longitudinal chest X-ray data. The longitudinal data consist of pairs of X-ray images, along with question-answer sets and radiologist's reports that describe the changes in lung abnormalities and diseases over time. Our experimental results show that the PLURAL model outperforms state-of-the-art methods not only in diff-VQA for longitudinal X-rays but also in conventional VQA for a single X-ray image. Through extensive experiments, we demonstrate the effectiveness of the proposed VLM architecture and pretraining method in improving the model's performance.