# Image Difference Captioning with Pre-training and Contrastive Learning

arXiv: https://arxiv.org/abs/2202.04298 · HTML: https://mlsanity.com/paper/2202.04298/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up](https://mlsanity.com/paper/1903.02827/) (2019)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning](https://mlsanity.com/paper/2106.14019/) (2021)
- [Neural Naturalist: Generating Fine-Grained Image Comparisons](https://mlsanity.com/paper/1909.04101/) (2019)
- [Stack-Captioning: Coarse-to-Fine Learning for Image Captioning](https://mlsanity.com/paper/1709.03376/) (2017)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [L2C: Describing Visual Differences Needs Semantic Understanding of Individuals](https://mlsanity.com/paper/2102.01860/) (2021)
- [Learning to Guide Decoding for Image Captioning](https://mlsanity.com/paper/1804.00887/) (2018)

## Built on by (most cited first, 30 of 39)

- [DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models](https://mlsanity.com/paper/2405.20985/) (2024)
- [Describing Differences in Image Sets with Natural Language](https://mlsanity.com/paper/2312.02974/) (2023)
- [Changes to Captions: An Attentive Network for Remote Sensing Change Captioning](https://mlsanity.com/paper/2304.01091/) (2023)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning](https://mlsanity.com/paper/2509.22647/) (2025)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [Analysing the Robustness of Vision-Language-Models to Common Corruptions](https://mlsanity.com/paper/2504.13690/) (2025)
- [OneDiff: A Generalist Model for Image Difference Captioning](https://mlsanity.com/paper/2407.05645/) (2024)
- [Video Action Differencing](https://mlsanity.com/paper/2503.07860/) (2025)
- [Supervised Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2209.00078/) (2022)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions](https://mlsanity.com/paper/2602.08711/) (2026)
- [Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays](https://mlsanity.com/paper/2402.08966/) (2024)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [Finetuning CLIP to Reason about Pairwise Differences](https://mlsanity.com/paper/2409.09721/) (2024)
- [Towards a multimodal framework for remote sensing image change retrieval and captioning](https://mlsanity.com/paper/2406.13424/) (2024)
- [ConViS-Bench: Estimating Video Similarity Through Semantic Concepts](https://mlsanity.com/paper/2509.19245/) (2025)
- [Edit As You Wish: Video Caption Editing with Multi-grained User Control](https://mlsanity.com/paper/2305.08389/) (2023)
- [Viewpoint Integration and Registration with Vision Language Foundation Model for Image Change Understanding](https://mlsanity.com/paper/2309.08585/) (2023)
- [Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations](https://mlsanity.com/paper/2410.00436/) (2024)
- [Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation](https://mlsanity.com/paper/2412.15939/) (2024)
- [Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports](https://mlsanity.com/paper/2505.16624/) (2025)
- [TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models](https://mlsanity.com/paper/2412.18675/) (2024)
- [ViDiC: Video Difference Captioning](https://mlsanity.com/paper/2512.03405/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective](https://mlsanity.com/paper/2410.16662/) (2024)
- [Image Difference Grounding with Natural Language](https://mlsanity.com/paper/2504.01952/) (2025)
- [Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images](https://mlsanity.com/paper/2504.08727/) (2025)

## Abstract (arXiv)

The Image Difference Captioning (IDC) task aims to describe the visual differences between two similar images with natural language. The major challenges of this task lie in two aspects: 1) fine-grained visual differences that require learning stronger vision and language association and 2) high-cost of manual annotations that leads to limited supervised data. To address these challenges, we propose a new modeling framework following the pre-training-finetuning paradigm. Specifically, we design three self-supervised tasks and contrastive learning strategies to align visual differences and text descriptions at a fine-grained level. Moreover, we propose a data expansion strategy to utilize extra cross-task supervision information, such as data for fine-grained image classification, to alleviate the limitation of available supervised IDC data. Extensive experiments on two IDC benchmark datasets, CLEVR-Change and Birds-to-Words, demonstrate the effectiveness of the proposed modeling framework. The codes and models will be released at https://github.com/yaolinli/IDC.