# Learning to Describe Differences Between Pairs of Similar Images

arXiv: https://arxiv.org/abs/1808.10584 · HTML: https://mlsanity.com/paper/1808.10584/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2018.

## Built on by (most cited first, 30 of 114)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation](https://mlsanity.com/paper/2207.14315/) (2022)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023)
- [MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://mlsanity.com/paper/2409.20566/) (2024)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models](https://mlsanity.com/paper/2501.14818/) (2025)
- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Changes to Captions: An Attentive Network for Remote Sensing Change Captioning](https://mlsanity.com/paper/2304.01091/) (2023)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Visual Instruction Tuning with Polite Flamingo](https://mlsanity.com/paper/2307.01003/) (2023)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [CLIP4IDC: CLIP for Image Difference Captioning](https://mlsanity.com/paper/2206.00629/) (2022)
- [VisMin: Visual Minimal-Change Understanding](https://mlsanity.com/paper/2407.16772/) (2024)
- [Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models](https://mlsanity.com/paper/2501.05767/) (2025)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval](https://mlsanity.com/paper/2204.00486/) (2022)
- [The Change You Want to See (Now in 3D)](https://mlsanity.com/paper/2308.10417/) (2023)

## Abstract (arXiv)

In this paper, we introduce the task of automatically generating text to describe the differences between two similar images. We collect a new dataset by crowd-sourcing difference descriptions for pairs of image frames extracted from video-surveillance footage. Annotators were asked to succinctly describe all the differences in a short paragraph. As a result, our novel dataset provides an opportunity to explore models that align language and vision, and capture visual salience. The dataset may also be a useful benchmark for coherent multi-sentence generation. We perform a firstpass visual analysis that exposes clusters of differing pixels as a proxy for object-level differences. We propose a model that captures visual salience by using a latent variable to align clusters of differing pixels with output sentences. We find that, for both single-sentence generation and as well as multi-sentence generation, the proposed model outperforms the models that use attention alone.