# Describing and Localizing Multiple Changes with Transformers

arXiv: https://arxiv.org/abs/2103.14146 · HTML: https://mlsanity.com/paper/2103.14146/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Fully Convolutional Siamese Networks for Change Detection](https://mlsanity.com/paper/1810.08462/) (2018)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [Rescan: Inductive Instance Segmentation for Indoor RGBD Scans](https://mlsanity.com/paper/1909.11268/) (2019)

## Built on by (most cited first, 30 of 47)

- [Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis](https://mlsanity.com/paper/2403.19646/) (2024)
- [Changes to Captions: An Attentive Network for Remote Sensing Change Captioning](https://mlsanity.com/paper/2304.01091/) (2023)
- [Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering](https://mlsanity.com/paper/2307.11986/) (2023)
- [RSCaMa: Remote Sensing Image Change Captioning with State Space Model](https://mlsanity.com/paper/2404.18895/) (2024)
- [Progressive Scale-aware Network for Remote sensing Image Change Captioning](https://mlsanity.com/paper/2303.00355/) (2023)
- [ChangeChat: An Interactive Model for Remote Sensing Change Analysis via Multimodal Instruction Tuning](https://mlsanity.com/paper/2409.08582/) (2024)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [The Change You Want to See (Now in 3D)](https://mlsanity.com/paper/2308.10417/) (2023)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [Pixel-Level Change Detection Pseudo-Label Learning for Remote Sensing Change Captioning](https://mlsanity.com/paper/2312.15311/) (2023)
- [Spatio-Temporal Attention and Gaussian Processes for Personalized Video Gaze Estimation](https://mlsanity.com/paper/2404.05215/) (2024)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [Step Differences in Instructional Video](https://mlsanity.com/paper/2404.16222/) (2024)
- [Mask Approximation Net: A Novel Diffusion Model Approach for Remote Sensing Change Captioning](https://mlsanity.com/paper/2412.19179/) (2024)
- [Context-aware Difference Distilling for Multi-change Captioning](https://mlsanity.com/paper/2405.20810/) (2024)
- [Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays](https://mlsanity.com/paper/2402.08966/) (2024)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [ChangeMinds: Multi-task Framework for Detecting and Describing Changes in Remote Sensing](https://mlsanity.com/paper/2410.10047/) (2024)
- [C-NERF: Representing Scene Changes as Directional Consistency Difference-based NeRF](https://mlsanity.com/paper/2312.02751/) (2023)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale](https://mlsanity.com/paper/2507.06959/) (2025)
- [Improving Reference-based Distinctive Image Captioning with Contrastive Rewards](https://mlsanity.com/paper/2306.14259/) (2023)
- [MV-CC: Mask Enhanced Video Model for Remote Sensing Change Caption](https://mlsanity.com/paper/2410.23946/) (2024)
- [Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation](https://mlsanity.com/paper/2412.15939/) (2024)
- [Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports](https://mlsanity.com/paper/2505.16624/) (2025)
- [STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models](https://mlsanity.com/paper/2510.22571/) (2025)
- [SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning](https://mlsanity.com/paper/2511.21420/) (2025)
- [Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images](https://mlsanity.com/paper/2405.12875/) (2024)
- [TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models](https://mlsanity.com/paper/2412.18675/) (2024)
- [Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model](https://mlsanity.com/paper/2510.11509/) (2025)

## Abstract (arXiv)

Change captioning tasks aim to detect changes in image pairs observed before and after a scene change and generate a natural language description of the changes. Existing change captioning studies have mainly focused on a single change.However, detecting and describing multiple changed parts in image pairs is essential for enhancing adaptability to complex scenarios. We solve the above issues from three aspects: (i) We propose a simulation-based multi-change captioning dataset; (ii) We benchmark existing state-of-the-art methods of single change captioning on multi-change captioning; (iii) We further propose Multi-Change Captioning transformers (MCCFormers) that identify change regions by densely correlating different regions in image pairs and dynamically determines the related change regions with words in sentences. The proposed method obtained the highest scores on four conventional change captioning evaluation metrics for multi-change captioning. Additionally, our proposed method can separate attention maps for each change and performs well with respect to change localization. Moreover, the proposed framework outperformed the previous state-of-the-art methods on an existing change captioning benchmark, CLEVR-Change, by a large margin (+6.1 on BLEU-4 and +9.7 on CIDEr scores), indicating its general ability in change captioning tasks.