# Neighborhood Contrastive Transformer for Change Captioning

arXiv: https://arxiv.org/abs/2303.03171 · HTML: https://mlsanity.com/paper/2303.03171/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network](https://mlsanity.com/paper/1908.10072/) (2019)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Multi-scale Matching Networks for Semantic Correspondence](https://mlsanity.com/paper/2108.00211/) (2021)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Context-Aware Group Captioning via Self-Attention and Contrastive Features](https://mlsanity.com/paper/2004.03708/) (2020)
- [Fine-Grained Image Captioning with Global-Local Discriminative Objective](https://mlsanity.com/paper/2007.10662/) (2020)
- [R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning](https://mlsanity.com/paper/2110.10328/) (2021)
- [Dual Attention on Pyramid Feature Maps for Image Captioning](https://mlsanity.com/paper/2011.01385/) (2020)

## Built on by (most cited first, 16 of 16)

- [Changes to Captions: An Attentive Network for Remote Sensing Change Captioning](https://mlsanity.com/paper/2304.01091/) (2023)
- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models](https://mlsanity.com/paper/2408.04594/) (2024)
- [OneDiff: A Generalist Model for Image Difference Captioning](https://mlsanity.com/paper/2407.05645/) (2024)
- [VIXEN: Visual Text Comparison Network for Image Difference Captioning](https://mlsanity.com/paper/2402.19119/) (2024)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [Towards a multimodal framework for remote sensing image change retrieval and captioning](https://mlsanity.com/paper/2406.13424/) (2024)
- [Viewpoint Integration and Registration with Vision Language Foundation Model for Image Change Understanding](https://mlsanity.com/paper/2309.08585/) (2023)
- [Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation](https://mlsanity.com/paper/2412.15939/) (2024)
- [Composed Multi-modal Retrieval: A Survey of Approaches and Applications](https://mlsanity.com/paper/2503.01334/) (2025)
- [Multimodal Fake News Video Explanation: Dataset, Analysis and Evaluation](https://mlsanity.com/paper/2501.08514/) (2025)
- [Image Difference Grounding with Natural Language](https://mlsanity.com/paper/2504.01952/) (2025)
- [Imagine How To Change: Explicit Procedure Modeling for Change Captioning](https://mlsanity.com/paper/2603.05969/) (2026)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning](https://mlsanity.com/paper/2609.01136/) (2026)

## Abstract (arXiv)

Change captioning is to describe the semantic change between a pair of similar images in natural language. It is more challenging than general image captioning, because it requires capturing fine-grained change information while being immune to irrelevant viewpoint changes, and solving syntax ambiguity in change descriptions. In this paper, we propose a neighborhood contrastive transformer to improve the model's perceiving ability for various changes under different scenes and cognition ability for complex syntax structure. Concretely, we first design a neighboring feature aggregating to integrate neighboring context into each feature, which helps quickly locate the inconspicuous changes under the guidance of conspicuous referents. Then, we devise a common feature distilling to compare two images at neighborhood level and extract common properties from each image, so as to learn effective contrastive information between them. Finally, we introduce the explicit dependencies between words to calibrate the transformer decoder, which helps better understand complex syntax structure during training. Extensive experimental results demonstrate that the proposed method achieves the state-of-the-art performance on three public datasets with different change scenarios. The code is available at https://github.com/tuyunbin/NCT.