# Self-supervised Cross-view Representation Reconstruction for Change Captioning

arXiv: https://arxiv.org/abs/2309.16283 · HTML: https://mlsanity.com/paper/2309.16283/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Image Difference Captioning with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2202.04298/) (2022)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/2207.08386/) (2022)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning](https://mlsanity.com/paper/2110.10328/) (2021)

## Built on by (most cited first, 20 of 20)

- [R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation](https://mlsanity.com/paper/2310.08872/) (2023)
- [Synergistic Multi-Agent Framework with Trajectory Learning for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2407.09893/) (2024)
- [OneDiff: A Generalist Model for Image Difference Captioning](https://mlsanity.com/paper/2407.05645/) (2024)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [Context-aware Difference Distilling for Multi-change Captioning](https://mlsanity.com/paper/2405.20810/) (2024)
- [Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing](https://mlsanity.com/paper/2503.12042/) (2025)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning](https://mlsanity.com/paper/2412.11467/) (2024)
- [Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation](https://mlsanity.com/paper/2412.15939/) (2024)
- [Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images](https://mlsanity.com/paper/2405.12875/) (2024)
- [Multimodal Fusion and Coherence Modeling for Video Topic Segmentation](https://mlsanity.com/paper/2408.00365/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)
- [Image Difference Grounding with Natural Language](https://mlsanity.com/paper/2504.01952/) (2025)
- [Imagine How To Change: Explicit Procedure Modeling for Change Captioning](https://mlsanity.com/paper/2603.05969/) (2026)
- [Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning](https://mlsanity.com/paper/2412.13543/) (2024)
- [Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning](https://mlsanity.com/paper/2603.12832/) (2026)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing](https://mlsanity.com/paper/2607.01728/) (2026)
- [Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning](https://mlsanity.com/paper/2609.01136/) (2026)

## Abstract (arXiv)

Change captioning aims to describe the difference between a pair of similar images. Its key challenge is how to learn a stable difference representation under pseudo changes caused by viewpoint change. In this paper, we address this by proposing a self-supervised cross-view representation reconstruction (SCORER) network. Concretely, we first design a multi-head token-wise matching to model relationships between cross-view features from similar/dissimilar images. Then, by maximizing cross-view contrastive alignment of two similar images, SCORER learns two view-invariant image representations in a self-supervised way. Based on these, we reconstruct the representations of unchanged objects by cross-attention, thus learning a stable difference representation for caption generation. Further, we devise a cross-modal backward reasoning to improve the quality of caption. This module reversely models a ``hallucination'' representation with the caption and ``before'' representation. By pushing it closer to the ``after'' representation, we enforce the caption to be informative about the difference in a self-supervised manner. Extensive experiments show our method achieves the state-of-the-art results on four datasets. The code is available at https://github.com/tuyunbin/SCORER.