# R3Net:Relation-embedded Representation Reconstruction Network for Change Captioning

arXiv: https://arxiv.org/abs/2110.10328 · HTML: https://mlsanity.com/paper/2110.10328/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation](https://mlsanity.com/paper/2007.08742/) (2020)
- [Semantic Compositional Networks for Visual Captioning](https://mlsanity.com/paper/1611.08002/) (2016)
- [Expressing Visual Relationships via Language](https://mlsanity.com/paper/1906.07689/) (2019)
- [Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach](https://mlsanity.com/paper/1909.02201/) (2019)
- [Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning](https://mlsanity.com/paper/2009.14352/) (2020)
- [A Neural Compositional Paradigm for Image Captioning](https://mlsanity.com/paper/1810.09630/) (2018)
- [REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning](https://mlsanity.com/paper/1909.02217/) (2019)
- [CapWAP: Captioning with a Purpose](https://mlsanity.com/paper/2011.04264/) (2020)

## Built on by (most cited first, 12 of 12)

- [Self-supervised Cross-view Representation Reconstruction for Change Captioning](https://mlsanity.com/paper/2309.16283/) (2023)
- [Neighborhood Contrastive Transformer for Change Captioning](https://mlsanity.com/paper/2303.03171/) (2023)
- [Improving Image Captioning via Predicting Structured Concepts](https://mlsanity.com/paper/2311.08223/) (2023)
- [OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning](https://mlsanity.com/paper/2503.11093/) (2025)
- [Context-aware Difference Distilling for Multi-change Captioning](https://mlsanity.com/paper/2405.20810/) (2024)
- [Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning](https://mlsanity.com/paper/2407.11683/) (2024)
- [Viewpoint Integration and Registration with Vision Language Foundation Model for Image Change Understanding](https://mlsanity.com/paper/2309.08585/) (2023)
- [Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning](https://mlsanity.com/paper/2412.11467/) (2024)
- [Diffusion-RSCC: Diffusion Probabilistic Model for Change Captioning in Remote Sensing Images](https://mlsanity.com/paper/2405.12875/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [Different Changes Require Different Reasoning: Change-Type-Specialized Experts for Robust Change Captioning](https://mlsanity.com/paper/2609.01136/) (2026)

## Abstract (arXiv)

Change captioning is to use a natural language sentence to describe the fine-grained disagreement between two similar images. Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the representation of real change. In this paper, we propose a Relation-embedded Representation Reconstruction Network (R3Net) to explicitly distinguish the real change from the large amount of clutter and irrelevant changes. Specifically, a relation-embedded module is first devised to explore potential changed objects in the large amount of clutter. Then, based on the semantic similarities of corresponding locations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and further model the difference representation. Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the semantic interaction between change localization and caption generation. Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets.