# Recurrent Relational Memory Network for Unsupervised Image Captioning

arXiv: https://arxiv.org/abs/2006.13611 · HTML: https://mlsanity.com/paper/2006.13611/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Text-based Person Search without Parallel Image-Text Data](https://mlsanity.com/paper/2305.12964/) (2023)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning](https://mlsanity.com/paper/2308.13218/) (2023)
- [CgT-GAN: CLIP-guided Text GAN for Image Captioning](https://mlsanity.com/paper/2308.12045/) (2023)
- [Object-Centric Unsupervised Image Captioning](https://mlsanity.com/paper/2112.00969/) (2021)
- [Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition](https://mlsanity.com/paper/2203.03195/) (2022)
- [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation](https://mlsanity.com/paper/2307.14750/) (2023)
- [Semi-Supervised Image Captioning Considering Wasserstein Graph Matching](https://mlsanity.com/paper/2403.17995/) (2024)
- [How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey](https://mlsanity.com/paper/2412.08158/) (2024)
- [AGIC: Attention-Guided Image Captioning to Improve Caption Relevance](https://mlsanity.com/paper/2508.06853/) (2025)
- [Generative Retrieval for Unsupervised Text-Based Person Search](https://mlsanity.com/paper/2609.12965/) (2026)

## Abstract (arXiv)

Unsupervised image captioning with no annotations is an emerging challenge in computer vision, where the existing arts usually adopt GAN (Generative Adversarial Networks) models. In this paper, we propose a novel memory-based network rather than GAN, named Recurrent Relational Memory Network (R2M). Unlike complicated and sensitive adversarial learning that non-ideally performs for long sentence generation, R2M implements a concepts-to-sentence memory translator through two-stage memory mechanisms: fusion and recurrent memories, correlating the relational reasoning between common visual concepts and the generated words for long periods. R2M encodes visual context through unsupervised training on images, while enabling the memory to learn from irrelevant textual corpus via supervised fashion. Our solution enjoys less learnable parameters and higher computational efficiency than GAN-based methods, which heavily bear parameter sensitivity. We experimentally validate the superiority of R2M than state-of-the-arts on all benchmark datasets.