# Learning a Recurrent Visual Representation for Image Caption Generation

arXiv: https://arxiv.org/abs/1411.5654 · HTML: https://mlsanity.com/paper/1411.5654/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2014.

## Built on by (most cited first, 12 of 12)

- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Text-Only Training for Image Captioning using Noise-Injected CLIP](https://mlsanity.com/paper/2211.00575/) (2022)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning](https://mlsanity.com/paper/2212.13563/) (2022)
- [A Frustratingly Simple Approach for End-to-End Image Captioning](https://mlsanity.com/paper/2201.12723/) (2022)
- [From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping](https://mlsanity.com/paper/2304.13273/) (2023)
- [Learning text-to-video retrieval from image captioning](https://mlsanity.com/paper/2404.17498/) (2024)
- [Bench-Marking And Improving Arabic Automatic Image Captioning Through The Use Of Multi-Task Learning Paradigm](https://mlsanity.com/paper/2202.05474/) (2022)
- [Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation](https://mlsanity.com/paper/2504.16516/) (2025)
- [TM-PATHVQA:90000+ Textless Multilingual Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2407.11383/) (2024)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps](https://mlsanity.com/paper/2410.15780/) (2024)

## Abstract (arXiv)

In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both sentences and images to a common embedding, we enable the generation of novel sentences given an image. Using the same model, we can also reconstruct the visual features associated with an image given its visual description. We use a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are preferred by humans over 19.8% of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.