# What is the Role of Recurrent Neural Networks (RNNs) in an Image Caption Generator?

arXiv: https://arxiv.org/abs/1708.02043 · HTML: https://mlsanity.com/paper/1708.02043/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2017.

## Built on by (most cited first, 7 of 7)

- [Improved Bengali Image Captioning via deep convolutional neural network based encoder-decoder model](https://mlsanity.com/paper/2102.07192/) (2021)
- [BAN-Cap: A Multi-Purpose English-Bangla Image Descriptions Dataset](https://mlsanity.com/paper/2205.14462/) (2022)
- [Bangla Image Caption Generation through CNN-Transformer based Encoder-Decoder Network](https://mlsanity.com/paper/2110.12442/) (2021)
- [Audio Captioning with Composition of Acoustic and Semantic Information](https://mlsanity.com/paper/2105.06355/) (2021)
- [Learning to embed semantic similarity for joint image-text retrieval](https://mlsanity.com/paper/2210.03838/) (2022)
- [Modeling and Predicting Blood Flow Characteristics through Double Stenosed Artery from CFD simulation using Deep Learning Models](https://mlsanity.com/paper/2112.03698/) (2021)
- [Facial Expression Recognition and Image Description Generation in Vietnamese](https://mlsanity.com/paper/2208.06117/) (2022)

## Abstract (arXiv)

In neural image captioning systems, a recurrent neural network (RNN) is typically viewed as the primary `generation' component. This view suggests that the image features should be `injected' into the RNN. This is in fact the dominant view in the literature. Alternatively, the RNN can instead be viewed as only encoding the previously generated words. This view suggests that the RNN should only be used to encode linguistic features and that only the final representation should be `merged' with the image features at a later stage. This paper compares these two architectures. We find that, in general, late merging outperforms injection, suggesting that RNNs are better viewed as encoders, rather than generators.