# Explain Images with Multimodal Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1410.1090 · HTML: https://mlsanity.com/paper/1410.1090/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2014.

## Built on by (most cited first, 15 of 15)

- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors](https://mlsanity.com/paper/2211.13224/) (2022)
- [Heterogeneous Contrastive Learning](https://mlsanity.com/paper/2105.09401/) (2021)
- [Object Recognition as Next Token Prediction](https://mlsanity.com/paper/2312.02142/) (2023)
- [User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning](https://mlsanity.com/paper/2312.04793/) (2023)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling](https://mlsanity.com/paper/2203.05346/) (2022)
- [Geometry-Entangled Visual Semantic Transformer for Image Captioning](https://mlsanity.com/paper/2109.14137/) (2021)
- [A Comprehensive Analysis of Real-World Image Captioning and Scene Identification](https://mlsanity.com/paper/2308.02833/) (2023)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [Pharmacophore-based design by learning on voxel grids](https://mlsanity.com/paper/2512.02031/) (2025)
- [Characterization and recognition of handwritten digits using Julia](https://mlsanity.com/paper/2102.11994/) (2021)
- [Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning](https://mlsanity.com/paper/2302.04676/) (2023)
- [Data Transformation Strategies to Remove Heterogeneity](https://mlsanity.com/paper/2507.12677/) (2025)
- [Using Deep Learning to Generate Semantically Correct Hindi Captions](https://mlsanity.com/paper/2602.13352/) (2026)

## Abstract (arXiv)

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.