# Image Captioning with Deep Bidirectional LSTMs

arXiv: https://arxiv.org/abs/1604.00790 · HTML: https://mlsanity.com/paper/1604.00790/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2016.

## Built on by (most cited first, 21 of 21)

- [Detector-Free Weakly Supervised Grounding by Separation](https://mlsanity.com/paper/2104.09829/) (2021)
- [A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval](https://mlsanity.com/paper/2208.02080/) (2022)
- [HICEScore: A Hierarchical Metric for Image Captioning Evaluation](https://mlsanity.com/paper/2407.18589/) (2024)
- [Shadow-Aware Dynamic Convolution for Shadow Removal](https://mlsanity.com/paper/2205.04908/) (2022)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet](https://mlsanity.com/paper/2109.05797/) (2021)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [Cross Modal Compression: Towards Human-comprehensible Semantic Compression](https://mlsanity.com/paper/2209.02574/) (2022)
- [A spatio-temporal LSTM model to forecast across multiple temporal and spatial scales](https://mlsanity.com/paper/2108.11875/) (2021)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [Progressive Tree-Structured Prototype Network for End-to-End Image Captioning](https://mlsanity.com/paper/2211.09460/) (2022)
- [NeuroView-RNN: It's About Time](https://mlsanity.com/paper/2202.11811/) (2022)
- [Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score](https://mlsanity.com/paper/2405.16091/) (2024)
- [Scene Graph Generation for Better Image Captioning?](https://mlsanity.com/paper/2109.11398/) (2021)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)
- [APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference](https://mlsanity.com/paper/2201.08830/) (2022)
- [Long Short-Term Memory to predict 3D Amino acids Positions in GPCR Molecular Dynamics](https://mlsanity.com/paper/2207.05682/) (2022)
- [AIC-AB NET: A Neural Network for Image Captioning with Spatial Attention and Text Attributes](https://mlsanity.com/paper/2307.07370/) (2023)
- [Predicting Outcomes in Video Games with Long Short Term Memory Networks](https://mlsanity.com/paper/2402.15923/) (2024)
- [A Critical Analysis on Machine Learning Techniques for Video-based Human Activity Recognition of Surveillance Systems: A Review](https://mlsanity.com/paper/2409.00731/) (2024)
- [Using Deep Learning to Generate Semantically Correct Hindi Captions](https://mlsanity.com/paper/2602.13352/) (2026)

## Abstract (arXiv)

This work presents an end-to-end trainable deep bidirectional LSTM (Long-Short Term Memory) model for image captioning. Our model builds on a deep convolutional neural network (CNN) and two separate LSTM networks. It is capable of learning long term visual-language interactions by making use of history and future context information at high level semantic space. Two novel deep bidirectional variant models, in which we increase the depth of nonlinearity transition in different way, are proposed to learn hierarchical visual-language embeddings. Data augmentation techniques such as multi-crop, multi-scale and vertical mirror are proposed to prevent overfitting in training deep models. We visualize the evolution of bidirectional LSTM internal states over time and qualitatively analyze how our models "translate" image to sentence. Our proposed models are evaluated on caption generation and image-sentence retrieval tasks with three benchmark datasets: Flickr8K, Flickr30K and MSCOCO datasets. We demonstrate that bidirectional LSTM models achieve highly competitive performance to the state-of-the-art results on caption generation even without integrating additional mechanism (e.g. object detection, attention model etc.) and significantly outperform recent methods on retrieval task.