# Hierarchical LSTMs with Adaptive Attention for Visual Captioning

arXiv: https://arxiv.org/abs/1812.11004 · HTML: https://mlsanity.com/paper/1812.11004/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2018.

## Built on by (most cited first, 18 of 18)

- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [A Spatial-channel-temporal-fused Attention for Spiking Neural Networks](https://mlsanity.com/paper/2209.10837/) (2022)
- [Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning](https://mlsanity.com/paper/2403.05770/) (2024)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Multispectral Object Detection with Deep Learning](https://mlsanity.com/paper/2102.03115/) (2021)
- [Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning](https://mlsanity.com/paper/2109.07799/) (2021)
- [Relation Regularized Scene Graph Generation](https://mlsanity.com/paper/2202.10826/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)
- [Attention in Reasoning: Dataset, Analysis, and Modeling](https://mlsanity.com/paper/2204.09774/) (2022)
- [SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities](https://mlsanity.com/paper/2411.01975/) (2024)
- [Multi-modal reward for visual relationships-based image captioning](https://mlsanity.com/paper/2303.10766/) (2023)
- [Collaborative Three-Stream Transformers for Video Captioning](https://mlsanity.com/paper/2309.09611/) (2023)
- [How to Understand Named Entities: Using Common Sense for News Captioning](https://mlsanity.com/paper/2403.06520/) (2024)
- [DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning](https://mlsanity.com/paper/2604.08084/) (2026)

## Abstract (arXiv)

Recent progress has been made in using attention based encoder-decoder framework for image and video captioning. Most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g. "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of visual captioning. Furthermore, the hierarchy of LSTMs enables more complex representation of visual data, capturing information at different scales. To address these issues, we propose a hierarchical LSTM with adaptive attention (hLSTMat) approach for image and video captioning. Specifically, the proposed framework utilizes the spatial or temporal attention for selecting specific regions or frames to predict the related words, while the adaptive attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the caption generation. We initially design our hLSTMat for video captioning task. Then, we further refine it and apply it to image captioning task. To demonstrate the effectiveness of our proposed framework, we test our method on both video and image captioning tasks. Experimental results show that our approach achieves the state-of-the-art performance for most of the evaluation metrics on both tasks. The effect of important components is also well exploited in the ablation study.