# Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

arXiv: https://arxiv.org/abs/1706.01231 · HTML: https://mlsanity.com/paper/1706.01231/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2017.

## Built on by (most cited first, 19 of 19)

- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [Relational Graph Learning for Grounded Video Description Generation](https://mlsanity.com/paper/2112.00967/) (2021)
- [NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative](https://mlsanity.com/paper/2406.06499/) (2024)
- [Structured Two-stream Attention Network for Video Question Answering](https://mlsanity.com/paper/2206.01017/) (2022)
- [One-shot Scene Graph Generation](https://mlsanity.com/paper/2202.10824/) (2022)
- [From Pixels to Objects: Cubic Visual Attention for Visual Question Answering](https://mlsanity.com/paper/2206.01923/) (2022)
- [SEM-POS: Grammatically and Semantically Correct Video Captioning](https://mlsanity.com/paper/2303.14829/) (2023)
- [Guided Attention for Interpretable Motion Captioning](https://mlsanity.com/paper/2310.07324/) (2023)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Boosting Video Captioning with Dynamic Loss Network](https://mlsanity.com/paper/2107.11707/) (2021)
- [Video Captioning in Compressed Video](https://mlsanity.com/paper/2101.00359/) (2021)
- [Exploiting long-term temporal dynamics for video captioning](https://mlsanity.com/paper/2202.10828/) (2022)
- [Micromobility Flow Prediction: A Bike Sharing Station-level Study via Multi-level Spatial-Temporal Attention Neural Network](https://mlsanity.com/paper/2507.16020/) (2025)
- [EVC-MF: End-to-end Video Captioning Network with Multi-scale Features](https://mlsanity.com/paper/2410.16624/) (2024)

## Abstract (arXiv)

Recent progress has been made in using attention based encoder-decoder framework for video captioning. However, most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g. "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of video captioning. To address this issue, we propose a hierarchical LSTM with adjusted temporal attention (hLSTMat) approach for video captioning. Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict the related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the video caption generation. To demonstrate the effectiveness of our proposed framework, we test our method on two prevalent datasets: MSVD and MSR-VTT, and experimental results show that our approach outperforms the state-of-the-art methods on both two datasets.