# Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning

arXiv: https://arxiv.org/abs/1511.03476 · HTML: https://mlsanity.com/paper/1511.03476/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2015.

## Built on by (most cited first, 30 of 30)

- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Reconstructive Sequence-Graph Network for Video Summarization](https://mlsanity.com/paper/2105.04066/) (2021)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Reinforced Video Captioning with Entailment Rewards](https://mlsanity.com/paper/1708.02300/) (2017)
- [Hierarchical Multimodal Transformer to Summarize Videos](https://mlsanity.com/paper/2109.10559/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering](https://mlsanity.com/paper/2504.10920/) (2025)
- [Video and Text Matching with Conditioned Embeddings](https://mlsanity.com/paper/2110.11298/) (2021)
- [Consistency and Diversity induced Human Motion Segmentation](https://mlsanity.com/paper/2202.04861/) (2022)
- [Multi Sentence Description of Complex Manipulation Action Videos](https://mlsanity.com/paper/2311.07285/) (2023)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Unsupervised Audio-Visual Lecture Segmentation](https://mlsanity.com/paper/2210.16644/) (2022)
- [Network Level Spatial Temporal Traffic State Forecasting with Hierarchical-Attention-LSTM (HierAttnLSTM)](https://mlsanity.com/paper/2201.05760/) (2022)
- [MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding](https://mlsanity.com/paper/2601.05495/) (2026)
- [Exploiting long-term temporal dynamics for video captioning](https://mlsanity.com/paper/2202.10828/) (2022)
- [Temporal superimposed crossover module for effective continuous sign language](https://mlsanity.com/paper/2211.03387/) (2022)
- [A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video](https://mlsanity.com/paper/2312.01575/) (2023)
- [Dense Video Captioning using Graph-based Sentence Summarization](https://mlsanity.com/paper/2506.20583/) (2025)
- [Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning](https://mlsanity.com/paper/2607.29045/) (2026)
- [A Review of Web Infodemic Analysis and Detection Trends across Multi-modalities using Deep Neural Networks](https://mlsanity.com/paper/2112.00803/) (2021)
- [Adaptive Perception Transformer for Temporal Action Localization](https://mlsanity.com/paper/2208.11908/) (2022)
- [VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate It](https://mlsanity.com/paper/2407.12005/) (2024)
- [Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization](https://mlsanity.com/paper/2506.20567/) (2025)
- [MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signals](https://mlsanity.com/paper/2601.08204/) (2026)
- [ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning](https://mlsanity.com/paper/2607.21022/) (2026)

## Abstract (arXiv)

Recently, deep learning approach, especially deep Convolutional Neural Networks (ConvNets), have achieved overwhelming accuracy with fast processing speed for image classification. Incorporating temporal structure with deep ConvNets for video representation becomes a fundamental problem for video content analysis. In this paper, we propose a new approach, namely Hierarchical Recurrent Neural Encoder (HRNE), to exploit temporal information of videos. Compared to recent video representation inference approaches, this paper makes the following three contributions. First, our HRNE is able to efficiently exploit video temporal structure in a longer range by reducing the length of input information flow, and compositing multiple consecutive inputs at a higher level. Second, computation operations are significantly lessened while attaining more non-linearity. Third, HRNE is able to uncover temporal transitions between frame chunks with different granularities, i.e., it can model the temporal transitions between frames as well as the transitions between segments. We apply the new method to video captioning where temporal information plays a crucial role. Experiments demonstrate that our method outperforms the state-of-the-art on video captioning benchmarks. Notably, even using a single network with only RGB stream as input, HRNE beats all the recent systems which combine multiple inputs, such as RGB ConvNet plus 3D ConvNet.