# Hierarchical Boundary-Aware Neural Encoder for Video Captioning

arXiv: https://arxiv.org/abs/1611.09312 · HTML: https://mlsanity.com/paper/1611.09312/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2016.

## Built on by (most cited first, 20 of 20)

- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering](https://mlsanity.com/paper/2504.10920/) (2025)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [Working Memory Connections for LSTM](https://mlsanity.com/paper/2109.00020/) (2021)
- [Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention](https://mlsanity.com/paper/2109.02955/) (2021)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [Frame Sampling Strategies Matter: A Benchmark for small vision language models](https://mlsanity.com/paper/2509.14769/) (2025)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [Syntax Customized Video Captioning by Imitating Exemplar Sentences](https://mlsanity.com/paper/2112.01062/) (2021)
- [The Role of the Input in Natural Language Video Description](https://mlsanity.com/paper/2102.05067/) (2021)
- [A Challenging Multimodal Video Summary: Simultaneously Extracting and Generating Keyframe-Caption Pairs from Video](https://mlsanity.com/paper/2312.01575/) (2023)
- [Scene Detection Policies and Keyframe Extraction Strategies for Large-Scale Video Analysis](https://mlsanity.com/paper/2506.00667/) (2025)
- [Mixup Helps Understanding Multimodal Video Better](https://mlsanity.com/paper/2510.10986/) (2025)

## Abstract (arXiv)

The use of Recurrent Neural Networks for video captioning has recently gained a lot of attention, since they can be used both to encode the input video and to generate the corresponding description. In this paper, we present a recurrent video encoding scheme which can discover and leverage the hierarchical structure of the video. Unlike the classical encoder-decoder approach, in which a video is encoded continuously by a recurrent layer, we propose a novel LSTM cell, which can identify discontinuity points between frames or segments and modify the temporal connections of the encoding layer accordingly. We evaluate our approach on three large-scale datasets: the Montreal Video Annotation dataset, the MPII Movie Description dataset and the Microsoft Video Description Corpus. Experiments show that our approach can discover appropriate hierarchical representations of input videos and improve the state of the art results on movie description datasets.