# Video Captioning via Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.11135 · HTML: https://mlsanity.com/paper/1711.11135/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2017.

## Built on by (most cited first, 30 of 47)

- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Frame-Voyager: Learning to Query Frames for Video Large Language Models](https://mlsanity.com/paper/2410.03226/) (2024)
- [VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation](https://mlsanity.com/paper/2206.08522/) (2022)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views](https://mlsanity.com/paper/2304.04617/) (2023)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding](https://mlsanity.com/paper/2307.16226/) (2023)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [ClawCraneNet: Leveraging Object-level Relation for Text-based Video Segmentation](https://mlsanity.com/paper/2103.10702/) (2021)
- [Models See Hallucinations: Evaluating the Factuality in Video Captioning](https://mlsanity.com/paper/2303.02961/) (2023)
- [GL-RG: Global-Local Representation Granularity for Video Captioning](https://mlsanity.com/paper/2205.10706/) (2022)
- [Relational Graph Learning for Grounded Video Description Generation](https://mlsanity.com/paper/2112.00967/) (2021)
- [ALaRM: Align Language Models via Hierarchical Rewards Modeling](https://mlsanity.com/paper/2403.06754/) (2024)
- [Question-Answering Dense Video Events](https://mlsanity.com/paper/2409.04388/) (2024)
- [VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimization](https://mlsanity.com/paper/2510.06040/) (2025)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [Movie101v2: Improved Movie Narration Benchmark](https://mlsanity.com/paper/2404.13370/) (2024)
- [Planning Irregular Object Packing via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2211.09382/) (2022)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approach](https://mlsanity.com/paper/2501.01042/) (2025)
- [IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning](https://mlsanity.com/paper/2507.18531/) (2025)
- [A Survey on Natural Language Video Localization](https://mlsanity.com/paper/2104.00234/) (2021)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Moment](https://mlsanity.com/paper/2307.02682/) (2023)

## Abstract (arXiv)

Video captioning is the task of automatically generating a textual description of the actions in a video. Although previous work (e.g. sequence-to-sequence model) has shown promising results in abstracting a coarse description of a short video, it is still very challenging to caption a video containing multiple fine-grained actions with a detailed description. This paper aims to address the challenge by proposing a novel hierarchical reinforcement learning framework for video captioning, where a high-level Manager module learns to design sub-goals and a low-level Worker module recognizes the primitive actions to fulfill the sub-goal. With this compositional framework to reinforce video captioning at different levels, our approach significantly outperforms all the baseline methods on a newly introduced large-scale dataset for fine-grained video captioning. Furthermore, our non-ensemble model has already achieved the state-of-the-art results on the widely-used MSR-VTT dataset.