# Video Summarisation by Classification with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1807.03089 · HTML: https://mlsanity.com/paper/1807.03089/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 10 of 10)

- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net](https://mlsanity.com/paper/2106.10528/) (2021)
- [Query-based Video Summarization with Pseudo Label Supervision](https://mlsanity.com/paper/2307.01945/) (2023)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [Causalainer: Causal Explainer for Automatic Video Summarization](https://mlsanity.com/paper/2305.00455/) (2023)
- [MHMS: Multimodal Hierarchical Multimedia Summarization](https://mlsanity.com/paper/2204.03734/) (2022)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization](https://mlsanity.com/paper/2204.08352/) (2022)
- [Personalized Video Summarization using Text-Based Queries and Conditional Modeling](https://mlsanity.com/paper/2408.14743/) (2024)

## Abstract (arXiv)

Most existing video summarisation methods are based on either supervised or unsupervised learning. In this paper, we propose a reinforcement learning-based weakly supervised method that exploits easy-to-obtain, video-level category labels and encourages summaries to contain category-related information and maintain category recognisability. Specifically, We formulate video summarisation as a sequential decision-making process and train a summarisation network with deep Q-learning (DQSN). A companion classification network is also trained to provide rewards for training the DQSN. With the classification network, we develop a global recognisability reward based on the classification result. Critically, a novel dense ranking-based reward is also proposed in order to cope with the temporally delayed and sparse reward problems for long sequence reinforcement learning. Extensive experiments on two benchmark datasets show that the proposed approach achieves state-of-the-art performance.