# Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net

arXiv: https://arxiv.org/abs/2106.10528 · HTML: https://mlsanity.com/paper/2106.10528/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 19 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [Collaborative Spatio-temporal Feature Learning for Video Action Recognition](https://mlsanity.com/paper/1903.01197/) (2019)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [Multiple Landmark Detection using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1907.00318/) (2019)
- [Tracking as Online Decision-Making: Learning a Policy from Streaming Videos with Reinforcement Learning](https://mlsanity.com/paper/1707.04991/) (2017)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [How Local is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization](https://mlsanity.com/paper/1807.04219/) (2018)
- [Video Summarisation by Classification with Deep Reinforcement Learning](https://mlsanity.com/paper/1807.03089/) (2018)
- [Ultrasound Video Summarization using Deep Reinforcement Learning](https://mlsanity.com/paper/2005.09531/) (2020)
- [Coupled Network for Robust Pedestrian Detection with Gated Multi-Layer Feature Extraction and Deformable Occlusion Handling](https://mlsanity.com/paper/1912.08661/) (2019)

## Built on by (most cited first, 22 of 22)

- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model](https://mlsanity.com/paper/2412.21080/) (2024)
- [Context Recovery and Knowledge Retrieval: A Novel Two-Stream Framework for Video Anomaly Detection](https://mlsanity.com/paper/2209.02899/) (2022)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimization](https://mlsanity.com/paper/2510.06040/) (2025)
- [DropletVideo: A Dataset and Approach to Explore Integral Spatio-Temporal Consistent Video Generation](https://mlsanity.com/paper/2503.06053/) (2025)
- [Deep Multi-Threshold Spiking-UNet for Image Processing](https://mlsanity.com/paper/2307.10974/) (2023)
- [Hierarchical Agent-based Reinforcement Learning Framework for Automated Quality Assessment of Fetal Ultrasound Video](https://mlsanity.com/paper/2304.07036/) (2023)
- [EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization](https://mlsanity.com/paper/2506.14356/) (2025)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Facilitating the Production of Well-tailored Video Summaries for Sharing on Social Media](https://mlsanity.com/paper/2312.02616/) (2023)
- [Video Summarization Techniques: A Comprehensive Review](https://mlsanity.com/paper/2410.04449/) (2024)
- [Video Summarization using Denoising Diffusion Probabilistic Model](https://mlsanity.com/paper/2412.08357/) (2024)
- [From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding](https://mlsanity.com/paper/2510.06509/) (2025)
- [Graph Unfolding and Sampling for Transitory Video Keyframe Selection via Gershgorin Disc Alignment](https://mlsanity.com/paper/2408.01859/) (2024)
- [A Low-Computational Video Synopsis Framework with a Standard Dataset](https://mlsanity.com/paper/2409.05230/) (2024)
- [Video Dataset Condensation with Diffusion Models](https://mlsanity.com/paper/2505.06670/) (2025)
- [Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization](https://mlsanity.com/paper/2506.23714/) (2025)
- [Re-Identifying K=ak=a with AI-Automated Video Key Frame Extraction](https://mlsanity.com/paper/2510.08775/) (2025)
- [AdSum: Two-stream Audio-visual Summarization for Automated Video Advertisement Clipping](https://mlsanity.com/paper/2510.26569/) (2025)
- [Semantic-Guided Unsupervised Video Summarization](https://mlsanity.com/paper/2601.14773/) (2026)
- [Identity-Centric Video Summarization via Hierarchical Fusion of Biometric, Appearance, and 3D Body Features](https://mlsanity.com/paper/2609.25837/) (2026)

## Abstract (arXiv)

Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatio-temporal U-Net is used to efficiently encode spatio-temporal information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatio-temporal latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatio-temporal CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information