# Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method

arXiv: https://arxiv.org/abs/2203.15778 · HTML: https://mlsanity.com/paper/2203.15778/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 68 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Unsupervised Learning and Segmentation of Complex Activities from Video](https://mlsanity.com/paper/1803.09490/) (2018)
- [Positional Encoding to Control Output Sequence Length](https://mlsanity.com/paper/1904.07418/) (2019)
- [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images](https://mlsanity.com/paper/1905.01273/) (2019)
- [See, Hear, and Read: Deep Aligned Representations](https://mlsanity.com/paper/1706.00932/) (2017)
- [Unsupervised Semantic Parsing of Video Collections](https://mlsanity.com/paper/1506.08438/) (2015)
- [Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation](https://mlsanity.com/paper/2003.09392/) (2020)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos](https://mlsanity.com/paper/1802.08722/) (2018)
- [EgoSampling: Fast-Forward and Stereo for Egocentric Videos](https://mlsanity.com/paper/1412.3596/) (2014)

## Built on by (most cited first, 2 of 2)

- [TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2605.01659/) (2026)
- [Identity-Centric Video Summarization via Hierarchical Fusion of Biometric, Appearance, and 3D Body Features](https://mlsanity.com/paper/2609.25837/) (2026)

## Abstract (arXiv)

The growth of videos in our digital age and the users' limited time raise the demand for processing untrimmed videos to produce shorter versions conveying the same information. Despite the remarkable progress that summarization methods have made, most of them can only select a few frames or skims, creating visual gaps and breaking the video context. This paper presents a novel weakly-supervised methodology based on a reinforcement learning formulation to accelerate instructional videos using text. A novel joint reward function guides our agent to select which frames to remove and reduce the input video to a target length without creating gaps in the final video. We also propose the Extended Visually-guided Document Attention Network (VDAN+), which can generate a highly discriminative embedding space to represent both textual and visual data. Our experiments show that our method achieves the best performance in Precision, Recall, and F1 Score against the baselines while effectively controlling the video's output length. Visit https://www.verlab.dcc.ufmg.br/semantic-hyperlapse/tpami2022/ for code and extra results.