# TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.01659 · HTML: https://mlsanity.com/paper/2605.01659/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 3 May 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Exploring global diverse attention via pairwise temporal relation for video summarization](https://mlsanity.com/paper/2009.10942/) (2020)
- [Supervised Video Summarization via Multiple Feature Sets with Parallel Attention](https://mlsanity.com/paper/2104.11530/) (2021)
- [CSTA: CNN-based Spatiotemporal Attention for Video Summarization](https://mlsanity.com/paper/2405.11905/) (2024)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [Hierarchical Multimodal Transformer to Summarize Videos](https://mlsanity.com/paper/2109.10559/) (2021)
- [Query Twice: Dual Mixture Attention Meta Learning for Video Summarization](https://mlsanity.com/paper/2008.08360/) (2020)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos](https://mlsanity.com/paper/1802.08722/) (2018)
- [Video Joint Modelling Based on Hierarchical Transformer for Co-summarization](https://mlsanity.com/paper/2112.13478/) (2021)
- [Straight to the Point: Fast-forwarding Videos via Reinforcement Learning Using Textual Data](https://mlsanity.com/paper/2003.14229/) (2020)
- [Collaborative Multi-Agent Video Fast-Forwarding](https://mlsanity.com/paper/2305.17569/) (2023)
- [Fast-Forward Video Based on Semantic Extraction](https://mlsanity.com/paper/1708.04160/) (2017)
- [Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method](https://mlsanity.com/paper/2203.15778/) (2022)
- [Masked Autoencoder for Unsupervised Video Summarization](https://mlsanity.com/paper/2306.01395/) (2023)
- [TRIM: A Self-Supervised Video Summarization Framework Maximizing Temporal Relative Information and Representativeness](https://mlsanity.com/paper/2506.20588/) (2025)

## Built on by (most cited first, 1 of 1)

- [SMI: Efficient Self-Supervised Learning via Mutual-Information-Inspired Dependency Optimization](https://mlsanity.com/paper/2606.08332/) (2026)

## Abstract (arXiv)

The rapid growth of video content across domains such as surveillance, education, and social media has made efficient content understanding increasingly critical. Video summarization addresses this challenge by generating concise yet semantically meaningful representations, but existing approaches often rely on expensive manual annotations, struggle to generalize across domains, and incur significant computational costs due to complex architectures. Moreover, unsupervised and weakly supervised methods typically underperform compared to supervised counterparts in capturing long-range temporal dependencies and semantic structure. In this work, we propose TRIMMER (Temporal Relative Information Maximization for Multi-objective Efficient Reinforcement), a novel self-supervised reinforcement learning framework for video summarization. TRIMMER operates in two stages: it first learns robust representations via self-supervised learning and then performs spatio-temporal decision making through reinforcement learning guided by information-theoretic reward functions. Unlike prior approaches that rely on similarity-based objectives, our method introduces entropy-based metrics to capture higher-order temporal dynamics and semantic diversity, while computing rewards directly over selected frame indices to improve computational efficiency. Extensive experiments on standard benchmarks demonstrate that TRIMMER achieves state-of-the-art performance among unsupervised and self-supervised methods, while remaining competitive with leading supervised approaches, highlighting its effectiveness for scalable and generalizable video summarization.