# Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders

arXiv: https://arxiv.org/abs/1510.01442 · HTML: https://mlsanity.com/paper/1510.01442/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2015.

## Built on by (most cited first, 30 of 30)

- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Cross-category Video Highlight Detection via Set-based Learning](https://mlsanity.com/paper/2108.11770/) (2021)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding](https://mlsanity.com/paper/2311.08835/) (2023)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Towards Open Set Video Anomaly Detection](https://mlsanity.com/paper/2208.11113/) (2022)
- [Bayesian Nonparametric Submodular Video Partition for Robust Anomaly Detection](https://mlsanity.com/paper/2203.12840/) (2022)
- [FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding](https://mlsanity.com/paper/2412.13441/) (2024)
- [Multimodal Action Quality Assessment](https://mlsanity.com/paper/2402.09444/) (2024)
- [Learning Pixel-Level Distinctions for Video Highlight Detection](https://mlsanity.com/paper/2204.04615/) (2022)
- [Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward](https://mlsanity.com/paper/2209.12164/) (2022)
- [Probing Visual-Audio Representation for Video Highlight Detection via Hard-Pairs Guided Contrastive Learning](https://mlsanity.com/paper/2206.10157/) (2022)
- [Unsupervised Modality-Transferable Video Highlight Detection with Representation Activation Sequence Learning](https://mlsanity.com/paper/2403.09401/) (2024)
- [Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence](https://mlsanity.com/paper/2407.13933/) (2024)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [Deep Visual Anomaly detection with Negative Learning](https://mlsanity.com/paper/2105.11058/) (2021)
- [HighlightMe: Detecting Highlights from Human-Centric Videos](https://mlsanity.com/paper/2110.01774/) (2021)
- [Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention](https://mlsanity.com/paper/2207.08352/) (2022)
- [Masked Autoencoder for Unsupervised Video Summarization](https://mlsanity.com/paper/2306.01395/) (2023)
- [DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2308.15109/) (2023)
- [ActPrompt: In-Domain Feature Adaptation via Action Cues for Video Temporal Grounding](https://mlsanity.com/paper/2408.06622/) (2024)
- [Test-Time Adaptation for Video Highlight Detection Using Meta-Auxiliary Learning and Cross-Modality Hallucinations](https://mlsanity.com/paper/2508.04924/) (2025)
- [Audiovisual Highlight Detection in Videos](https://mlsanity.com/paper/2102.05811/) (2021)
- [Exploring Domain Incremental Video Highlights Detection with the LiveFood Benchmark](https://mlsanity.com/paper/2209.05166/) (2022)
- [Scene Summarization: Clustering Scene Videos into Spatially Diverse Frames](https://mlsanity.com/paper/2311.17940/) (2023)
- [Motion-Based Sign Language Video Summarization using Curvature and Torsion](https://mlsanity.com/paper/2305.16801/) (2023)
- [Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception](https://mlsanity.com/paper/2603.13176/) (2026)

## Abstract (arXiv)

With the growing popularity of short-form video sharing platforms such as emInstagram and emVine, there has been an increasing need for techniques that automatically extract highlights from video. Whereas prior works have approached this problem with heuristic rules or supervised learning, we present an unsupervised learning approach that takes advantage of the abundance of user-edited videos on social media websites such as YouTube. Based on the idea that the most significant sub-events within a video class are commonly present among edited videos while less interesting ones appear less frequently, we identify the significant sub-events via a robust recurrent auto-encoder trained on a collection of user-edited videos queried for each particular class of interest. The auto-encoder is trained using a proposed shrinking exponential loss function that makes it robust to noise in the web-crawled training data, and is configured with bidirectional long short term memory (LSTM)~citeLSTM:97 cells to better model the temporal structure of highlight segments. Different from supervised techniques, our method can infer highlights using only a set of downloaded edited videos, without also needing their pre-edited counterparts which are rarely available online. Extensive experiments indicate the promise of our proposed solution in this challenging unsupervised settin