# Deep Unsupervised Key Frame Extraction for Efficient Video Classification

arXiv: https://arxiv.org/abs/2211.06742 · HTML: https://mlsanity.com/paper/2211.06742/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 77 references that are in the map.
- First posted on arXiv 12 Nov 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [Joint Unsupervised Learning of Deep Representations and Image Clusters](https://mlsanity.com/paper/1604.03628/) (2016)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Spatiotemporal Residual Networks for Video Action Recognition](https://mlsanity.com/paper/1611.02155/) (2016)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks](https://mlsanity.com/paper/1510.00562/) (2015)
- [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017)
- [Spatiotemporal Pyramid Network for Video Action Recognition](https://mlsanity.com/paper/1903.01038/) (2019)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)
- [Procedural Generation of Videos to Train Deep Action Recognition Networks](https://mlsanity.com/paper/1612.00881/) (2016)
- [Actions ~ Transformations](https://mlsanity.com/paper/1512.00795/) (2015)
- [Im2Flow: Motion Hallucination from Static Images for Action Recognition](https://mlsanity.com/paper/1712.04109/) (2017)
- [A robust and efficient video representation for action recognition](https://mlsanity.com/paper/1504.05524/) (2015)
- [Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice](https://mlsanity.com/paper/1405.4506/) (2014)
- [Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition](https://mlsanity.com/paper/1411.6660/) (2014)
- [AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos](https://mlsanity.com/paper/1611.08240/) (2016)

## Built on by (most cited first, 14 of 14)

- [Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning](https://mlsanity.com/paper/2407.05862/) (2024)
- [Large Model based Sequential Keyframe Extraction for Video Summarization](https://mlsanity.com/paper/2401.04962/) (2024)
- [M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval](https://mlsanity.com/paper/2401.17797/) (2024)
- [F-Hash: Feature-Based Hash Design for Time-Varying Volume Visualization via Multi-Resolution Tesseract Encoding](https://mlsanity.com/paper/2507.03836/) (2025)
- [TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representations](https://mlsanity.com/paper/2506.05395/) (2025)
- [AyE-Edge: Automated Deployment Space Search Empowering Accuracy yet Efficient Real-Time Object Detection on the Edge](https://mlsanity.com/paper/2408.05363/) (2024)
- [From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding](https://mlsanity.com/paper/2510.06509/) (2025)
- [HORAE: A Domain-Agnostic Language for Automated Service Regulation](https://mlsanity.com/paper/2406.06600/) (2024)
- [Efficient Retail Video Annotation: A Robust Key Frame Generation Approach for Product and Customer Interaction Analysis](https://mlsanity.com/paper/2506.14854/) (2025)
- [Causal-Aware Intelligent QoE Optimization for VR Interaction with Adaptive Keyframe Extraction](https://mlsanity.com/paper/2506.19890/) (2025)
- [T2VParser: Adaptive Decomposition Tokens for Partial Alignment in Text to Video Retrieval](https://mlsanity.com/paper/2507.20518/) (2025)
- [Re-Identifying K=ak=a with AI-Automated Video Key Frame Extraction](https://mlsanity.com/paper/2510.08775/) (2025)
- [T2M Mamba: Motion Periodicity-Saliency Coupling Approach for Stable Text-Driven Motion Generation](https://mlsanity.com/paper/2602.01352/) (2026)
- [Zero-shot video highlight detection based on text descriptions and synthetic images](https://mlsanity.com/paper/2609.14790/) (2026)

## Abstract (arXiv)

Video processing and analysis have become an urgent task since a huge amount of videos (e.g., Youtube, Hulu) are uploaded online every day. The extraction of representative key frames from videos is very important in video processing and analysis since it greatly reduces computing resources and time. Although great progress has been made recently, large-scale video classification remains an open problem, as the existing methods have not well balanced the performance and efficiency simultaneously. To tackle this problem, this work presents an unsupervised method to retrieve the key frames, which combines Convolutional Neural Network (CNN) and Temporal Segment Density Peaks Clustering (TSDPC). The proposed TSDPC is a generic and powerful framework and it has two advantages compared with previous works, one is that it can calculate the number of key frames automatically. The other is that it can preserve the temporal information of the video. Thus it improves the efficiency of video classification. Furthermore, a Long Short-Term Memory network (LSTM) is added on the top of the CNN to further elevate the performance of classification. Moreover, a weight fusion strategy of different input networks is presented to boost the performance. By optimizing both video classification and key frame extraction simultaneously, we achieve better classification performance and higher efficiency. We evaluate our method on two popular datasets (i.e., HMDB51 and UCF101) and the experimental results consistently demonstrate that our strategy achieves competitive performance and efficiency compared with the state-of-the-art approaches.