mlsanityData to 7 Oct 2026

Video Summarization Using Deep Neural Networks: A Survey

arXiv 2101.06072 ↗First posted 15 Jan 2021cs.CV cs.LG cs.MMcited by 85

Not on the map. Why?

Abstract (arXiv)

Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content. Several approaches have been developed over the last couple of decades and the current state of the art is represented by methods that rely on modern deep neural network architectures. This work focuses on the recent advances in the area and provides a comprehensive survey of the existing deep-learning-based methods for generic video summarization. After presenting the motivation behind the development of technologies for video summarization, we formulate the video summarization task and discuss the main characteristics of a typical deep-learning-based analysis pipeline. Then, we suggest a taxonomy of the existing algorithms and provide a systematic review of the relevant literature that shows the evolution of the deep-learning-based video summarization technologies and leads to suggestions for future developments. We then report on protocols for the objective evaluation of video summarization algorithms and we compare the performance of several deep-learning-based approaches. Based on the outcomes of these comparisons, as well as some documented considerations about the amount of annotated data and the suitability of evaluation protocols, we indicate potential future research directions.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 55 of its 175 references that are in the map. First posted on arXiv 15 Jan 2021.

Builds on

55 of 175 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  12. 2015Pointer Networkscited by 882
38 more
  1. 2017Graph Signal Processing: Overview, Challenges and Applicationscited by 409
  2. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  3. 2015Recent Advances in Convolutional Neural Networkscited by 360
  4. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 281
  5. 2016Video Summarization with Long Short-term Memorycited by 142
  6. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  7. 2017Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Rewardcited by 105
  8. 2018Lightweight Probabilistic Deep Networkscited by 77
  9. 2018Video Summarization Using Fully Convolutional Sequence Networkscited by 71
  10. 2018Summarizing Videos with Attentioncited by 69
  11. 2017Video Summarization with Attention-Based Encoder-Decoder Networkscited by 63
  12. 2019Rethinking the Evaluation of Video Summariescited by 62
  13. 2019Hierarchical Recurrent Neural Network for Video Summarizationcited by 44
  14. 2018Graph Spectral Image Processingcited by 41
  15. 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
  16. 2018Discriminative Feature Learning for Unsupervised Video Summarizationcited by 37
  17. 2018Video Summarization by Learning from Unpaired Datacited by 36
  18. 2020Exploring global diverse attention via pairwise temporal relation for video summarizationcited by 29
  19. 2018Robustness of Conditional GANs to Noisy Labelscited by 27
  20. 2020A Survey on Deep Learning for Neuroimaging-based Brain Disorder Analysiscited by 27
  21. 2016Query-Focused Extractive Video Summarizationcited by 26
  22. 2018Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluationcited by 26
  23. 2016Summary Transfer: Exemplar-based Subset Selection for Video Summarizationcited by 24
  24. 2019Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarizationcited by 24
  25. 2020Query-controllable Video Summarizationcited by 24
  26. 2019Image Difficulty Curriculum for Generative Adversarial Networks (CuGAN)cited by 23
  27. 2017Query-adaptive Video Summarization via Quality-aware Relevance Estimationcited by 22
  28. 2017Denoising Adversarial Autoencoderscited by 20
  29. 2020Weakly Supervised Video Summarization by Hierarchical Reinforcement Learningcited by 18
  30. 2016Title Generation for User Generated Videoscited by 17
  31. 2016Video Summarization using Deep Semantic Featurescited by 15
  32. 2017Summarizing First-Person Videos from Third Persons' Points of Viewscited by 15
  33. 2018On-line Adaptative Curriculum Learning for GANscited by 14
  34. 2015Understanding deep features with computer-generated imagerycited by 13
  35. 2019A General Framework for Edited Video and Raw Video Summarizationcited by 12
  36. 2017Video Highlight Prediction Using Audience Chat Reactionscited by 11
  37. 2018Dilated Temporal Relational Adversarial Network for Generic Video Summarizationcited by 11
  38. 2018Query-Conditioned Three-Player Adversarial Network for Video Summarizationcited by 11

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 17; 2023: 20; 2024: 20; 2025: 15; 2026: 6.

85 papers.

Sort:

By area:

  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  3. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  4. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  5. 2023Diffusion Action Segmentationcited by 74
  6. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 52
  7. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50
  8. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  9. 2024R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Groundingcited by 34
  10. 2021GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarizationcited by 26
  11. 2022A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directionscited by 21
  12. 2021Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reductioncited by 14
38 more
  1. 2022A Survey on Medical Document Summarizationcited by 14
  2. 2022Summary-Oriented Vision Modeling for Multimodal Abstractive Summarizationcited by 14
  3. 2021Diverse Generation from a Single Video Made Possiblecited by 13
  4. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  5. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  6. 2022A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detectioncited by 10
  7. 2024Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Groundingcited by 10
  8. 2024A Survey of Video Datasets for Grounded Event Understandingcited by 9
  9. 2024MMSummary: Multimodal Summary Generation for Fetal Ultrasound Videocited by 9
  10. 2024Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomalycited by 9
  11. 2022Temporal and Contextual Transformer for Multi-Camera Editing of TV Showscited by 8
  12. 2022Tighter Abstract Queries in Neural Network Verificationcited by 8
  13. 2023Causalainer: Causal Explainer for Automatic Video Summarizationcited by 8
  14. 2023Experience and Evidence are the eyes of an excellent summarizer! Towards Knowledge Infused Multi-modal Clinical Conversation Summarizationcited by 8
  15. 2021Unsupervised Video Summarization via Multi-source Featurescited by 7
  16. 2021AutoPhoto: Aesthetic Photo Capture using Reinforcement Learningcited by 7
  17. 2022MHMS: Multimodal Hierarchical Multimedia Summarizationcited by 6
  18. 2022Video Summarization Overviewcited by 6
  19. 2023Conditional Modeling Based Automatic Video Summarizationcited by 6
  20. 2024Cluster-based Video Summarization with Temporal Context Awarenesscited by 6
  21. 2024EgoSonics: Generating Synchronized Audio for Silent Egocentric Videoscited by 6
  22. 2024On the Efficiency of NLP-Inspired Methods for Tabular Deep Learningcited by 6
  23. 2025From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representationscited by 6
  24. 2023Learning to Summarize and Answer Questions about a Virtual Robot's Past Actionscited by 5
  25. 2023VideoSum: A Python Library for Surgical Video Summarizationcited by 4
  26. 2023Causal Video Summarizer for Video Explorationcited by 4
  27. 2023Prompt-enhanced Hierarchical Transformer Elevating Cardiopulmonary Resuscitation Instruction via Temporal Action Segmentationcited by 4
  28. 2025Moment Quantization for Video Temporal Groundingcited by 4
  29. 2026HVPNet: A Bio-Inspired Network for General Salient and Camouflaged Object Detectioncited by 4
  30. 2021Use of Affective Visual Information for Summarization of Human-Centric Videoscited by 3
  31. 2022Summarizing a virtual robot's past actions in natural languagecited by 3
  32. 2022TLDW: Extreme Multimodal Summarisation of News Videoscited by 3
  33. 2023Video Timeline Modeling For News Story Understandingcited by 3
  34. 2023Unsupervised Video Summarization via Iterative Training and Simplified GANcited by 3
  35. 2024VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrievalcited by 3
  36. 2025SD-VSum: A Method and Dataset for Script-Driven Video Summarizationcited by 3
  37. 2025Prompts to Summaries: Zero-Shot Language-Guided Video Summarization with Large Language and Video Modelscited by 3
  38. 2026Where to Focus: Query-Modulated Multimodal Keyframe Selection for Long Video Understandingcited by 3

50 most cited of 85 All 85 (JSON)