# Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

arXiv: https://arxiv.org/abs/1707.04960 · HTML: https://mlsanity.com/paper/1707.04960/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2017.

## Built on by (most cited first, 30 of 42)

- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Hierarchical Video-Moment Retrieval and Step-Captioning](https://mlsanity.com/paper/2303.16406/) (2023)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Subset Selection](https://mlsanity.com/paper/2103.00128/) (2021)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Submodularity In Machine Learning and Artificial Intelligence](https://mlsanity.com/paper/2202.00132/) (2022)
- [TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency](https://mlsanity.com/paper/2208.06773/) (2022)
- [Reconstructive Sequence-Graph Network for Video Summarization](https://mlsanity.com/paper/2105.04066/) (2021)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [Scaling Up Video Summarization Pretraining with Large Language Models](https://mlsanity.com/paper/2404.03398/) (2024)
- [Transcript to Video: Efficient Clip Sequencing from Texts](https://mlsanity.com/paper/2107.11851/) (2021)
- [Long Story Short: Story-level Video Understanding from 20K Short Films](https://mlsanity.com/paper/2406.10221/) (2024)
- [Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Grounding](https://mlsanity.com/paper/2408.16272/) (2024)
- [DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization](https://mlsanity.com/paper/2105.06441/) (2021)
- [OmniSTVG: Toward Spatio-Temporal Omni-Object Video Grounding](https://mlsanity.com/paper/2503.10500/) (2025)
- [How Good is a Video Summary? A New Benchmarking Dataset and Evaluation Framework Towards Realistic Video Summarization](https://mlsanity.com/paper/2101.10514/) (2021)
- [Video Summarization Overview](https://mlsanity.com/paper/2210.11707/) (2022)
- [LeCoPCR: Legal Concept-guided Prior Case Retrieval for European Court of Human Rights cases](https://mlsanity.com/paper/2501.14114/) (2025)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Multimodal Frame-Scoring Transformer for Video Summarization](https://mlsanity.com/paper/2207.01814/) (2022)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [Personalized Video Summarization by Multimodal Video Understanding](https://mlsanity.com/paper/2411.03531/) (2024)
- [SD-VSum: A Method and Dataset for Script-Driven Video Summarization](https://mlsanity.com/paper/2505.03319/) (2025)
- [Prompts to Summaries: Zero-Shot Language-Guided Video Summarization with Large Language and Video Models](https://mlsanity.com/paper/2506.10807/) (2025)

## Abstract (arXiv)

Recent years have witnessed a resurgence of interest in video summarization. However, one of the main obstacles to the research on video summarization is the user subjectivity - users have various preferences over the summaries. The subjectiveness causes at least two problems. First, no single video summarizer fits all users unless it interacts with and adapts to the individual users. Second, it is very challenging to evaluate the performance of a video summarizer. To tackle the first problem, we explore the recently proposed query-focused video summarization which introduces user preferences in the form of text queries about the video into the summarization process. We propose a memory network parameterized sequential determinantal point process in order to attend the user query onto different video frames and shots. To address the second challenge, we contend that a good evaluation metric for video summarization should focus on the semantic information that humans can perceive rather than the visual features or temporal overlaps. To this end, we collect dense per-video-shot concept annotations, compile a new dataset, and suggest an efficient evaluation method defined upon the concept annotations. We conduct extensive experiments contrasting our video summarizer to existing ones and present detailed analyses about the dataset and the new evaluation method.