# Query-controllable Video Summarization

arXiv: https://arxiv.org/abs/2004.03661 · HTML: https://mlsanity.com/paper/2004.03661/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 24 of 24)

- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics](https://mlsanity.com/paper/2406.15264/) (2024)
- [Contextualized Keyword Representations for Multi-modal Retinal Image Captioning](https://mlsanity.com/paper/2104.12471/) (2021)
- [Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"](https://mlsanity.com/paper/2105.14538/) (2021)
- [Transcript to Video: Efficient Clip Sequencing from Texts](https://mlsanity.com/paper/2107.11851/) (2021)
- [Query-based Video Summarization with Pseudo Label Supervision](https://mlsanity.com/paper/2307.01945/) (2023)
- [A Novel Evaluation Framework for Image2Text Generation](https://mlsanity.com/paper/2408.01723/) (2024)
- [Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions](https://mlsanity.com/paper/2304.03147/) (2023)
- [QFMTS: Generating Query-Focused Summaries over Multi-Table Inputs](https://mlsanity.com/paper/2405.05109/) (2024)
- [Causalainer: Causal Explainer for Automatic Video Summarization](https://mlsanity.com/paper/2305.00455/) (2023)
- [Conditional Modeling Based Automatic Video Summarization](https://mlsanity.com/paper/2311.12159/) (2023)
- [Causal Video Summarizer for Video Exploration](https://mlsanity.com/paper/2307.01947/) (2023)
- [Personalized Video Summarization by Multimodal Video Understanding](https://mlsanity.com/paper/2411.03531/) (2024)
- [SD-VSum: A Method and Dataset for Script-Driven Video Summarization](https://mlsanity.com/paper/2505.03319/) (2025)
- [A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation](https://mlsanity.com/paper/2408.12398/) (2024)
- ["Previously on ..." From Recaps to Story Summarization](https://mlsanity.com/paper/2405.11487/) (2024)
- [DeepEyeNet: Generating Medical Report for Retinal Images](https://mlsanity.com/paper/2509.12534/) (2025)
- [Find the Cliffhanger: Multi-Modal Trailerness in Soap Operas](https://mlsanity.com/paper/2401.16076/) (2024)
- [SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasets](https://mlsanity.com/paper/2510.05652/) (2025)
- [A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech](https://mlsanity.com/paper/2604.06327/) (2026)
- [LVSum: A Benchmark for Timestamp-Aware Long Video Summarization](https://mlsanity.com/paper/2604.10024/) (2026)
- [FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding](https://mlsanity.com/paper/2607.25266/) (2026)

## Abstract (arXiv)

When video collections become huge, how to explore both within and across videos efficiently is challenging. Video summarization is one of the ways to tackle this issue. Traditional summarization approaches limit the effectiveness of video exploration because they only generate one fixed video summary for a given input video independent of the information need of the user. In this work, we introduce a method which takes a text-based query as input and generates a video summary corresponding to it. We do so by modeling video summarization as a supervised learning problem and propose an end-to-end deep learning based method for query-controllable video summarization to generate a query-dependent video summary. Our proposed method consists of a video summary controller, video summary generator, and video summary output module. To foster the research of query-controllable video summarization and conduct our experiments, we introduce a dataset that contains frame-based relevance score labels. Based on our experimental result, it shows that the text-based query helps control the video summary. It also shows the text-based query improves our model performance. Our code and dataset: https://github.com/Jhhuangkay/Query-controllable-Video-Summarization.