# Prompts to Summaries: Zero-Shot Language-Guided Video Summarization with Large Language and Video Models

arXiv: https://arxiv.org/abs/2506.10807 · HTML: https://mlsanity.com/paper/2506.10807/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach](https://mlsanity.com/paper/1707.04960/) (2017)
- [Query-Focused Extractive Video Summarization](https://mlsanity.com/paper/1607.05177/) (2016)
- [Supervised Video Summarization via Multiple Feature Sets with Parallel Attention](https://mlsanity.com/paper/2104.11530/) (2021)
- [Hypotheses Tree Building for One-Shot Temporal Sentence Localization](https://mlsanity.com/paper/2301.01871/) (2023)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [Video Summarization using Deep Semantic Features](https://mlsanity.com/paper/1609.08758/) (2016)
- [Convolutional Hierarchical Attention Network for Query-Focused Video Summarization](https://mlsanity.com/paper/2002.03740/) (2020)

## Built on by (most cited first, 3 of 3)

- [See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection](https://mlsanity.com/paper/2601.10707/) (2026)
- [Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization](https://mlsanity.com/paper/2510.17501/) (2025)
- [Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields](https://mlsanity.com/paper/2609.03117/) (2026)

## Abstract (arXiv)

The explosive growth of video data intensified the need for flexible user-controllable summarization tools that operate without training data. Existing methods either rely on domain-specific datasets, limiting generalization, or cannot incorporate user intent expressed in natural language. We introduce Prompts-to-Summaries: the first zero-shot, text-queryable video-summarizer that converts off-the-shelf video-language models (VidLMs) captions into user-guided skims via large-language-models (LLMs) judging, without the use of training data, beating unsupervised and matching supervised methods. Our pipeline (i) segments video into scenes, (ii) produces scene descriptions with a memory-efficient batch prompting scheme that scales to hours on a single GPU, (iii) scores scene importance with an LLM via tailored prompts, and (iv) propagates scores to frames using new consistency (temporal coherence) and uniqueness (novelty) metrics for fine-grained frame importance. On SumMe and TVSum, our approach surpasses all prior data-hungry unsupervised methods and performs competitively on the Query-Focused Video Summarization benchmark, where the competing methods require supervised frame-level importance. We release VidSum-Reason, a query-driven dataset featuring long-tailed concepts and multi-step reasoning, where our framework serves as the first challenging baseline. Overall, we demonstrate that pretrained multi-modal models, when orchestrated with principled prompting and score propagation, provide a powerful foundation for universal, text-queryable video summarization.