# DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization

arXiv: https://arxiv.org/abs/2105.06441 · HTML: https://mlsanity.com/paper/2105.06441/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 83 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [Order Matters: Sequence to sequence for sets](https://mlsanity.com/paper/1511.06391/) (2015)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach](https://mlsanity.com/paper/1707.04960/) (2017)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [Query-Focused Extractive Video Summarization](https://mlsanity.com/paper/1607.05177/) (2016)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [FFNet: Video Fast-Forwarding via Reinforcement Learning](https://mlsanity.com/paper/1805.02792/) (2018)
- [Dilated Temporal Relational Adversarial Network for Generic Video Summarization](https://mlsanity.com/paper/1804.11228/) (2018)
- [Query-Conditioned Three-Player Adversarial Network for Video Summarization](https://mlsanity.com/paper/1807.06677/) (2018)
- [Improving Sequential Determinantal Point Processes for Supervised Video Summarization](https://mlsanity.com/paper/1807.10957/) (2018)
- [VideoSET: Video Summary Evaluation through Text](https://mlsanity.com/paper/1406.5824/) (2014)
- [Diversity-aware Multi-Video Summarization](https://mlsanity.com/paper/1706.03123/) (2017)
- [Viewpoint-aware Video Summarization](https://mlsanity.com/paper/1804.02843/) (2018)
- [Can We Learn Heuristics For Graphical Model Inference Using Reinforcement Learning?](https://mlsanity.com/paper/2005.01508/) (2020)

## Built on by (most cited first, 8 of 8)

- [Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval](https://mlsanity.com/paper/2308.07648/) (2023)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [Multi-modal Segment Assemblage Network for Ad Video Editing with Importance-Coherence Reward](https://mlsanity.com/paper/2209.12164/) (2022)
- [Exploiting Pseudo Image Captions for Multimodal Summarization](https://mlsanity.com/paper/2305.05496/) (2023)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [SITransformer: Shared Information-Guided Transformer for Extreme Multimodal Summarization](https://mlsanity.com/paper/2408.15829/) (2024)
- [MMCIG: Multimodal Cover Image Generation for Text-only Documents and Its Dataset Construction via Pseudo-labeling](https://mlsanity.com/paper/2508.17199/) (2025)

## Abstract (arXiv)

The recent growth of web video sharing platforms has increased the demand for systems that can efficiently browse, retrieve and summarize video content. Query-aware multi-video summarization is a promising technique that caters to this demand. In this work, we introduce a novel Query-Aware Hierarchical Pointer Network for Multi-Video Summarization, termed DeepQAMVS, that jointly optimizes multiple criteria: (1) conciseness, (2) representativeness of important query-relevant events and (3) chronological soundness. We design a hierarchical attention model that factorizes over three distributions, each collecting evidence from a different modality, followed by a pointer network that selects frames to include in the summary. DeepQAMVS is trained with reinforcement learning, incorporating rewards that capture representativeness, diversity, query-adaptability and temporal coherence. We achieve state-of-the-art results on the MVS1K dataset, with inference time scaling linearly with the number of input video frames.