# MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization

arXiv: https://arxiv.org/abs/2204.08352 · HTML: https://mlsanity.com/paper/2204.08352/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [SumGraph: Video Summarization via Recursive Graph Modeling](https://mlsanity.com/paper/2007.08809/) (2020)
- [Exploring global diverse attention via pairwise temporal relation for video summarization](https://mlsanity.com/paper/2009.10942/) (2020)
- [GPT2MVS: Generative Pre-trained Transformer-2 for Multi-modal Video Summarization](https://mlsanity.com/paper/2104.12465/) (2021)
- [Summary Transfer: Exemplar-based Subset Selection for Video Summarization](https://mlsanity.com/paper/1603.03369/) (2016)
- [Cycle-SUM: Cycle-consistent Adversarial LSTM Networks for Unsupervised Video Summarization](https://mlsanity.com/paper/1904.08265/) (2019)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [Video Summarization using Deep Semantic Features](https://mlsanity.com/paper/1609.08758/) (2016)
- [Video Summarisation by Classification with Deep Reinforcement Learning](https://mlsanity.com/paper/1807.03089/) (2018)
- [Transforming Multi-Concept Attention into Video Summarization](https://mlsanity.com/paper/2006.01410/) (2020)
- [Diversity-aware Multi-Video Summarization](https://mlsanity.com/paper/1706.03123/) (2017)
- [EgoSampling: Fast-Forward and Stereo for Egocentric Videos](https://mlsanity.com/paper/1412.3596/) (2014)
- [Multi-View Surveillance Video Summarization via Joint Embedding and Sparse Optimization](https://mlsanity.com/paper/1706.03121/) (2017)
- [AudioVisual Video Summarization](https://mlsanity.com/paper/2105.07667/) (2021)

## Built on by (most cited first, 2 of 2)

- [QKSAN: A Quantum Kernel Self-Attention Network](https://mlsanity.com/paper/2308.13422/) (2023)
- [SITransformer: Shared Information-Guided Transformer for Extreme Multimodal Summarization](https://mlsanity.com/paper/2408.15829/) (2024)

## Abstract (arXiv)

Video summarization intends to produce a concise video summary by effectively capturing and combining the most informative parts of the whole content. Existing approaches for video summarization regard the task as a frame-wise keyframe selection problem and generally construct the frame-wise representation by combining the long-range temporal dependency with the unimodal or bimodal information. However, the optimal video summaries need to reflect the most valuable keyframe with its own information, and one with semantic power of the whole content. Thus, it is critical to construct a more powerful and robust frame-wise representation and predict the frame-level importance score in a fair and comprehensive manner. To tackle the above issues, we propose a multimodal hierarchical shot-aware convolutional network, denoted as MHSCNet, to enhance the frame-wise representation via combining the comprehensive available multimodal information. Specifically, we design a hierarchical ShotConv network to incorporate the adaptive shot-aware frame-level representation by considering the short-range and long-range temporal dependency. Based on the learned shot-aware representations, MHSCNet can predict the frame-level importance score in the local and global view of the video. Extensive experiments on two standard video summarization datasets demonstrate that our proposed method consistently outperforms state-of-the-art baselines. Source code will be made publicly available.