# Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention

arXiv: https://arxiv.org/abs/2207.08352 · HTML: https://mlsanity.com/paper/2207.08352/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Less is More: Learning Highlight Detection from Video Duration](https://mlsanity.com/paper/1903.00859/) (2019)
- [Hierarchical Recurrent Neural Network for Video Summarization](https://mlsanity.com/paper/1904.12251/) (2019)
- [Video2GIF: Automatic Generation of Animated GIFs from Video](https://mlsanity.com/paper/1605.04850/) (2016)
- [Video Summarization by Learning from Unpaired Data](https://mlsanity.com/paper/1805.12174/) (2018)
- [Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders](https://mlsanity.com/paper/1510.01442/) (2015)
- [Query-Focused Extractive Video Summarization](https://mlsanity.com/paper/1607.05177/) (2016)
- [Query-adaptive Video Summarization via Quality-aware Relevance Estimation](https://mlsanity.com/paper/1705.00581/) (2017)
- [Adaptive Video Highlight Detection by Learning from User History](https://mlsanity.com/paper/2007.09598/) (2020)
- [PHD-GIFs: Personalized Highlight Detection for Automatic GIF Creation](https://mlsanity.com/paper/1804.06604/) (2018)
- [Collaborative Summarization of Topic-Related Videos](https://mlsanity.com/paper/1706.03114/) (2017)
- [A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Video](https://mlsanity.com/paper/1801.10312/) (2018)
- [Hybrid-Attention based Decoupled Metric Learning for Zero-Shot Image Retrieval](https://mlsanity.com/paper/1907.11832/) (2019)
- [Noisy Submodular Maximization via Adaptive Sampling with Applications to Crowdsourced Image Collection Summarization](https://mlsanity.com/paper/1511.07211/) (2015)
- [PR-Net: Preference Reasoning for Personalized Video Highlight Detection](https://mlsanity.com/paper/2109.01799/) (2021)
- [From Thumbnails to Summaries - A single Deep Neural Network to Rule Them All](https://mlsanity.com/paper/1808.00184/) (2018)
- [HighlightMe: Detecting Highlights from Human-Centric Videos](https://mlsanity.com/paper/2110.01774/) (2021)

## Built on by (most cited first, 2 of 2)

- [ContentCTR: Frame-level Live Streaming Click-Through Rate Prediction with Multimodal Transformer](https://mlsanity.com/paper/2306.14392/) (2023)
- [A Multimodal Transformer for Live Streaming Highlight Prediction](https://mlsanity.com/paper/2407.12002/) (2024)

## Abstract (arXiv)

We propose a method to detect individualized highlights for users on given target videos based on their preferred highlight clips marked on previous videos they have watched. Our method explicitly leverages the contents of both the preferred clips and the target videos using pre-trained features for the objects and the human activities. We design a multi-head attention mechanism to adaptively weigh the preferred clips based on their object- and human-activity-based contents, and fuse them using these weights into a single feature representation for each user. We compute similarities between these per-user feature representations and the per-frame features computed from the desired target videos to estimate the user-specific highlight clips from the target videos. We test our method on a large-scale highlight detection dataset containing the annotated highlights of individual users. Compared to current baselines, we observe an absolute improvement of 2-4% in the mean average precision of the detected highlights. We also perform extensive ablation experiments on the number of preferred highlight clips associated with each user as well as on the object- and human-activity-based feature representations to validate that our method is indeed both content-based and user-specific.