# Use of Affective Visual Information for Summarization of Human-Centric Videos

arXiv: https://arxiv.org/abs/2107.03783 · HTML: https://mlsanity.com/paper/2107.03783/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Video Summarization Using Fully Convolutional Sequence Networks](https://mlsanity.com/paper/1805.10538/) (2018)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Rethinking the Evaluation of Video Summaries](https://mlsanity.com/paper/1903.11328/) (2019)
- [Discriminative Feature Learning for Unsupervised Video Summarization](https://mlsanity.com/paper/1811.09791/) (2018)
- [Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network](https://mlsanity.com/paper/1902.01019/) (2019)
- [Exploring global diverse attention via pairwise temporal relation for video summarization](https://mlsanity.com/paper/2009.10942/) (2020)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization](https://mlsanity.com/paper/1511.04798/) (2015)
- [End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning](https://mlsanity.com/paper/1802.01115/) (2018)
- [A Multi-task Neural Approach for Emotion Attribution, Classification and Summarization](https://mlsanity.com/paper/1812.09041/) (2018)
- [Adversarial-based neural networks for affect estimations in the wild](https://mlsanity.com/paper/2002.00883/) (2020)
- [Multimodal Continuous Emotion Recognition using Deep Multi-Task Learning with Correlation Loss](https://mlsanity.com/paper/2011.00876/) (2020)

## Built on by (most cited first, 3 of 3)

- [Learning Annotation Consensus for Continuous Emotion Recognition](https://mlsanity.com/paper/2505.21196/) (2025)
- [Video Summarization Techniques: A Comprehensive Review](https://mlsanity.com/paper/2410.04449/) (2024)
- [Role of Audio in Audio-Visual Video Summarization](https://mlsanity.com/paper/2212.01040/) (2022)

## Abstract (arXiv)

Increasing volume of user-generated human-centric video content and their applications, such as video retrieval and browsing, require compact representations that are addressed by the video summarization literature. Current supervised studies formulate video summarization as a sequence-to-sequence learning problem and the existing solutions often neglect the surge of human-centric view, which inherently contains affective content. In this study, we investigate the affective-information enriched supervised video summarization task for human-centric videos. First, we train a visual input-driven state-of-the-art continuous emotion recognition model (CER-NET) on the RECOLA dataset to estimate emotional attributes. Then, we integrate the estimated emotional attributes and the high-level representations from the CER-NET with the visual information to define the proposed affective video summarization architectures (AVSUM). In addition, we investigate the use of attention to improve the AVSUM architectures and propose two new architectures based on temporal attention (TA-AVSUM) and spatial attention (SA-AVSUM). We conduct video summarization experiments on the TvSum database. The proposed AVSUM-GRU architecture with an early fusion of high level GRU embeddings and the temporal attention based TA-AVSUM architecture attain competitive video summarization performances by bringing strong performance improvements for the human-centric videos compared to the state-of-the-art in terms of F-score and self-defined face recall metrics.