# AttendAffectNet: Self-Attention based Networks for Predicting Affective Responses from Movies

arXiv: https://arxiv.org/abs/2010.11188 · HTML: https://mlsanity.com/paper/2010.11188/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges](https://mlsanity.com/paper/2406.08809/) (2024)
- [Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses](https://mlsanity.com/paper/2202.10453/) (2022)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Temporal Convolution Networks with Positional Encoding for Evoked Expression Estimation](https://mlsanity.com/paper/2106.08596/) (2021)

## Abstract (arXiv)

In this work, we propose different variants of the self-attention based network for emotion prediction from movies, which we call AttendAffectNet. We take both audio and video into account and incorporate the relation among multiple modalities by applying self-attention mechanism in a novel manner into the extracted features for emotion prediction. We compare it to the typically temporal integration of the self-attention based model, which in our case, allows to capture the relation of temporal representations of the movie while considering the sequential dependencies of emotion responses. We demonstrate the effectiveness of our proposed architectures on the extended COGNIMUSE dataset [1], [2] and the MediaEval 2016 Emotional Impact of Movies Task [3], which consist of movies with emotion annotations. Our results show that applying the self-attention mechanism on the different audio-visual features, rather than in the time domain, is more effective for emotion prediction. Our approach is also proven to outperform many state-ofthe-art models for emotion prediction. The code to reproduce our results with the models' implementation is available at: https://github.com/ivyha010/AttendAffectNet.