# Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

arXiv: https://arxiv.org/abs/1909.06957 · HTML: https://mlsanity.com/paper/1909.06957/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2019.

## Built on by (most cited first, 8 of 8)

- [Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition](https://mlsanity.com/paper/2104.06517/) (2021)
- [Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses](https://mlsanity.com/paper/2202.10453/) (2022)
- [An evaluation of LLMs for generating movie reviews: GPT-4o, Gemini-2.0 and DeepSeek-V3](https://mlsanity.com/paper/2506.00312/) (2025)
- [Recognizing Emotions evoked by Movies using Multitask Learning](https://mlsanity.com/paper/2107.14529/) (2021)
- [Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections](https://mlsanity.com/paper/2608.11576/) (2026)
- [Temporal Convolution Networks with Positional Encoding for Evoked Expression Estimation](https://mlsanity.com/paper/2106.08596/) (2021)
- [Enhancing the Prediction of Emotional Experience in Movies using Deep Neural Networks: The Significance of Audio and Language](https://mlsanity.com/paper/2306.10397/) (2023)
- [Emotion Manipulation Through Music -- A Deep Learning Interactive Visual Approach](https://mlsanity.com/paper/2406.08623/) (2024)

## Abstract (arXiv)

The goal of this study is to develop and analyze multimodal models for predicting experienced affective responses of viewers watching movie clips. We develop hybrid multimodal prediction models based on both the video and audio of the clips. For the video content, we hypothesize that both image content and motion are crucial features for evoked emotion prediction. To capture such information, we extract features from RGB frames and optical flow using pre-trained neural networks. For the audio model, we compute an enhanced set of low-level descriptors including intensity, loudness, cepstrum, linear predictor coefficients, pitch and voice quality. Both visual and audio features are then concatenated to create audio-visual features, which are used to predict the evoked emotion. To classify the movie clips into the corresponding affective response categories, we propose two approaches based on deep neural network models. The first one is based on fully connected layers without memory on the time component, the second incorporates the sequential dependency with a long short-term memory recurrent neural network (LSTM). We perform a thorough analysis of the importance of each feature set. Our experiments reveal that in our set-up, predicting emotions at each time step independently gives slightly better accuracy performance than with the LSTM. Interestingly, we also observe that the optical flow is more informative than the RGB in videos, and overall, models using audio features are more accurate than those based on video features when making the final prediction of evoked emotions.