# Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality

arXiv: https://arxiv.org/abs/2103.06541 · HTML: https://mlsanity.com/paper/2103.06541/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 87 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [Automatic Understanding of Image and Video Advertisements](https://mlsanity.com/paper/1707.03067/) (2017)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Dynamic Coattention Networks For Question Answering](https://mlsanity.com/paper/1611.01604/) (2016)
- [Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias](https://mlsanity.com/paper/1808.02212/) (2018)
- [The OMG-Emotion Behavior Dataset](https://mlsanity.com/paper/1803.05434/) (2018)
- [DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1410.8586/) (2014)
- [ADVISE: Symbolism and External Knowledge for Decoding Advertisements](https://mlsanity.com/paper/1711.06666/) (2017)
- [Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset](https://mlsanity.com/paper/1912.05008/) (2019)
- [Neural Granger Causality](https://mlsanity.com/paper/1802.05842/) (2018)
- [Scalable Matrix-valued Kernel Learning for High-dimensional Nonlinear Multivariate Regression and Granger Causality](https://mlsanity.com/paper/1408.2066/) (2014)
- [Multimodal Content Analysis for Effective Advertisements on YouTube](https://mlsanity.com/paper/1709.03946/) (2017)
- [GLA in MediaEval 2018 Emotional Impact of Movies Task](https://mlsanity.com/paper/1911.12361/) (2019)
- [High-Level Concepts for Affective Understanding of Images](https://mlsanity.com/paper/1705.02751/) (2017)
- [Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning](https://mlsanity.com/paper/1912.10248/) (2019)

## Built on by (most cited first, 17 of 17)

- [Learning Emotion Representations from Verbal and Nonverbal Communication](https://mlsanity.com/paper/2305.13500/) (2023)
- [ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture](https://mlsanity.com/paper/2211.10780/) (2022)
- [Dual-path Collaborative Generation Network for Emotional Video Captioning](https://mlsanity.com/paper/2408.03006/) (2024)
- [Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos](https://mlsanity.com/paper/2208.01954/) (2022)
- [The Invariant Ground Truth of Affect](https://mlsanity.com/paper/2210.07630/) (2022)
- [GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion Recognition](https://mlsanity.com/paper/2210.15834/) (2022)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer](https://mlsanity.com/paper/2404.17205/) (2024)
- [Benchmarking VLMs' Reasoning About Persuasive Atypical Images](https://mlsanity.com/paper/2409.10719/) (2024)
- [A Review and Roadmap of Deep Learning Causal Discovery in Different Variable Paradigms](https://mlsanity.com/paper/2209.06367/) (2022)
- [JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models](https://mlsanity.com/paper/2403.04798/) (2024)
- [Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation](https://mlsanity.com/paper/2302.01555/) (2023)
- [Audio-Guided Fusion Techniques for Multimodal Emotion Analysis](https://mlsanity.com/paper/2409.05007/) (2024)
- [Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM](https://mlsanity.com/paper/2409.09362/) (2024)
- [Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning](https://mlsanity.com/paper/2607.29045/) (2026)
- [Temporal Convolution Networks with Positional Encoding for Evoked Expression Estimation](https://mlsanity.com/paper/2106.08596/) (2021)
- [Learning Subjective Time-Series Data via Utopia Label Distribution Approximation](https://mlsanity.com/paper/2307.07682/) (2023)

## Abstract (arXiv)

We present Affect2MM, a learning method for time-series emotion prediction for multimedia content. Our goal is to automatically capture the varying emotions depicted by characters in real-life human-centric situations and behaviors. We use the ideas from emotion causation theories to computationally model and determine the emotional state evoked in clips of movies. Affect2MM explicitly models the temporal causality using attention-based methods and Granger causality. We use a variety of components like facial features of actors involved, scene understanding, visual aesthetics, action/situation description, and movie script to obtain an affective-rich representation to understand and perceive the scene. We use an LSTM-based learning model for emotion perception. To evaluate our method, we analyze and compare our performance on three datasets, SENDv1, MovieGraphs, and the LIRIS-ACCEDE dataset, and observe an average of 10-15% increase in the performance over SOTA methods for all three datasets.