# EmoNets: Multimodal deep learning approaches for emotion recognition in video

arXiv: https://arxiv.org/abs/1503.01800 · HTML: https://mlsanity.com/paper/1503.01800/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2015.

## Built on by (most cited first, 20 of 20)

- [MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning](https://mlsanity.com/paper/2304.08981/) (2023)
- [MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records](https://mlsanity.com/paper/2102.02340/) (2021)
- [MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition](https://mlsanity.com/paper/2404.17113/) (2024)
- [A Survey on Integrated Sensing, Communication, and Computation](https://mlsanity.com/paper/2408.08074/) (2024)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [Identity-aware Facial Expression Recognition in Compressed Video](https://mlsanity.com/paper/2101.00317/) (2021)
- [Conversational Agents: Theory and Applications](https://mlsanity.com/paper/2202.03164/) (2022)
- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approach](https://mlsanity.com/paper/2507.08217/) (2025)
- [DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning](https://mlsanity.com/paper/2109.12065/) (2021)
- [Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers](https://mlsanity.com/paper/2203.11441/) (2022)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)
- [Which CNNs and Training Settings to Choose for Action Unit Detection? A Study Based on a Large-Scale Dataset](https://mlsanity.com/paper/2111.08320/) (2021)
- [Choose Settings Carefully: Comparing Action Unit detection at Different Settings Using a Large-Scale Dataset](https://mlsanity.com/paper/2111.08324/) (2021)
- [Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition](https://mlsanity.com/paper/2408.09438/) (2024)
- [DouFu: A Double Fusion Joint Learning Method For Driving Trajectory Representation](https://mlsanity.com/paper/2205.08356/) (2022)
- [Effective Multimodal Reinforcement Learning with Modality Alignment and Importance Enhancement](https://mlsanity.com/paper/2302.09318/) (2023)
- [Speech-Based Depression Prediction Using Encoder-Weight-Only Transfer Learning and a Large Corpus](https://mlsanity.com/paper/2412.16900/) (2024)
- [Analysis of the use of color and its emotional relationship in visual creations based on experiences during the context of the COVID-19 pandemic](https://mlsanity.com/paper/2203.13770/) (2022)

## Abstract (arXiv)

The task of the emotion recognition in the wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convolutional neural network, focusing on capturing visual information in detected faces, a deep belief net focusing on the representation of the audio stream, a K-Means based "bag-of-mouths" model, which extracts visual features around the mouth region and a relational autoencoder, which addresses spatio-temporal aspects of videos. We explore multiple methods for the combination of cues from these modalities into one common classifier. This achieves a considerably greater accuracy than predictions from our strongest single-modality classifier. Our method was the winning submission in the 2013 EmotiW challenge and achieved a test set accuracy of 47.67% on the 2014 dataset.