# Emotion Recognition from the perspective of Activity Recognition

arXiv: https://arxiv.org/abs/2403.16263 · HTML: https://mlsanity.com/paper/2403.16263/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 74 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications](https://mlsanity.com/paper/1606.03237/) (2016)
- [Training Deep Neural Networks with Different Datasets In-the-wild: The Emotion Recognition Paradigm](https://mlsanity.com/paper/1809.04359/) (2018)
- [Comparison of Reinforcement Learning algorithms applied to the Cart Pole problem](https://mlsanity.com/paper/1810.01940/) (2018)
- [Learning Dynamics from Kinematics: Estimating 2D Foot Pressure Maps from Video Frames](https://mlsanity.com/paper/1811.12607/) (2018)
- [Emotion Recognition with Spatial Attention and Temporal Softmax Pooling](https://mlsanity.com/paper/1910.01254/) (2019)
- [PatchRefineNet: Improving Binary Segmentation by Incorporating Signals from Optimal Patch-wise Binarization](https://mlsanity.com/paper/2211.06560/) (2022)
- [Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregator](https://mlsanity.com/paper/1608.02833/) (2016)
- [Estimating Uncertainty in Landslide Segmentation Models](https://mlsanity.com/paper/2311.11138/) (2023)

## Built on by (most cited first, 2 of 2)

- [SAMIC: Segment Anything with In-Context Spatial Prompt Engineering](https://mlsanity.com/paper/2412.11998/) (2024)
- [SmartScan: An AI-based Interactive Framework for Automated Region Extraction from Satellite Images](https://mlsanity.com/paper/2504.00200/) (2025)

## Abstract (arXiv)

Applications of an efficient emotion recognition system can be found in several domains such as medicine, driver fatigue surveillance, social robotics, and human-computer interaction. Appraising human emotional states, behaviors, and reactions displayed in real-world settings can be accomplished using latent continuous dimensions. Continuous dimensional models of human affect, such as those based on valence and arousal are more accurate in describing a broad range of spontaneous everyday emotions than more traditional models of discrete stereotypical emotion categories (e.g. happiness, surprise). Most of the prior work on estimating valence and arousal considers laboratory settings and acted data. But, for emotion recognition systems to be deployed and integrated into real-world mobile and computing devices, we need to consider data collected in the world. Action recognition is a domain of Computer Vision that involves capturing complementary information on appearance from still frames and motion between frames. In this paper, we treat emotion recognition from the perspective of action recognition by exploring the application of deep learning architectures specifically designed for action recognition, for continuous affect recognition. We propose a novel three-stream end-to-end deep learning regression pipeline with an attention mechanism, which is an ensemble design based on sub-modules of multiple state-of-the-art action recognition systems. The pipeline constitutes a novel data pre-processing approach with a spatial self-attention mechanism to extract keyframes. The optical flow of high-attention regions of the face is extracted to capture temporal context. AFEW-VA in-the-wild dataset has been used to conduct comparative experiments. Quantitative analysis shows that the proposed model outperforms multiple standard baselines of both emotion recognition and action recognition models.