# MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation

arXiv: https://arxiv.org/abs/2303.00180 · HTML: https://mlsanity.com/paper/2303.00180/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 1 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition](https://mlsanity.com/paper/2109.07270/) (2021)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [POSTER++: A simpler and stronger facial expression recognition network](https://mlsanity.com/paper/2301.12149/) (2023)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [Multi-View Dynamic Facial Action Unit Detection](https://mlsanity.com/paper/1704.07863/) (2017)
- [Audio-video Emotion Recognition in the Wild using Deep Hybrid Networks](https://mlsanity.com/paper/2002.09023/) (2020)

## Built on by (most cited first, 16 of 16)

- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [COVID-19 Computer-aided Diagnosis through AI-assisted CT Imaging Analysis: Deploying a Medical AI System](https://mlsanity.com/paper/2403.06242/) (2024)
- [BTDNet: a Multi-Modal Approach for Brain Tumor Radiogenomic Classification](https://mlsanity.com/paper/2310.03485/) (2023)
- [DVD: A Comprehensive Dataset for Advancing Violence Detection in Real-World Scenarios](https://mlsanity.com/paper/2506.05372/) (2025)
- [Ensuring UAV Safety: A Vision-only and Real-time Framework for Collision Avoidance Through Object Detection, Tracking, and Distance Estimation](https://mlsanity.com/paper/2405.06749/) (2024)
- [Can Machine Learning Assist in Diagnosis of Primary Immune Thrombocytopenia? A feasibility study](https://mlsanity.com/paper/2405.20562/) (2024)
- [Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers](https://mlsanity.com/paper/2303.09164/) (2023)
- [A Dual Branch Network for Emotional Reaction Intensity Estimation](https://mlsanity.com/paper/2303.09210/) (2023)
- [Integrating Holistic and Local Information to Estimate Emotional Reaction Intensity](https://mlsanity.com/paper/2305.05534/) (2023)
- [Common Corruptions for Enhancing and Evaluating Robustness in Air-to-Air Visual Object Detection](https://mlsanity.com/paper/2405.06765/) (2024)
- [Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC](https://mlsanity.com/paper/2508.06564/) (2025)
- [GANmut: Generating and Modifying Facial Expressions](https://mlsanity.com/paper/2406.11079/) (2024)
- [Multi-Task Learning for Affect Analysis](https://mlsanity.com/paper/2407.00679/) (2024)
- [Emotion Detection through Body Gesture and Face](https://mlsanity.com/paper/2407.09913/) (2024)

## Abstract (arXiv)

This paper presents MMA-MRNNet, a novel deep learning architecture for dynamic multi-output Facial Expression Intensity Estimation (FEIE) from video data. Traditional approaches to this task often rely on complex 3-D CNNs, which require extensive pre-training and assume that facial expressions are uniformly distributed across all frames of a video. These methods struggle to handle videos of varying lengths, often resorting to ad-hoc strategies that either discard valuable information or introduce bias. MMA-MRNNet addresses these challenges through a two-stage process. First, the Multiple Models of Affect (MMA) extractor component is a Multi-Task Learning CNN that concurrently estimates valence-arousal, recognizes basic facial expressions, and detects action units in each frame. These representations are then processed by a Masked RNN component, which captures temporal dependencies and dynamically updates weights according to the true length of the input video, ensuring that only the most relevant features are used for the final prediction. The proposed unimodal non-ensemble learning MMA-MRNNet was evaluated on the Hume-Reaction dataset and demonstrated significantly superior performance, surpassing state-of-the-art methods by a wide margin, regardless of whether they were unimodal, multimodal, or ensemble approaches. Finally, we demonstrated the effectiveness of the MMA component of our proposed method across multiple in-the-wild datasets, where it consistently outperformed all state-of-the-art methods across various metrics.