# MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

arXiv: https://arxiv.org/abs/2208.00847 · HTML: https://mlsanity.com/paper/2208.00847/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild](https://mlsanity.com/paper/2008.05924/) (2020)
- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Frame attention networks for facial expression recognition in videos](https://mlsanity.com/paper/1907.00193/) (2019)
- [Emotion Recognition in Speech using Cross-Modal Transfer in the Wild](https://mlsanity.com/paper/1808.05561/) (2018)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [EMOPAIN Challenge 2020: Multimodal Pain Evaluation from Facial and Bodily Expressions](https://mlsanity.com/paper/2001.07739/) (2020)
- [Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction](https://mlsanity.com/paper/1901.01805/) (2019)

## Built on by (most cited first, 30 of 65)

- [R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning](https://mlsanity.com/paper/2503.05379/) (2025)
- [HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding](https://mlsanity.com/paper/2501.15111/) (2025)
- [GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning](https://mlsanity.com/paper/2506.16141/) (2025)
- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models](https://mlsanity.com/paper/2501.16566/) (2025)
- [MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2307.02227/) (2023)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition](https://mlsanity.com/paper/2310.16640/) (2023)
- [HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2401.05698/) (2024)
- [FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs](https://mlsanity.com/paper/2407.02157/) (2024)
- [EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning](https://mlsanity.com/paper/2408.11424/) (2024)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1](https://mlsanity.com/paper/2503.24376/) (2025)
- [SVFAP: Self-supervised Video Facial Affect Perceiver](https://mlsanity.com/paper/2401.00416/) (2023)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models](https://mlsanity.com/paper/2508.09210/) (2025)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild](https://mlsanity.com/paper/2404.09010/) (2024)
- [Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning](https://mlsanity.com/paper/2504.07198/) (2025)
- [Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding](https://mlsanity.com/paper/2505.06685/) (2025)
- [Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data](https://mlsanity.com/paper/2409.06154/) (2024)
- [AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels](https://mlsanity.com/paper/2410.22506/) (2024)
- [Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion](https://mlsanity.com/paper/2502.07203/) (2025)
- [Noise-Resistant Multimodal Transformer for Emotion Recognition](https://mlsanity.com/paper/2305.02814/) (2023)
- [Ig3D: Integrating 3D Face Representations in Facial Expression Inference](https://mlsanity.com/paper/2408.16907/) (2024)
- [FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO](https://mlsanity.com/paper/2503.09158/) (2025)
- [OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing](https://mlsanity.com/paper/2606.15920/) (2026)
- [All rivers run into the sea: Unified Modality Brain-like Emotional Central Mechanism](https://mlsanity.com/paper/2407.15590/) (2024)
- [VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection](https://mlsanity.com/paper/2505.02331/) (2025)
- [Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding](https://mlsanity.com/paper/2601.16449/) (2026)

## Abstract (arXiv)

Dynamic facial expression recognition (FER) databases provide important data support for affective computing and applications. However, most FER databases are annotated with several basic mutually exclusive emotional categories and contain only one modality, e.g., videos. The monotonous labels and modality cannot accurately imitate human emotions and fulfill applications in the real world. In this paper, we propose MAFW, a large-scale multi-modal compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment. To ensure high quality of the labels, we filter out the unreliable annotations by an Expectation Maximization (EM) algorithm, and then obtain 11 single-label emotion categories and 32 multi-label emotion categories. To the best of our knowledge, MAFW is the first in-the-wild multi-modal database annotated with compound emotion annotations and emotion-related captions. Additionally, we also propose a novel Transformer-based expression snippet feature learning method to recognize the compound emotions leveraging the expression-change relations among different emotions and modalities. Extensive experiments on MAFW database show the advantages of the proposed method over other state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database is publicly available from https://mafw-database.github.io/MAFW.