# AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels

arXiv: https://arxiv.org/abs/2410.22506 · HTML: https://mlsanity.com/paper/2410.22506/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 126 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Deep Self-Learning From Noisy Labels](https://mlsanity.com/paper/1908.02160/) (2019)
- [Delving Deep into Label Smoothing](https://mlsanity.com/paper/2011.12562/) (2020)
- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [From Facial Expression Recognition to Interpersonal Relation Prediction](https://mlsanity.com/paper/1609.06426/) (2016)
- [POSTER: A Pyramid Cross-Fusion Transformer Network for Facial Expression Recognition](https://mlsanity.com/paper/2204.04083/) (2022)
- [DAiSEE: Towards User Engagement Recognition in the Wild](https://mlsanity.com/paper/1609.01885/) (2016)
- [Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](https://mlsanity.com/paper/2212.05463/) (2022)
- [Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition](https://mlsanity.com/paper/2203.13052/) (2022)
- [BReG-NeXt: Facial Affect Computing Using Adaptive Residual Networks With Bounded Gradient](https://mlsanity.com/paper/2004.08495/) (2020)
- [Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher](https://mlsanity.com/paper/2205.14361/) (2022)
- [ASMNet: a Lightweight Deep Neural Network for Face Alignment and Pose Estimation](https://mlsanity.com/paper/2103.00119/) (2021)
- [Uncertain Facial Expression Recognition via Multi-task Assisted Correction](https://mlsanity.com/paper/2212.07144/) (2022)
- [THIN: THrowable Information Networks and Application for Facial Expression Recognition In The Wild](https://mlsanity.com/paper/2010.07614/) (2020)
- [Assessing Demographic Bias Transfer from Dataset to Model: A Case Study in Facial Expression Recognition](https://mlsanity.com/paper/2205.10049/) (2022)
- [Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition](https://mlsanity.com/paper/2204.11053/) (2022)
- [FEAFA+: An Extended Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animation](https://mlsanity.com/paper/2111.02751/) (2021)
- [ACR Loss: Adaptive Coordinate-based Regression Loss for Face Alignment](https://mlsanity.com/paper/2203.15835/) (2022)

## Built on by (most cited first, 9 of 9)

- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach](https://mlsanity.com/paper/2603.12848/) (2026)
- [Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.08566/) (2026)
- [SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs](https://mlsanity.com/paper/2606.05376/) (2026)
- [A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition](https://mlsanity.com/paper/2603.12221/) (2026)
- [EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning](https://mlsanity.com/paper/2511.20106/) (2025)
- [Real-Time Multi-Modal Embedded Vision Framework for Object Detection Facial Emotion Recognition and Biometric Identification on Low-Power Edge Platforms](https://mlsanity.com/paper/2601.11970/) (2026)
- [AffectFuse: Cross-Task Feature Fusion with Temporal Modeling for Multi-Task Affective Behavior Analysis](https://mlsanity.com/paper/2607.16546/) (2026)
- [Whose Ground Truth? Embracing Ambiguity in Human-Centered AI](https://mlsanity.com/paper/2610.10805/) (2026)

## Abstract (arXiv)

Automated Facial Expression Recognition (FER) is challenging due to intra-class variations and inter-class similarities. FER can be especially difficult when facial expressions reflect a mixture of various emotions (aka compound expressions). Existing FER datasets, such as AffectNet, provide discrete emotion labels (hard-labels), where a single category of emotion is assigned to an expression. To alleviate inter- and intra-class challenges, as well as provide a better facial expression descriptor, we propose a new approach to create FER datasets through a labeling method in which an image is labeled with more than one emotion (called soft-labels), each with different confidences. Specifically, we introduce the notion of soft-labels for facial expression datasets, a new approach to affective computing for more realistic recognition of facial expressions. To achieve this goal, we propose a novel methodology to accurately calculate soft-labels: a vector representing the extent to which multiple categories of emotion are simultaneously present within a single facial expression. Finding smoother decision boundaries, enabling multi-labeling, and mitigating bias and imbalanced data are some of the advantages of our proposed method. Building upon AffectNet, we introduce AffectNet+, the next-generation facial expression dataset. This dataset contains soft-labels, three categories of data complexity subsets, and additional metadata such as age, gender, ethnicity, head pose, facial landmarks, valence, and arousal. AffectNet+ will be made publicly accessible to researchers.