# Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition

arXiv: https://arxiv.org/abs/2310.19636 · HTML: https://mlsanity.com/paper/2310.19636/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [Remix: Rebalanced Mixup](https://mlsanity.com/paper/2007.03943/) (2020)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [Imbalance Problems in Object Detection: A Review](https://mlsanity.com/paper/1909.00169/) (2019)
- [Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.05830/) (2015)
- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [TransFER: Learning Relation-aware Facial Expression Representations with Transformers](https://mlsanity.com/paper/2108.11116/) (2021)
- [Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition](https://mlsanity.com/paper/2207.10299/) (2022)
- [Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition](https://mlsanity.com/paper/2104.05160/) (2021)
- [Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin](https://mlsanity.com/paper/2203.12341/) (2022)
- [MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition](https://mlsanity.com/paper/2205.04442/) (2022)
- [Uncertainty-aware Label Distribution Learning for Facial Expression Recognition](https://mlsanity.com/paper/2209.10448/) (2022)

## Built on by (most cited first, 17 of 17)

- [EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning](https://mlsanity.com/paper/2408.11424/) (2024)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [ExpLLM: Towards Chain of Thought for Facial Expression Recognition](https://mlsanity.com/paper/2409.02828/) (2024)
- [From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech](https://mlsanity.com/paper/2503.16956/) (2025)
- [Exploring Facial Expression Recognition through Semi-Supervised Pretraining and Temporal Modeling](https://mlsanity.com/paper/2403.11942/) (2024)
- [SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data](https://mlsanity.com/paper/2410.09865/) (2024)
- [Robust Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2502.16129/) (2025)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)
- [DIG-FACE: De-biased Learning for Generalized Facial Expression Category Discovery](https://mlsanity.com/paper/2409.20098/) (2024)
- [V-NAW: Video-based Noise-aware Adaptive Weighting for Facial Expression Recognition](https://mlsanity.com/paper/2503.15970/) (2025)
- [Personalizing Federated Learning for Hierarchical Edge Networks with Non-IID Data](https://mlsanity.com/paper/2504.08872/) (2025)
- [Instruction-Driven 3D Facial Expression Generation and Transition](https://mlsanity.com/paper/2601.08179/) (2026)
- [Learning with Geometry: Including Riemannian Geometric Features in Coefficient of Pressure Prediction on Aircraft Wings](https://mlsanity.com/paper/2401.09452/) (2023)
- [Self-Supervised Learning with Probabilistic Density Labeling for Rainfall Probability Estimation](https://mlsanity.com/paper/2412.05825/) (2024)
- [A Visual Self-attention Mechanism Facial Expression Recognition Network beyond Convnext](https://mlsanity.com/paper/2504.09077/) (2025)
- [Face anonymization preserving facial expressions and photometric realism](https://mlsanity.com/paper/2603.17567/) (2026)
- [On Applicability of Synthetic Datasets for Facial Expression Recognition](https://mlsanity.com/paper/2605.17483/) (2026)

## Abstract (arXiv)

Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at https://github.com/zyh-uaiaaaa.