# Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications

arXiv: https://arxiv.org/abs/1606.03237 · HTML: https://mlsanity.com/paper/1606.03237/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2016.

## Built on by (most cited first, 30 of 45)

- [Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms](https://mlsanity.com/paper/2201.12728/) (2022)
- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [Micron-BERT: BERT-based Facial Micro-Expression Recognition](https://mlsanity.com/paper/2304.03195/) (2023)
- [Global-Local Attention for Emotion Recognition](https://mlsanity.com/paper/2111.04129/) (2021)
- [Neuromorphic Event-based Facial Expression Recognition](https://mlsanity.com/paper/2304.06351/) (2023)
- [Graph-based Facial Affect Analysis: A Review](https://mlsanity.com/paper/2103.15599/) (2021)
- [Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition](https://mlsanity.com/paper/2106.03487/) (2021)
- [Towards Unbiased Visual Emotion Recognition via Causal Intervention](https://mlsanity.com/paper/2107.12096/) (2021)
- [CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routing](https://mlsanity.com/paper/2101.03503/) (2021)
- [A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions](https://mlsanity.com/paper/2305.07665/) (2023)
- [3D Face Reconstruction: the Road to Forensics](https://mlsanity.com/paper/2309.11357/) (2023)
- [Neuromorphic Facial Analysis with Cross-Modal Supervision](https://mlsanity.com/paper/2409.10213/) (2024)
- [An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis](https://mlsanity.com/paper/2203.15829/) (2022)
- [DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization](https://mlsanity.com/paper/2103.02484/) (2021)
- [When Facial Expression Recognition Meets Few-Shot Learning: A Joint and Alternate Learning Framework](https://mlsanity.com/paper/2201.06781/) (2022)
- [Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer](https://mlsanity.com/paper/2404.17205/) (2024)
- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)
- [Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation](https://mlsanity.com/paper/2106.13393/) (2021)
- [Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects](https://mlsanity.com/paper/2106.14174/) (2021)
- [Critically examining the Domain Generalizability of Facial Expression Recognition models](https://mlsanity.com/paper/2106.15453/) (2021)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Multi-Scale Spatio-Temporal Graph Convolutional Network for Facial Expression Spotting](https://mlsanity.com/paper/2403.15994/) (2024)
- [HDRT: A Large-Scale Dataset for Infrared-Guided HDR Imaging](https://mlsanity.com/paper/2406.05475/) (2024)
- [Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network](https://mlsanity.com/paper/2203.14045/) (2022)
- [Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coach](https://mlsanity.com/paper/2311.05567/) (2023)
- [SpotFormer: Multi-Scale Spatio-Temporal Transformer for Facial Expression Spotting](https://mlsanity.com/paper/2407.20799/) (2024)
- [Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition](https://mlsanity.com/paper/2107.11061/) (2021)
- [VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition](https://mlsanity.com/paper/2208.11450/) (2022)
- [Emotion Recognition from the perspective of Activity Recognition](https://mlsanity.com/paper/2403.16263/) (2024)
- [Spatio-temporal Transformers for Action Unit Classification with Event Cameras](https://mlsanity.com/paper/2410.21958/) (2024)

## Abstract (arXiv)

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.