mlsanityData to 7 Oct 2026

ExpLLM: Towards Chain of Thought for Facial Expression Recognition

arXiv 2409.02828 ↗First posted 4 Sep 2024cs.CV cs.MMcited by 15

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Facial expression recognition (FER) is a critical task in multimedia with significant implications across various domains. However, analyzing the causes of facial expressions is essential for accurately recognizing them. Current approaches, such as those based on facial action units (AUs), typically provide AU names and intensities but lack insight into the interactions and relationships between AUs and the overall expression. In this paper, we propose a novel method called ExpLLM, which leverages large language models to generate an accurate chain of thought (CoT) for facial expression recognition. Specifically, we have designed the CoT mechanism from three key perspectives: key observations, overall emotional interpretation, and conclusion. The key observations describe the AU's name, intensity, and associated emotions. The overall emotional interpretation provides an analysis based on multiple AUs and their interactions, identifying the dominant emotions and their relationships. Finally, the conclusion presents the final expression label derived from the preceding analysis. Furthermore, we also introduce the Exp-CoT Engine, designed to construct this expression CoT and generate instruction-description data for training our ExpLLM. Extensive experiments on the RAF-DB and AffectNet datasets demonstrate that ExpLLM outperforms current state-of-the-art FER methods. ExpLLM also surpasses the latest GPT-4o in expression CoT generation, particularly in recognizing micro-expressions where GPT-4o frequently fails.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 84 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

54 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning10 Apr

Other references · 53

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  3. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  4. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  5. 2023Faithful Chain-of-Thought Reasoningcited by 285
  6. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  7. 2018Deep Facial Expression Recognition: A Surveycited by 269
  8. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  9. 2020Suppressing Uncertainties for Large-Scale Facial Expression Recognitioncited by 157
  10. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  11. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  12. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  13. 2021Sample and Computation Redistribution for Efficient Face Detectioncited by 86
  14. 2022Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognitioncited by 81
  15. 2022Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognitioncited by 67
  16. 2021Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognitioncited by 55
  17. 2022I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classificationcited by 54
  18. 2019Semantic Relationships Guided Representation Learning for Facial Action Unit Recognitioncited by 53
  19. 2021Facial Expression Recognition with Visual Transformers and Attentional Selective Fusioncited by 49
  20. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  21. 2020Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learningcited by 46
  22. 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognitioncited by 31
  23. 2023CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognitioncited by 27
  24. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  25. 2017EAC-Net: A Region-based Deep Enhancing and Cropping Approach for Facial Action Unit Detectioncited by 26
  26. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  27. 2015Learning Social Relation Traits from Face Imagescited by 25
  28. 2024Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfercited by 18
  29. 2023LA-Net: Landmark-Aware Learning for Reliable Facial Expression Recognition under Label Noisecited by 17
  30. 2023Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognitioncited by 17
  31. 2024Self-Supervised Facial Representation Learning with Facial Region Awarenesscited by 17
  32. 2024LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Modelcited by 17
  33. 2022Pose-disentangled Contrastive Learning for Self-supervised Facial Representationcited by 16
  34. 2023Weakly-Supervised Text-driven Contrastive Learning for Facial Behavior Understandingcited by 16
  35. 2023Latent-OFER: Detect, Mask, and Reconstruct with Latent Vectors for Occluded Facial Expression Recognitioncited by 13
  36. 2023FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Featurescited by 12
  37. 2022Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detectioncited by 11
  38. 2020THIN: THrowable Information Networks and Application for Facial Expression Recognition In The Wildcited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  2. 2025FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoningcited by 5
  3. 2026AULLM++: Structured-Token-Conditioned Large Language Models for Micro-Expression Action Unit Detectioncited by 4
  4. 2025CausalAffect: Causal Discovery for Facial Affective Understandingcited by 3
  5. 2024Towards Unified Facial Action Unit Recognition Framework by Large Language Modelscited by 2
  6. 2025MELLM: A Flow-Guided Large Language Model for Micro-Expression Understandingcited by 2
  7. 2026Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extractioncited by 2
  8. 2025From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answeringcited by 1
  9. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?cited by 1
  10. 2024MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesiscited by 0
  11. 2025Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Modelscited by 0
  12. 2025Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captionscited by 0
3 more
  1. 2025Point-Supervised Facial Expression Spotting with Gaussian-Based Instance-Adaptive Intensity Modelingcited by 0
  2. 2026LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognitioncited by 0
  3. 2026AUCH-Net: Action Unit-Based Consistency-Aware Hypergraph Network for Cross-Domain Few-Shot Facial Expression Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis
  2. 2026TAG: Thinking with Action Unit Grounding for Facial Expression Recognition
  3. 2025Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond
  4. 2025Compound Expression Recognition via Large Vision-Language Models
  5. 2025Facial Emotion Learning with Text-Guided Multiview Fusion via Vision-Language Model for 3D/4D Facial Expression Recognition
  6. 2024Zero-shot Compound Expression Recognition with Visual Language Model at the 6th ABAW Challenge
  7. 2025AU-LLM: Micro-Expression Action Unit Detection via Enhanced LLM-Based Feature Fusion
  8. 2025Self-Supervised Multi-View Representation Learning using Vision-Language Model for 3D/4D Facial Expression Recognition