mlsanityData to 7 Oct 2026

Towards Unified Facial Action Unit Recognition Framework by Large Language Models

arXiv 2409.08444 ↗First posted 13 Sep 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Facial Action Units (AUs) are of great significance in the realm of affective computing. In this paper, we propose AU-LLaVA, the first unified AU recognition framework based on the Large Language Model (LLM). AU-LLaVA consists of a visual encoder, a linear projector layer, and a pre-trained LLM. We meticulously craft the text descriptions and fine-tune the model on various AU datasets, allowing it to generate different formats of AU recognition results for the same input image. On the BP4D and DISFA datasets, AU-LLaVA delivers the most accurate recognition results for nearly half of the AUs. Our model achieves improvements of F1-score up to 11.4% in specific AU recognition compared to previous benchmark results. On the FEAFA dataset, our method achieves significant improvements over all 24 AUs compared to previous benchmark results. AU-LLaVA demonstrates exceptional performance and versatility in AU recognition.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

24 of 41 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2019Multi-Label Image Recognition with Graph Convolutional Networkscited by 320
  11. 2021Sample and Computation Redistribution for Efficient Face Detectioncited by 86
  12. 2018Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignmentcited by 55
12 more
  1. 2019Semantic Relationships Guided Representation Learning for Facial Action Unit Recognitioncited by 53
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  3. 2018Deep Structure Inference Network for Facial Action Unit Recognitioncited by 38
  4. 2024AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectorscited by 29
  5. 2017Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusingcited by 28
  6. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 27
  7. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  8. 2024Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfercited by 18
  9. 2024LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Modelcited by 17
  10. 2024ExpLLM: Towards Chain of Thought for Facial Expression Recognitioncited by 15
  11. 2019FEAFA: A Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animationcited by 7
  12. 2021FEAFA+: An Extended Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuningcited by 9
  2. 2026Facial-Expression-Aware Prompting for Empathetic LLM Tutoringcited by 0