# AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition

arXiv: https://arxiv.org/abs/2012.14587 · HTML: https://mlsanity.com/paper/2012.14587/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation](https://mlsanity.com/paper/2309.13237/) (2023)
- [Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2205.13092/) (2022)
- [Privileged Attribution Constrained Deep Networks for Facial Expression Recognition](https://mlsanity.com/paper/2203.12905/) (2022)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels](https://mlsanity.com/paper/2412.10843/) (2024)
- [AU-Supervised Convolutional Vision Transformers for Synthetic Facial Expression Recognition](https://mlsanity.com/paper/2207.09777/) (2022)
- [Prior-based Objective Inference Mining Potential Uncertainty for Facial Expression Recognition](https://mlsanity.com/paper/2411.13024/) (2024)
- [Learning Semantic-Aware Threshold for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2507.23263/) (2025)

## Abstract (arXiv)

Recognizing human emotion/expressions automatically is quite an expected ability for intelligent robotics, as it can promote better communication and cooperation with humans. Current deep-learning-based algorithms may achieve impressive performance in some lab-controlled environments, but they always fail to recognize the expressions accurately for the uncontrolled in-the-wild situation. Fortunately, facial action units (AU) describe subtle facial behaviors, and they can help distinguish uncertain and ambiguous expressions. In this work, we explore the correlations among the action units and facial expressions, and devise an AU-Expression Knowledge Constrained Representation Learning (AUE-CRL) framework to learn the AU representations without AU annotations and adaptively use representations to facilitate facial expression recognition. Specifically, it leverages AU-expression correlations to guide the learning of the AU classifiers, and thus it can obtain AU representations without incurring any AU annotations. Then, it introduces a knowledge-guided attention mechanism that mines useful AU representations under the constraint of AU-expression correlations. In this way, the framework can capture local discriminative and complementary features to enhance facial representation for facial expression recognition. We conduct experiments on the challenging uncontrolled datasets to demonstrate the superiority of the proposed framework over current state-of-the-art methods. Codes and trained models are available at https://github.com/HCPLab-SYSU/AUE-CRL.