# Contrastive Learning of Person-independent Representations for Facial Action Unit Detection

arXiv: https://arxiv.org/abs/2403.03400 · HTML: https://mlsanity.com/paper/2403.03400/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 73 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning](https://mlsanity.com/paper/2008.00975/) (2020)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment](https://mlsanity.com/paper/1803.05588/) (2018)
- [Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/1904.09939/) (2019)
- [Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion](https://mlsanity.com/paper/2103.16854/) (2021)
- [Facial Action Unit Detection Using Attention and Relation Learning](https://mlsanity.com/paper/1808.03457/) (2018)
- [Self-supervised learning of a facial attribute embedding from video](https://mlsanity.com/paper/1808.06882/) (2018)
- [Deep Structure Inference Network for Facial Action Unit Recognition](https://mlsanity.com/paper/1803.05873/) (2018)
- [Cycle-Contrast for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2010.14810/) (2020)
- [Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing](https://mlsanity.com/paper/1704.03067/) (2017)
- [EAC-Net: A Region-based Deep Enhancing and Cropping Approach for Facial Action Unit Detection](https://mlsanity.com/paper/1702.02925/) (2017)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection](https://mlsanity.com/paper/1812.05788/) (2018)
- [RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations](https://mlsanity.com/paper/2008.05196/) (2020)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)

## Built on by (most cited first, 5 of 5)

- [Representation Learning and Identity Adversarial Training for Facial Behavior Understanding](https://mlsanity.com/paper/2407.11243/) (2024)
- [MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control](https://mlsanity.com/paper/2501.02260/) (2025)
- [Hierarchical Vision-Language Interaction for Facial Action Unit Detection](https://mlsanity.com/paper/2602.14425/) (2026)
- [Long-Tailed Object Detection Pre-training: Dynamic Rebalancing Contrastive Learning with Dual Reconstruction](https://mlsanity.com/paper/2411.09453/) (2024)
- [Micro-AU CLIP: Fine-Grained Contrastive Learning from Local Independence to Global Dependency for Micro-Expression Action Unit Detection](https://mlsanity.com/paper/2603.16302/) (2026)

## Abstract (arXiv)

Facial action unit (AU) detection, aiming to classify AU present in the facial image, has long suffered from insufficient AU annotations. In this paper, we aim to mitigate this data scarcity issue by learning AU representations from a large number of unlabelled facial videos in a contrastive learning paradigm. We formulate the self-supervised AU representation learning signals in two-fold: (1) AU representation should be frame-wisely discriminative within a short video clip; (2) Facial frames sampled from different identities but show analogous facial AUs should have consistent AU representations. As to achieve these goals, we propose to contrastively learn the AU representation within a video clip and devise a cross-identity reconstruction mechanism to learn the person-independent representations. Specially, we adopt a margin-based temporal contrastive learning paradigm to perceive the temporal AU coherence and evolution characteristics within a clip that consists of consecutive input facial frames. Moreover, the cross-identity reconstruction mechanism facilitates pushing the faces from different identities but show analogous AUs close in the latent embedding space. Experimental results on three public AU datasets demonstrate that the learned AU representation is discriminative for AU detection. Our method outperforms other contrastive learning methods and significantly closes the performance gap between the self-supervised and supervised AU detection approaches.