# Representation Learning and Identity Adversarial Training for Facial Behavior Understanding

arXiv: https://arxiv.org/abs/2407.11243 · HTML: https://mlsanity.com/paper/2407.11243/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 91 references that are in the map.
- First posted on arXiv 15 Jul 2024.

## Builds on (most cited first, 30 of 64)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)

## Built on by (most cited first, 14 of 14)

- [Ig3D: Integrating 3D Face Representations in Facial Expression Inference](https://mlsanity.com/paper/2408.16907/) (2024)
- [DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation](https://mlsanity.com/paper/2410.08470/) (2024)
- [Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age](https://mlsanity.com/paper/2410.24148/) (2024)
- [SynPAIN: A Synthetic Dataset of Pain and Non-Pain Facial Expressions](https://mlsanity.com/paper/2507.19673/) (2025)
- [Facial-R1: Aligning Reasoning and Recognition for Facial Emotion Analysis](https://mlsanity.com/paper/2511.10254/) (2025)
- [Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization](https://mlsanity.com/paper/2602.09121/) (2026)
- [Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model](https://mlsanity.com/paper/2411.10863/) (2024)
- [Mini-ResEmoteNet: Leveraging Knowledge Distillation for Human-Centered Design](https://mlsanity.com/paper/2501.18538/) (2025)
- [MAUGen: A Unified Diffusion Approach for Multi-Identity Facial Expression and AU Label Generation](https://mlsanity.com/paper/2602.00583/) (2026)
- [PriorProbe: Recovering Individual-Level Priors for Personalizing Neural Networks in Facial Expression Recognition](https://mlsanity.com/paper/2602.03882/) (2026)
- [Evaluating Test-Time Adaptation For Facial Expression Recognition Under Natural Cross-Dataset Distribution Shifts](https://mlsanity.com/paper/2603.19994/) (2026)
- [Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation](https://mlsanity.com/paper/2604.02162/) (2026)
- [Context Matters: Vision-Based Depression Detection Comparing Classical and Deep Approaches](https://mlsanity.com/paper/2604.10344/) (2026)
- [DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding](https://mlsanity.com/paper/2610.17015/) (2026)

## Abstract (arXiv)

Facial Action Unit (AU) detection has gained significant attention as it enables the breakdown of complex facial expressions into individual muscle movements. In this paper, we revisit two fundamental factors in AU detection: diverse and large-scale data and subject identity regularization. Motivated by recent advances in foundation models, we highlight the importance of data and introduce Face9M, a diverse dataset comprising 9 million facial images from multiple public sources. Pretraining a masked autoencoder on Face9M yields strong performance in AU detection and facial expression tasks. More importantly, we emphasize that the Identity Adversarial Training (IAT) has not been well explored in AU tasks. To fill this gap, we first show that subject identity in AU datasets creates shortcut learning for the model and leads to sub-optimal solutions to AU predictions. Secondly, we demonstrate that strong IAT regularization is necessary to learn identity-invariant features. Finally, we elucidate the design space of IAT and empirically show that IAT circumvents the identity-based shortcut learning and results in a better solution. Our proposed methods, Facial Masked Autoencoder (FMAE) and IAT, are simple, generic and effective. Remarkably, the proposed FMAE-IAT approach achieves new state-of-the-art F1 scores on BP4D (67.1%), BP4D+ (66.8%), and DISFA (70.1%) databases, significantly outperforming previous work. We release the code and model at https://github.com/forever208/FMAE-IAT.