# Pre-training strategies and datasets for facial representation learning

arXiv: https://arxiv.org/abs/2103.16554 · HTML: https://mlsanity.com/paper/2103.16554/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 96 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)

## Built on by (most cited first, 26 of 26)

- [General Facial Representation Learning in a Visual-Linguistic Manner](https://mlsanity.com/paper/2112.03109/) (2021)
- [MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://mlsanity.com/paper/2211.06627/) (2022)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [SVFAP: Self-supervised Video Facial Affect Perceiver](https://mlsanity.com/paper/2401.00416/) (2023)
- [Self-Supervised Facial Representation Learning with Facial Region Awareness](https://mlsanity.com/paper/2403.02138/) (2024)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [Representation Learning and Identity Adversarial Training for Facial Behavior Understanding](https://mlsanity.com/paper/2407.11243/) (2024)
- [LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition](https://mlsanity.com/paper/2403.08161/) (2024)
- [Faceptor: A Generalist Model for Face Perception](https://mlsanity.com/paper/2403.09500/) (2024)
- [Toward High Quality Facial Representation Learning](https://mlsanity.com/paper/2309.03575/) (2023)
- [Learning from Label Relationships in Human Affect](https://mlsanity.com/paper/2207.05577/) (2022)
- [Deep Recurrent Semi-Supervised EEG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2107.13505/) (2021)
- [CAGE: Circumplex Affect Guided Expression Inference](https://mlsanity.com/paper/2404.14975/) (2024)
- [Ig3D: Integrating 3D Face Representations in Facial Expression Inference](https://mlsanity.com/paper/2408.16907/) (2024)
- [Unsupervised Facial Expression Representation Learning with Contrastive Local Warping](https://mlsanity.com/paper/2303.09034/) (2023)
- [ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation](https://mlsanity.com/paper/2311.01929/) (2023)
- [FaceInsight: A Multimodal Large Language Model for Face Perception](https://mlsanity.com/paper/2504.15624/) (2025)
- [SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video](https://mlsanity.com/paper/2210.11341/) (2022)
- [ComFace: Facial Representation Learning with Synthetic Data for Comparing Faces](https://mlsanity.com/paper/2405.16016/) (2024)
- [PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training](https://mlsanity.com/paper/2508.09691/) (2025)
- [Individualized Deepfake Detection Exploiting Traces Due to Double Neural-Network Operations](https://mlsanity.com/paper/2312.08034/) (2023)
- [OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance](https://mlsanity.com/paper/2405.14709/) (2024)
- [U-FaceBP: Uncertainty-aware Bayesian Ensemble Deep Learning for Face Video-based Blood Pressure Estimation](https://mlsanity.com/paper/2412.10679/) (2024)
- [Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition](https://mlsanity.com/paper/2412.13541/) (2024)
- [Improving Generalization in Deepfake Detection with Face Foundation Models and Metric Learning](https://mlsanity.com/paper/2508.19730/) (2025)
- [When, Where, and Why: Hierarchical Causal Explainability for Video Face Understanding with MLLMs](https://mlsanity.com/paper/2610.17791/) (2026)

## Abstract (arXiv)

What is the best way to learn a universal face representation? Recent work on Deep Learning in the area of face analysis has focused on supervised learning for specific tasks of interest (e.g. face recognition, facial landmark localization etc.) but has overlooked the overarching question of how to find a facial representation that can be readily adapted to several facial analysis tasks and datasets. To this end, we make the following 4 contributions: (a) we introduce, for the first time, a comprehensive evaluation benchmark for facial representation learning consisting of 5 important face analysis tasks. (b) We systematically investigate two ways of large-scale representation learning applied to faces: supervised and unsupervised pre-training. Importantly, we focus our evaluations on the case of few-shot facial learning. (c) We investigate important properties of the training datasets including their size and quality (labelled, unlabelled or even uncurated). (d) To draw our conclusions, we conducted a very large number of experiments. Our main two findings are: (1) Unsupervised pre-training on completely in-the-wild, uncurated data provides consistent and, in some cases, significant accuracy improvements for all facial tasks considered. (2) Many existing facial video datasets seem to have a large amount of redundancy. We will release code, and pre-trained models to facilitate future research.