# Self-supervised learning of a facial attribute embedding from video

arXiv: https://arxiv.org/abs/1808.06882 · HTML: https://mlsanity.com/paper/1808.06882/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2018.

## Built on by (most cited first, 30 of 45)

- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control](https://mlsanity.com/paper/2104.11224/) (2021)
- [Video Autoencoder: self-supervised disentanglement of static 3D structure and motion](https://mlsanity.com/paper/2110.02951/) (2021)
- [HiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extraction](https://mlsanity.com/paper/2205.02225/) (2022)
- [Pre-training strategies and datasets for facial representation learning](https://mlsanity.com/paper/2103.16554/) (2021)
- [Unsupervised Human Pose Estimation through Transforming Shape Templates](https://mlsanity.com/paper/2105.04154/) (2021)
- [Multi-task head pose estimation in-the-wild](https://mlsanity.com/paper/2202.02299/) (2022)
- [Learning Facial Representations from the Cycle-consistency of Face](https://mlsanity.com/paper/2108.03427/) (2021)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [SVFAP: Self-supervised Video Facial Affect Perceiver](https://mlsanity.com/paper/2401.00416/) (2023)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [Transformer-Based Self-Supervised Learning for Emotion Recognition](https://mlsanity.com/paper/2204.05103/) (2022)
- [Self-supervised Facial Action Unit Detection with Region and Relation Learning](https://mlsanity.com/paper/2303.05708/) (2023)
- [MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Pose](https://mlsanity.com/paper/2210.07181/) (2022)
- [Speaker Recognition in Realistic Scenario Using Multimodal Data](https://mlsanity.com/paper/2302.13033/) (2023)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)
- [Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images](https://mlsanity.com/paper/2111.04237/) (2021)
- [Invisible Backdoor Attacks Using Data Poisoning in the Frequency Domain](https://mlsanity.com/paper/2207.04209/) (2022)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [Task-adaptive Q-Face](https://mlsanity.com/paper/2405.09059/) (2024)
- [Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2412.09784/) (2024)
- [Landmarks Augmentation with Manifold-Barycentric Oversampling](https://mlsanity.com/paper/2104.00925/) (2021)
- [Weakly Supervised Regional and Temporal Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2204.00379/) (2022)
- [LEAD: Self-Supervised Landmark Estimation by Aligning Distributions of Feature Similarity](https://mlsanity.com/paper/2204.02958/) (2022)
- [ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation](https://mlsanity.com/paper/2311.01929/) (2023)
- [Pose-Guided Self-Training with Two-Stage Clustering for Unsupervised Landmark Discovery](https://mlsanity.com/paper/2403.16194/) (2024)
- [Facial Anatomical Landmark Detection using Regularized Transfer Learning with Application to Fetal Alcohol Syndrome Recognition](https://mlsanity.com/paper/2109.05485/) (2021)
- [Unsupervised 3D Keypoint Discovery with Multi-View Geometry](https://mlsanity.com/paper/2211.12829/) (2022)

## Abstract (arXiv)

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.