# Towards an objective characterization of an individual's facial movements using Self-Supervised Person-Specific-Models

arXiv: https://arxiv.org/abs/2211.08279 · HTML: https://mlsanity.com/paper/2211.08279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 54 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 9 of 9)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Disentangled Representation Learning for 3D Face Shape](https://mlsanity.com/paper/1902.09887/) (2019)
- [Self-supervised learning of a facial attribute embedding from video](https://mlsanity.com/paper/1808.06882/) (2018)
- [A Compact Embedding for Facial Expression Similarity](https://mlsanity.com/paper/1811.11283/) (2018)
- [Learning Facial Representations from the Cycle-consistency of Face](https://mlsanity.com/paper/2108.03427/) (2021)
- [Action Units Recognition by Pairwise Deep Architecture](https://mlsanity.com/paper/2010.00288/) (2020)

## Abstract (arXiv)

Disentangling facial movements from other facial characteristics, particularly from facial identity, remains a challenging task, as facial movements display great variation between individuals. In this paper, we aim to characterize individual-specific facial movements. We present a novel training approach to learn facial movements independently of other facial characteristics, focusing on each individual separately. We propose self-supervised Person-Specific Models (PSMs), in which one model per individual can learn to extract an embedding of the facial movements independently of the person's identity and other structural facial characteristics from unlabeled facial video. These models are trained using encoder-decoder-like architectures. We provide quantitative and qualitative evidence that a PSM learns a meaningful facial embedding that discovers fine-grained movements otherwise not characterized by a General Model (GM), which is trained across individuals and characterizes general patterns of facial movements. We present quantitative and qualitative evidence that this approach is easily scalable and generalizable for new individuals: facial movements knowledge learned on a person can quickly and effectively be transferred to a new person. Lastly, we propose a novel PSM using curriculum temporal learning to leverage the temporal contiguity between video frames. Our code, analysis details, and all pretrained models are available in Github and Supplementary Materials.