# Discrete Facial Encoding: : A Framework for Data-driven Facial Display Discovery

arXiv: https://arxiv.org/abs/2510.01662 · HTML: https://mlsanity.com/paper/2510.01662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://mlsanity.com/paper/2211.06627/) (2022)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Can Language Models Learn to Listen?](https://mlsanity.com/paper/2308.10897/) (2023)
- [Dyadic Interaction Modeling for Social Behavior Generation](https://mlsanity.com/paper/2403.09069/) (2024)
- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features](https://mlsanity.com/paper/2308.12380/) (2023)
- [Norface: Improving Facial Expression Analysis by Identity Normalization](https://mlsanity.com/paper/2407.15617/) (2024)
- [Beyond FACS: Data-driven Facial Expression Dictionaries, with Application to Predicting Autism](https://mlsanity.com/paper/2505.24679/) (2025)

## Abstract (arXiv)

Facial expression analysis is central to understanding human behavior, yet existing coding systems such as the Facial Action Coding System (FACS) are constrained by limited coverage and costly manual annotation. In this work, we introduce Discrete Facial Encoding (DFE), an unsupervised, data-driven alternative of compact and interpretable dictionary of facial expressions from 3D mesh sequences learned through a Residual Vector Quantized Variational Autoencoder (RVQ-VAE). Our approach first extracts identity-invariant expression features from images using a 3D Morphable Model (3DMM), effectively disentangling factors such as head pose and facial geometry. We then encode these features using an RVQ-VAE, producing a sequence of discrete tokens from a shared codebook, where each token captures a specific, reusable facial deformation pattern that contributes to the overall expression. Through extensive experiments, we demonstrate that Discrete Facial Encoding captures more precise facial behaviors than FACS and other facial encoding alternatives. We evaluate the utility of our representation across three high-level psychological tasks: stress detection, personality prediction, and depression detection. Using a simple Bag-of-Words model built on top of the learned tokens, our system consistently outperforms both FACS-based pipelines and strong image and video representation learning models such as Masked Autoencoders. Further analysis reveals that our representation covers a wider variety of facial displays, highlighting its potential as a scalable and effective alternative to FACS for psychological and affective computing applications.