# Generating Multiple 4D Expression Transitions by Learning Face Landmark Trajectories

arXiv: https://arxiv.org/abs/2208.00050 · HTML: https://mlsanity.com/paper/2208.00050/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 28 of 28)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [Deformable Shape Completion with Graph Convolutional Autoencoders](https://mlsanity.com/paper/1712.00268/) (2017)
- [Speech Driven Talking Face Generation from a Single Image and an Emotion Condition](https://mlsanity.com/paper/2008.03592/) (2020)
- [Automatic Face Reenactment](https://mlsanity.com/paper/1602.02651/) (2016)
- [A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3D Meshes](https://mlsanity.com/paper/1809.06664/) (2018)
- [Multilinear Wavelets: A Statistical Shape Space for Human Faces](https://mlsanity.com/paper/1401.2818/) (2014)
- [Gaussian Process Morphable Models](https://mlsanity.com/paper/1603.07254/) (2016)
- [Supervision by Registration and Triangulation for Landmark Detection](https://mlsanity.com/paper/2101.09866/) (2021)
- [ExpNet: Landmark-Free, Deep, 3D Facial Expressions](https://mlsanity.com/paper/1802.00542/) (2018)
- [3DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translation](https://mlsanity.com/paper/1905.00307/) (2019)
- [Every Smile is Unique: Landmark-Guided Diverse Smile Generation](https://mlsanity.com/paper/1802.01873/) (2018)
- [FacePoseNet: Making a Case for Landmark-Free Face Alignment](https://mlsanity.com/paper/1708.07517/) (2017)
- [Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets](https://mlsanity.com/paper/1907.10087/) (2019)
- [Sparse to Dense Dynamic 3D Facial Expression Generation](https://mlsanity.com/paper/2105.07463/) (2021)
- [Robust Facial Landmark Detection by Multi-order Multi-constraint Deep Networks](https://mlsanity.com/paper/2012.04927/) (2020)
- [Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation](https://mlsanity.com/paper/1808.02992/) (2018)
- [A Sparse and Locally Coherent Morphable Face Model for Dense Semantic Correspondence Across Heterogeneous 3D Faces](https://mlsanity.com/paper/2006.03840/) (2020)
- [Learning to Generate Customized Dynamic 3D Facial Expressions](https://mlsanity.com/paper/2007.09805/) (2020)
- [A Riemannian Framework for Analysis of Human Body Surface](https://mlsanity.com/paper/2108.11449/) (2021)
- [A Novel Space-Time Representation on the Positive Semidefinite Con for Facial Expression Recognition](https://mlsanity.com/paper/1707.06440/) (2017)
- [Deep Structured Prediction for Facial Landmark Detection](https://mlsanity.com/paper/2010.09035/) (2020)

## Built on by (most cited first, 7 of 7)

- [Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation](https://mlsanity.com/paper/2306.01415/) (2023)
- [Neuromorphic Facial Analysis with Cross-Modal Supervision](https://mlsanity.com/paper/2409.10213/) (2024)
- [Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads](https://mlsanity.com/paper/2410.11041/) (2024)
- [JambaTalk: Speech-Driven 3D Talking Head Generation Based on Hybrid Transformer-Mamba Model](https://mlsanity.com/paper/2408.01627/) (2024)
- [Towards Consistent and Controllable Image Synthesis for Face Editing](https://mlsanity.com/paper/2502.02465/) (2025)
- [Spatio-temporal Transformers for Action Unit Classification with Event Cameras](https://mlsanity.com/paper/2410.21958/) (2024)
- [FreeTalk: Emotional Topology-Free 3D Talking Heads](https://mlsanity.com/paper/2603.15512/) (2026)

## Abstract (arXiv)

In this work, we address the problem of 4D facial expressions generation. This is usually addressed by animating a neutral 3D face to reach an expression peak, and then get back to the neutral state. In the real world though, people show more complex expressions, and switch from one expression to another. We thus propose a new model that generates transitions between different expressions, and synthesizes long and composed 4D expressions. This involves three sub-problems: (i) modeling the temporal dynamics of expressions, (ii) learning transitions between them, and (iii) deforming a generic mesh. We propose to encode the temporal evolution of expressions using the motion of a set of 3D landmarks, that we learn to generate by training a manifold-valued GAN (Motion3DGAN). To allow the generation of composed expressions, this model accepts two labels encoding the starting and the ending expressions. The final sequence of meshes is generated by a Sparse2Dense mesh Decoder (S2D-Dec) that maps the landmark displacements to a dense, per-vertex displacement of a known mesh topology. By explicitly working with motion trajectories, the model is totally independent from the identity. Extensive experiments on five public datasets show that our proposed approach brings significant improvements with respect to previous solutions, while retaining good generalization to unseen data.