# Self-supervised Deformation Modeling for Facial Expression Editing

arXiv: https://arxiv.org/abs/1911.00735 · HTML: https://mlsanity.com/paper/1911.00735/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2019.

## Built on by (most cited first, 7 of 7)

- [RigNeRF: Fully Controllable Neural 3D Portraits](https://mlsanity.com/paper/2206.06481/) (2022)
- [FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation](https://mlsanity.com/paper/2108.04913/) (2021)
- [Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos](https://mlsanity.com/paper/2402.03723/) (2024)
- [Universal Facial Encoding of Codec Avatars from VR Headsets](https://mlsanity.com/paper/2407.13038/) (2024)
- [JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation](https://mlsanity.com/paper/2409.12156/) (2024)
- [LES-Talker: Fine-Grained Emotion Editing for Talking Head Generation in Linear Emotion Space](https://mlsanity.com/paper/2411.09268/) (2024)
- [Controllable Dynamic Appearance for Neural 3D Portraits](https://mlsanity.com/paper/2309.11009/) (2023)

## Abstract (arXiv)

Recent advances in deep generative models have demonstrated impressive results in photo-realistic facial image synthesis and editing. Facial expressions are inherently the result of muscle movement. However, existing neural network-based approaches usually only rely on texture generation to edit expressions and largely neglect the motion information. In this work, we propose a novel end-to-end network that disentangles the task of facial editing into two steps: a " "motion-editing" step and a "texture-editing" step. In the "motion-editing" step, we explicitly model facial movement through image deformation, warping the image into the desired expression. In the "texture-editing" step, we generate necessary textures, such as teeth and shading effects, for a photo-realistic result. Our physically-based task-disentanglement system design allows each step to learn a focused task, removing the need of generating texture to hallucinate motion. Our system is trained in a self-supervised manner, requiring no ground truth deformation annotation. Using Action Units [8] as the representation for facial expression, our method improves the state-of-the-art facial expression editing performance in both qualitative and quantitative evaluations.