# Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild

arXiv: https://arxiv.org/abs/2305.08296 · HTML: https://mlsanity.com/paper/2305.08296/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 18 of 18)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [DiffusionNet: Discretization Agnostic Learning on Surfaces](https://mlsanity.com/paper/2012.00888/) (2020)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator](https://mlsanity.com/paper/1911.05856/) (2019)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [Variational Autoencoders for Deforming 3D Mesh Models](https://mlsanity.com/paper/1709.04307/) (2017)
- [Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels](https://mlsanity.com/paper/2006.04325/) (2020)
- [Disentangled Representation Learning for 3D Face Shape](https://mlsanity.com/paper/1902.09887/) (2019)
- [Multilinear Wavelets: A Statistical Shape Space for Human Faces](https://mlsanity.com/paper/1401.2818/) (2014)
- [Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes](https://mlsanity.com/paper/2205.02904/) (2022)
- [Accurate Face Rig Approximation with Deep Differential Subspace Reconstruction](https://mlsanity.com/paper/2006.01746/) (2020)
- [JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling](https://mlsanity.com/paper/2007.06755/) (2020)

## Built on by (most cited first, 25 of 25)

- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [Anymate: A Dataset and Baselines for Learning 3D Object Rigging](https://mlsanity.com/paper/2505.06227/) (2025)
- [ScanTalk: 3D Talking Heads from Unregistered Scans](https://mlsanity.com/paper/2403.10942/) (2024)
- [HeadEvolver: Text to Head Avatars via Expressive and Attribute-Preserving Mesh Deformation](https://mlsanity.com/paper/2403.09326/) (2024)
- [DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image](https://mlsanity.com/paper/2406.17988/) (2024)
- [Category-Agnostic Neural Object Rigging](https://mlsanity.com/paper/2505.20283/) (2025)
- [ShapeFusion: A 3D diffusion model for localized shape editing](https://mlsanity.com/paper/2403.19773/) (2024)
- [SMF: Template-free and Rig-free Animation Transfer using Kinetic Codes](https://mlsanity.com/paper/2504.04831/) (2025)
- [PoissonNet: A Local-Global Approach for Learning on Surfaces](https://mlsanity.com/paper/2510.14146/) (2025)
- [Basis restricted elastic shape analysis on the space of unregistered surfaces](https://mlsanity.com/paper/2311.04382/) (2023)
- [Locally Adaptive Neural 3D Morphable Models](https://mlsanity.com/paper/2401.02937/) (2024)
- [Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads](https://mlsanity.com/paper/2410.11041/) (2024)
- [Neural Face Skinning for Mesh-agnostic Facial Expression Cloning](https://mlsanity.com/paper/2505.22416/) (2025)
- [LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example](https://mlsanity.com/paper/2403.15227/) (2024)
- [RigAnyFace: Scaling Neural Facial Mesh Auto-Rigging with Unlabeled Data](https://mlsanity.com/paper/2511.18601/) (2025)
- [Democratizing the Creation of Animatable Facial Avatars](https://mlsanity.com/paper/2401.16534/) (2024)
- [PaNDaS: Learnable Deformation Modeling with Localized Control](https://mlsanity.com/paper/2412.02306/) (2024)
- [ScanMove: Motion Prediction and Transfer for Unregistered Body Meshes](https://mlsanity.com/paper/2508.21095/) (2025)
- [OmniFaceRig: Fully Automatic Inner-Mouth-Aware Face Rigging Across Diverse 3D Character Topologies](https://mlsanity.com/paper/2606.08043/) (2026)
- [Improving Facial Rig Semantics for Tracking and Retargeting](https://mlsanity.com/paper/2508.08429/) (2025)
- [Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment](https://mlsanity.com/paper/2509.16727/) (2025)
- [Learning Domain Agnostic Latent Embeddings of 3D Faces for Zero-shot Animal Expression Transfer](https://mlsanity.com/paper/2601.06484/) (2026)
- [FreeTalk: Emotional Topology-Free 3D Talking Heads](https://mlsanity.com/paper/2603.15512/) (2026)
- [AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model](https://mlsanity.com/paper/2608.16143/) (2026)
- [TopoRig: Topology-Agnostic Facial Rigging via Multi-Source Supervision](https://mlsanity.com/paper/2609.15746/) (2026)

## Abstract (arXiv)

We propose an end-to-end deep-learning approach for automatic rigging and retargeting of 3D models of human faces in the wild. Our approach, called Neural Face Rigging (NFR), holds three key properties: (i) NFR's expression space maintains human-interpretable editing parameters for artistic controls; (ii) NFR is readily applicable to arbitrary facial meshes with different connectivity and expressions; (iii) NFR can encode and produce fine-grained details of complex expressions performed by arbitrary subjects. To the best of our knowledge, NFR is the first approach to provide realistic and controllable deformations of in-the-wild facial meshes, without the manual creation of blendshapes or correspondence. We design a deformation autoencoder and train it through a multi-dataset training scheme, which benefits from the unique advantages of two data sources: a linear 3DMM with interpretable control parameters as in FACS, and 4D captures of real faces with fine-grained details. Through various experiments, we show NFR's ability to automatically produce realistic and accurate facial deformations across a wide range of existing datasets as well as noisy facial scans in-the-wild, while providing artist-controlled, editable parameters.