# I M Avatar: Implicit Morphable Head Avatars from Videos

arXiv: https://arxiv.org/abs/2112.07471 · HTML: https://mlsanity.com/paper/2112.07471/

## Key facts

- Cited by 231 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 74 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 30 of 49)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes](https://mlsanity.com/paper/2104.03953/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks](https://mlsanity.com/paper/2104.03313/) (2021)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)

## Built on by (most cited first, 30 of 231)

- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads](https://mlsanity.com/paper/2305.03027/) (2023)
- [PointAvatar: Deformable Point-based Head Avatars from Videos](https://mlsanity.com/paper/2212.08377/) (2022)
- [SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting](https://mlsanity.com/paper/2403.05087/) (2024)
- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)](https://mlsanity.com/paper/2210.00379/) (2022)
- [SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video](https://mlsanity.com/paper/2201.12792/) (2022)
- [ARAH: Animatable Volume Rendering of Articulated Human SDFs](https://mlsanity.com/paper/2210.10036/) (2022)
- [Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition](https://mlsanity.com/paper/2302.11566/) (2023)
- [Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video](https://mlsanity.com/paper/2210.06108/) (2022)
- [Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars](https://mlsanity.com/paper/2211.11208/) (2022)
- [Realistic One-shot Mesh-based Head Avatars](https://mlsanity.com/paper/2206.08343/) (2022)
- [FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding](https://mlsanity.com/paper/2312.02214/) (2023)
- [Generative Neural Articulated Radiance Fields](https://mlsanity.com/paper/2206.14314/) (2022)
- [EventNeRF: Neural Radiance Fields from a Single Colour Event Camera](https://mlsanity.com/paper/2206.11896/) (2022)
- [MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar](https://mlsanity.com/paper/2312.04558/) (2023)
- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation](https://mlsanity.com/paper/2309.04946/) (2023)
- [KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints](https://mlsanity.com/paper/2205.04992/) (2022)
- [Imitator: Personalized Speech-driven 3D Facial Animation](https://mlsanity.com/paper/2301.00023/) (2022)
- [GPAvatar: Generalizable and Precise Head Avatar from Image(s)](https://mlsanity.com/paper/2401.10215/) (2024)
- [AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels](https://mlsanity.com/paper/2211.13206/) (2022)
- [Learning Neural Parametric Head Models](https://mlsanity.com/paper/2212.02761/) (2022)
- [gDNA: Towards Generative Detailed Neural Avatars](https://mlsanity.com/paper/2201.04123/) (2022)
- [OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering](https://mlsanity.com/paper/2303.14662/) (2023)
- [X-Avatar: Expressive Human Avatars](https://mlsanity.com/paper/2303.04805/) (2023)
- [EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars](https://mlsanity.com/paper/2404.19110/) (2024)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splatting](https://mlsanity.com/paper/2312.02902/) (2023)

## Abstract (arXiv)

Traditional 3D morphable face models (3DMMs) provide fine-grained control over expression but cannot easily capture geometric and appearance details. Neural volumetric representations approach photorealism but are hard to animate and do not generalize well to unseen expressions. To tackle this problem, we propose IMavatar (Implicit Morphable avatar), a novel method for learning implicit head avatars from monocular videos. Inspired by the fine-grained control mechanisms afforded by conventional 3DMMs, we represent the expression- and pose- related deformations via learned blendshapes and skinning fields. These attributes are pose-independent and can be used to morph the canonical geometry and texture fields given novel expression and pose parameters. We employ ray marching and iterative root-finding to locate the canonical surface intersection for each pixel. A key contribution is our novel analytical gradient formulation that enables end-to-end training of IMavatars from videos. We show quantitatively and qualitatively that our method improves geometry and covers a more complete expression space compared to state-of-the-art methods.