# FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation

arXiv: https://arxiv.org/abs/2108.04913 · HTML: https://mlsanity.com/paper/2108.04913/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 30 of 32)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes](https://mlsanity.com/paper/2104.06935/) (2021)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Portrait Neural Radiance Fields from a Single Image](https://mlsanity.com/paper/2012.05903/) (2020)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Head2Head: Video-based Neural Head Synthesis](https://mlsanity.com/paper/2005.10954/) (2020)
- [X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation](https://mlsanity.com/paper/2010.00450/) (2020)
- [Head2Head++: Deep Facial Attributes Re-Targeting](https://mlsanity.com/paper/2006.10199/) (2020)

## Built on by (most cited first, 30 of 33)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar](https://mlsanity.com/paper/2312.04558/) (2023)
- [KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints](https://mlsanity.com/paper/2205.04992/) (2022)
- [GPAvatar: Generalizable and Precise Head Avatar from Image(s)](https://mlsanity.com/paper/2401.10215/) (2024)
- [AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels](https://mlsanity.com/paper/2211.13206/) (2022)
- [LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar](https://mlsanity.com/paper/2305.01190/) (2023)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [HHAvatar: Gaussian Head Avatar with Dynamic Hairs](https://mlsanity.com/paper/2312.03029/) (2023)
- [NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields](https://mlsanity.com/paper/2211.07968/) (2022)
- [RGBAvatar: Reduced Gaussian Blendshapes for Online Modeling of Head Avatars](https://mlsanity.com/paper/2503.12886/) (2025)
- [MEGANE: Morphable Eyeglass and Avatar Network](https://mlsanity.com/paper/2302.04868/) (2023)
- [Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization](https://mlsanity.com/paper/2305.03043/) (2023)
- [GPHM: Gaussian Parametric Head Model for Monocular Head Avatar Reconstruction](https://mlsanity.com/paper/2407.15070/) (2024)
- [Neural Radiance Fields: Past, Present, and Future](https://mlsanity.com/paper/2304.10050/) (2023)
- [FastAvatar: Towards Unified and Fast 3D Avatar Reconstruction with Large Gaussian Reconstruction Transformers](https://mlsanity.com/paper/2508.19754/) (2025)
- [Total Selfie: Generating Full-Body Selfies](https://mlsanity.com/paper/2308.14740/) (2023)
- [High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field](https://mlsanity.com/paper/2310.06275/) (2023)
- [Dynamic NeRF: A Review](https://mlsanity.com/paper/2405.08609/) (2024)
- [CoNFies: Controllable Neural Face Avatars](https://mlsanity.com/paper/2211.08610/) (2022)
- [MI-NeRF: Learning a Single Face NeRF from Multiple Identities](https://mlsanity.com/paper/2403.19920/) (2024)
- [JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation](https://mlsanity.com/paper/2409.12156/) (2024)
- [GGAvatar: Geometric Adjustment of Gaussian Head Avatar](https://mlsanity.com/paper/2405.11993/) (2024)
- [PrismAvatar: Real-time animated 3D neural head avatars on edge devices](https://mlsanity.com/paper/2502.07030/) (2025)
- [PhysHead: Simulation-Ready Gaussian Head Avatars](https://mlsanity.com/paper/2604.06467/) (2026)
- [NeRFlame: FLAME-based conditioning of NeRF for 3D face rendering](https://mlsanity.com/paper/2303.06226/) (2023)
- [SemFaceEdit: Semantic Face Editing on Generative Radiance Manifolds](https://mlsanity.com/paper/2506.22833/) (2025)
- [Generating Fit Check Videos with a Handheld Camera](https://mlsanity.com/paper/2505.23886/) (2025)
- [Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture](https://mlsanity.com/paper/2512.16397/) (2025)
- [FHAvatar: Fast and High-Fidelity Reconstruction of Face-and-Hair Composable 3D Head Avatar from Few Casual Captures](https://mlsanity.com/paper/2603.23345/) (2026)
- [SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstruction](https://mlsanity.com/paper/2606.15659/) (2026)

## Abstract (arXiv)

This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), has enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low-dimensional expression representation. We leverage the expression space of a 3D morphable face model (3DMM) to represent the distribution of human facial expressions, and use it to condition the NeRF volumetric function. Furthermore, we impose a spatial prior brought by 3DMM fitting to guide the network to learn disentangled control for scene appearance and facial actions. We demonstrate the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.