# Deep Appearance Models for Face Rendering

arXiv: https://arxiv.org/abs/1808.00362 · HTML: https://mlsanity.com/paper/1808.00362/

## Key facts

- Cited by 164 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2018.

## Built on by (most cited first, 30 of 164)

- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [ARCH++: Animation-Ready Clothed Human Reconstruction Revisited](https://mlsanity.com/paper/2108.07845/) (2021)
- [H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion](https://mlsanity.com/paper/2110.13746/) (2021)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [MegaPortraits: One-shot Megapixel Neural Head Avatars](https://mlsanity.com/paper/2207.07621/) (2022)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [Realistic One-shot Mesh-based Head Avatars](https://mlsanity.com/paper/2206.08343/) (2022)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar](https://mlsanity.com/paper/2312.04558/) (2023)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints](https://mlsanity.com/paper/2205.04992/) (2022)
- [DiffusionRig: Learning Personalized Priors for Facial Appearance Editing](https://mlsanity.com/paper/2304.06711/) (2023)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [Modeling Clothing as a Separate Layer for an Animatable Human Avatar](https://mlsanity.com/paper/2106.14879/) (2021)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [Drivable Volumetric Avatars using Texel-Aligned Features](https://mlsanity.com/paper/2207.09774/) (2022)
- [Embodied AI Agents: Modeling the World](https://mlsanity.com/paper/2506.22355/) (2025)
- [HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splatting](https://mlsanity.com/paper/2312.02902/) (2023)
- [3D GAN Inversion for Controllable Portrait Image Animation](https://mlsanity.com/paper/2203.13441/) (2022)
- [StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video](https://mlsanity.com/paper/2305.00942/) (2023)
- [LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar](https://mlsanity.com/paper/2305.01190/) (2023)

## Abstract (arXiv)

We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).