# Towards Practical Capture of High-Fidelity Relightable Avatars

arXiv: https://arxiv.org/abs/2309.04247 · HTML: https://mlsanity.com/paper/2309.04247/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 67 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video](https://mlsanity.com/paper/2210.06108/) (2022)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations](https://mlsanity.com/paper/2203.14510/) (2022)
- [Drivable Volumetric Avatars using Texel-Aligned Features](https://mlsanity.com/paper/2207.09774/) (2022)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [Disentangled Representation Learning for 3D Face Shape](https://mlsanity.com/paper/1902.09887/) (2019)
- [End-to-end 3D face reconstruction with deep neural networks](https://mlsanity.com/paper/1704.05020/) (2017)
- [MEGANE: Morphable Eyeglass and Avatar Network](https://mlsanity.com/paper/2302.04868/) (2023)
- [Light Stage Super-Resolution: Continuous High-Frequency Relighting](https://mlsanity.com/paper/2010.08888/) (2020)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking](https://mlsanity.com/paper/1907.10815/) (2019)
- [An Implicit Parametric Morphable Dental Model](https://mlsanity.com/paper/2211.11402/) (2022)
- [NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images](https://mlsanity.com/paper/2303.14092/) (2023)

## Built on by (most cited first, 30 of 34)

- [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://mlsanity.com/paper/2407.03168/) (2024)
- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Video](https://mlsanity.com/paper/2411.15604/) (2024)
- [GaussianHeads: End-to-End Learning of Drivable Gaussian Head Avatars from Coarse-to-fine Representations](https://mlsanity.com/paper/2409.11951/) (2024)
- [Lite2Relight: 3D-aware Single Image Portrait Relighting](https://mlsanity.com/paper/2407.10487/) (2024)
- [Relightable Full-Body Gaussian Codec Avatars](https://mlsanity.com/paper/2501.14726/) (2025)
- [VRMM: A Volumetric Relightable Morphable Head Model](https://mlsanity.com/paper/2402.04101/) (2024)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars](https://mlsanity.com/paper/2410.11682/) (2024)
- [Gaussian Shadow Casting for Neural Characters](https://mlsanity.com/paper/2401.06116/) (2024)
- [Relightable Neural Actor with Intrinsic Decomposition and Pose Control](https://mlsanity.com/paper/2312.11587/) (2023)
- [SEGA: Drivable 3D Gaussian Head Avatar from a Single Image](https://mlsanity.com/paper/2504.14373/) (2025)
- [3DPR: Single Image 3D Portrait Relight using Generative Priors](https://mlsanity.com/paper/2510.15846/) (2025)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [Universal Facial Encoding of Codec Avatars from VR Headsets](https://mlsanity.com/paper/2407.13038/) (2024)
- [HairCUP: Hair Compositional Universal Prior for 3D Gaussian Avatars](https://mlsanity.com/paper/2507.19481/) (2025)
- [HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis](https://mlsanity.com/paper/2508.09137/) (2025)
- [Artist-Friendly Relightable and Animatable Neural Heads](https://mlsanity.com/paper/2312.03420/) (2023)
- [Interactive Rendering of Relightable and Animatable Gaussian Avatars](https://mlsanity.com/paper/2407.10707/) (2024)
- [BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading](https://mlsanity.com/paper/2506.06271/) (2025)
- [Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis](https://mlsanity.com/paper/2505.21502/) (2025)
- [STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction](https://mlsanity.com/paper/2511.19854/) (2025)
- [Large-scale Codec Avatars: The Unreasonable Effectiveness of Large-scale Avatar Pretraining](https://mlsanity.com/paper/2604.02320/) (2026)
- [Monocular Facial Appearance Capture in the Wild](https://mlsanity.com/paper/2412.12765/) (2024)
- [Relightable and Dynamic Gaussian Avatar Reconstruction from Monocular Video](https://mlsanity.com/paper/2512.09335/) (2025)
- [GTAvatar: Bridging Gaussian Splatting and Texture Mapping for Relightable and Editable Gaussian Avatars](https://mlsanity.com/paper/2512.09162/) (2025)
- [Fast and Physically-based Neural Explicit Surface for Relightable Human Avatars](https://mlsanity.com/paper/2503.18408/) (2025)
- [Generative Head-Mounted Camera Captures for Photorealistic Avatars](https://mlsanity.com/paper/2507.05620/) (2025)

## Abstract (arXiv)

In this paper, we propose a novel framework, Tracking-free Relightable Avatar (TRAvatar), for capturing and reconstructing high-fidelity 3D avatars. Compared to previous methods, TRAvatar works in a more practical and efficient setting. Specifically, TRAvatar is trained with dynamic image sequences captured in a Light Stage under varying lighting conditions, enabling realistic relighting and real-time animation for avatars in diverse scenes. Additionally, TRAvatar allows for tracking-free avatar capture and obviates the need for accurate surface tracking under varying illumination conditions. Our contributions are two-fold: First, we propose a novel network architecture that explicitly builds on and ensures the satisfaction of the linear nature of lighting. Trained on simple group light captures, TRAvatar can predict the appearance in real-time with a single forward pass, achieving high-quality relighting effects under illuminations of arbitrary environment maps. Second, we jointly optimize the facial geometry and relightable appearance from scratch based on image sequences, where the tracking is implicitly learned. This tracking-free approach brings robustness for establishing temporal correspondences between frames under different lighting conditions. Extensive qualitative and quantitative experiments demonstrate that our framework achieves superior performance for photorealistic avatar animation and relighting.