# Relightable Neural Actor with Intrinsic Decomposition and Pose Control arXiv: https://arxiv.org/abs/2312.11587 ยท HTML: https://mlsanity.com/paper/2312.11587/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 49 of its 69 references that are in the map. - First posted on arXiv 18 Dec 2023. ## Builds on (most cited first, 30 of 49) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016) - [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021) - [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019) - [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020) - [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018) - [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020) - [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022) - [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020) - [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018) - [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021) - [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022) - [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018) - [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020) - [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022) - [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023) - [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023) - [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020) - [Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering](https://mlsanity.com/paper/2109.07448/) (2021) - [ARAH: Animatable Volume Rendering of Articulated Human SDFs](https://mlsanity.com/paper/2210.10036/) (2022) - [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021) - [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022) - [LEAP: Learning Articulated Occupancy of People](https://mlsanity.com/paper/2104.06849/) (2021) - [ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering](https://mlsanity.com/paper/2312.05941/) (2023) - [imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose](https://mlsanity.com/paper/2108.10842/) (2021) - [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018) - [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022) - [LOLNeRF: Learn from One Look](https://mlsanity.com/paper/2111.09996/) (2021) ## Built on by (most cited first, 11 of 11) - [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024) - [Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2407.15212/) (2024) - [MetaCap: Meta-learning Priors from Multi-View Imagery for Sparse-view Human Performance Capture and Rendering](https://mlsanity.com/paper/2403.18820/) (2024) - [HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis](https://mlsanity.com/paper/2508.09137/) (2025) - [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025) - [Relightable Holoported Characters: Capturing and Relighting Dynamic Human Performance from Sparse Views](https://mlsanity.com/paper/2512.00255/) (2025) - [Relightable and Dynamic Gaussian Avatar Reconstruction from Monocular Video](https://mlsanity.com/paper/2512.09335/) (2025) - [Light4D: Training-Free Extreme Viewpoint 4D Video Relighting](https://mlsanity.com/paper/2602.11769/) (2026) - [POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling](https://mlsanity.com/paper/2512.13192/) (2025) - [LumiMotion: Improving Gaussian Relighting with Scene Dynamics](https://mlsanity.com/paper/2604.10994/) (2026) - [Generative Relightable Avatars](https://mlsanity.com/paper/2606.22718/) (2026) ## Abstract (arXiv) Creating a controllable and relightable digital avatar from multi-view video with fixed illumination is a very challenging problem since humans are highly articulated, creating pose-dependent appearance effects, and skin as well as clothing require space-varying BRDF modeling. Existing works on creating animatible avatars either do not focus on relighting at all, require controlled illumination setups, or try to recover a relightable avatar from very low cost setups, i.e. a single RGB video, at the cost of severely limited result quality, e.g. shadows not even being modeled. To address this, we propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted, allows appearance editing, and models pose-dependent effects such as wrinkles and self-shadows. Importantly, for training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To tackle this challenging problem, we leverage an implicit geometry representation of the actor with a drivable density field that models pose-dependent deformations and derive a dynamic mapping between 3D and UV spaces, where normal, visibility, and materials are effectively encoded. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors, providing the first benchmark of its kind for human relighting, and demonstrating state-of-the-art relighting results for novel human poses.