# High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model arXiv: https://arxiv.org/abs/2502.19894 ยท HTML: https://mlsanity.com/paper/2502.19894/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Audio, speech and music, one of 14 directions in Diffusion models. - Builds on 50 of its 64 references that are in the map. - First posted on arXiv 27 Feb 2025. ## Placed in - Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/) ## Builds on (most cited first, 30 of 50) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022) - [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020) - [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023) - [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024) - [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017) - [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020) - [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021) - [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022) - [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022) - [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020) - [Composer: Creative and Controllable Image Synthesis with Composable Conditions](https://mlsanity.com/paper/2302.09778/) (2023) - [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021) - [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021) ## Built on by (most cited first, 9 of 9) - [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025) - [PersonaLive! Expressive Portrait Image Animation for Live Streaming](https://mlsanity.com/paper/2512.11253/) (2025) - [Total-Editing: Head Avatar with Editable Appearance, Motion, and Lighting](https://mlsanity.com/paper/2505.20582/) (2025) - [ReLumix: Extending Image Relighting to Video via Video Diffusion Models](https://mlsanity.com/paper/2509.23769/) (2025) - [HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures](https://mlsanity.com/paper/2601.02103/) (2026) - [PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing](https://mlsanity.com/paper/2603.19731/) (2026) - [MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation](https://mlsanity.com/paper/2605.08050/) (2026) - [HarmoVid: Relightful Video Portrait Harmonization](https://mlsanity.com/paper/2605.28811/) (2026) - [PixReenact: Pixel-Conditioned Causal Video Diffusion for Streaming Head-Avatar Reenactment](https://mlsanity.com/paper/2610.05233/) (2026) ## Abstract (arXiv) Relightable portrait animation aims to animate a static reference portrait to match the head movements and expressions of a driving video while adapting to user-specified or reference lighting conditions. Existing portrait animation methods fail to achieve relightable portraits because they do not separate and manipulate intrinsic (identity and appearance) and extrinsic (pose and lighting) features. In this paper, we present a Lighting Controllable Video Diffusion model (LCVD) for high-fidelity, relightable portrait animation. We address this limitation by distinguishing these feature types through dedicated subspaces within the feature space of a pre-trained image-to-video diffusion model. Specifically, we employ the 3D mesh, pose, and lighting-rendered shading hints of the portrait to represent the extrinsic attributes, while the reference represents the intrinsic attributes. In the training phase, we employ a reference adapter to map the reference into the intrinsic feature subspace and a shading adapter to map the shading hints into the extrinsic feature subspace. By merging features from these subspaces, the model achieves nuanced control over lighting, pose, and expression in generated animations. Extensive evaluations show that LCVD outperforms state-of-the-art methods in lighting realism, image quality, and video consistency, setting a new benchmark in relightable portrait animation.