# Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset

arXiv: https://arxiv.org/abs/2503.14485 · HTML: https://mlsanity.com/paper/2503.14485/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 60 of its 97 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 60)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023)
- [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](https://mlsanity.com/paper/2305.10474/) (2023)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution](https://mlsanity.com/paper/2205.03409/) (2022)
- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2312.06640/) (2023)
- [MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion](https://mlsanity.com/paper/2311.12052/) (2023)

## Built on by (most cited first, 28 of 28)

- [UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting](https://mlsanity.com/paper/2506.15673/) (2025)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025)
- [Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis](https://mlsanity.com/paper/2508.05580/) (2025)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [LiveLight: Real-time Streaming Video Relighting with Interactive Control](https://mlsanity.com/paper/2608.01771/) (2026)
- [FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement](https://mlsanity.com/paper/2511.18346/) (2025)
- [Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures](https://mlsanity.com/paper/2510.14179/) (2025)
- [UniLight: A Unified Representation for Lighting](https://mlsanity.com/paper/2512.04267/) (2025)
- [DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models](https://mlsanity.com/paper/2604.06161/) (2026)
- [GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers](https://mlsanity.com/paper/2604.20715/) (2026)
- [Relightable Holoported Characters: Capturing and Relighting Dynamic Human Performance from Sparse Views](https://mlsanity.com/paper/2512.00255/) (2025)
- [Silhouette-based Gait Foundation Model](https://mlsanity.com/paper/2512.00691/) (2025)
- [Lighting in Motion: Spatiotemporal HDR Lighting Estimation](https://mlsanity.com/paper/2512.13597/) (2025)
- [ReLumix: Extending Image Relighting to Video via Video Diffusion Models](https://mlsanity.com/paper/2509.23769/) (2025)
- [POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling](https://mlsanity.com/paper/2512.13192/) (2025)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [ID-V2V: Identity-Preserving Video Restylization](https://mlsanity.com/paper/2607.22830/) (2026)
- [HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures](https://mlsanity.com/paper/2601.02103/) (2026)
- [RelightAnyone: A Generalized Relightable 3D Gaussian Head Model](https://mlsanity.com/paper/2601.03357/) (2026)
- [SyncLight: Single-Edit Multi-View Relighting](https://mlsanity.com/paper/2601.16981/) (2026)
- [Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics](https://mlsanity.com/paper/2602.01391/) (2026)
- [CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control](https://mlsanity.com/paper/2603.14241/) (2026)
- [FusionRelight: Relighting Portraits in Real Time via Hybrid Domain Knowledge Fusion](https://mlsanity.com/paper/2604.23094/) (2026)
- [BodyReLux: Temporally Consistent Full-Body Video Relighting](https://mlsanity.com/paper/2605.21766/) (2026)
- [HarmoVid: Relightful Video Portrait Harmonization](https://mlsanity.com/paper/2605.28811/) (2026)
- [Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction](https://mlsanity.com/paper/2606.02919/) (2026)
- [Generative Relightable Avatars](https://mlsanity.com/paper/2606.22718/) (2026)

## Abstract (arXiv)

Video portrait relighting remains challenging because the results need to be both photorealistic and temporally stable. This typically requires a strong model design that can capture complex facial reflections as well as intensive training on a high-quality paired video dataset, such as dynamic one-light-at-a-time (OLAT). In this work, we introduce Lux Post Facto, a novel portrait video relighting method that produces both photorealistic and temporally consistent lighting effects. From the model side, we design a new conditional video diffusion model built upon state-of-the-art pre-trained video diffusion model, alongside a new lighting injection mechanism to enable precise control. This way we leverage strong spatial and temporal generative capability to generate plausible solutions to the ill-posed relighting problem. Our technique uses a hybrid dataset consisting of static expression OLAT data and in-the-wild portrait performance videos to jointly learn relighting and temporal modeling. This avoids the need to acquire paired video data in different lighting conditions. Our extensive experiments show that our model produces state-of-the-art results both in terms of photorealism and temporal consistency.