# LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

arXiv: https://arxiv.org/abs/2607.08016 · HTML: https://mlsanity.com/paper/2607.08016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 30)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [Stable Virtual Camera: Generative View Synthesis with Diffusion Models](https://mlsanity.com/paper/2503.14489/) (2025)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models](https://mlsanity.com/paper/2503.01774/) (2025)
- [NeILF: Neural Incident Light Field for Physically-based Material Estimation](https://mlsanity.com/paper/2203.07182/) (2022)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimation](https://mlsanity.com/paper/2303.17147/) (2023)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2410.02619/) (2024)
- [UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting](https://mlsanity.com/paper/2506.15673/) (2025)
- [Light-A-Video: Training-free Video Relighting via Progressive Light Fusion](https://mlsanity.com/paper/2502.08590/) (2025)
- [MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors](https://mlsanity.com/paper/2409.15273/) (2024)
- [FlowR: Flowing from Sparse to Dense 3D Reconstructions](https://mlsanity.com/paper/2504.01647/) (2025)
- [LightSwitch: Multi-view Relighting with Material-guided Diffusion](https://mlsanity.com/paper/2508.06494/) (2025)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025)
- [UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback](https://mlsanity.com/paper/2511.01678/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer](https://mlsanity.com/paper/2602.24096/) (2026)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation](https://mlsanity.com/paper/2603.20192/) (2026)
- [GR3EN: Generative Relighting for 3D Environments](https://mlsanity.com/paper/2601.16272/) (2026)

## Abstract (arXiv)

Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.