# Relit-LiVE: Relight Video by Jointly Learning Environment Video

arXiv: https://arxiv.org/abs/2605.06658 · HTML: https://mlsanity.com/paper/2605.06658/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 29 of 29)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal](https://mlsanity.com/paper/2406.16864/) (2024)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [PhysGen3D: Crafting a Miniature Interactive World from a Single Image](https://mlsanity.com/paper/2503.20746/) (2025)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [Light-A-Video: Training-free Video Relighting via Progressive Light Fusion](https://mlsanity.com/paper/2502.08590/) (2025)
- [LightLab: Controlling Light Sources in Images with Diffusion Models](https://mlsanity.com/paper/2505.09608/) (2025)
- [STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimation](https://mlsanity.com/paper/2302.01334/) (2023)
- [OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data](https://mlsanity.com/paper/2501.07397/) (2025)
- [RelightVid: Temporal-Consistent Diffusion Model for Video Relighting](https://mlsanity.com/paper/2501.16330/) (2025)
- [NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion Priors](https://mlsanity.com/paper/2504.11427/) (2025)
- [IntrinsiX: High-Quality PBR Generation using Image Priors](https://mlsanity.com/paper/2504.01008/) (2025)
- [Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion](https://mlsanity.com/paper/2412.15050/) (2024)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer](https://mlsanity.com/paper/2506.18904/) (2025)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)
- [RelightMaster: Precise Video Relighting with Multi-plane Light Images](https://mlsanity.com/paper/2511.06271/) (2025)
- [CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion](https://mlsanity.com/paper/2511.21129/) (2025)
- [V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties](https://mlsanity.com/paper/2512.11799/) (2025)
- [FrameDiffuser: G-Buffer-Conditioned Diffusion for Neural Forward Frame Rendering](https://mlsanity.com/paper/2512.16670/) (2025)

## Built on by (most cited first, 1 of 1)

- [Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models](https://mlsanity.com/paper/2606.20233/) (2026)

## Abstract (arXiv)

Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.