Video Generation Models Are Inherent Lighting Estimators
Not on the map. Why?
Abstract (arXiv)
Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 6 Jul 2026.
Builds on
39 of 46 references.
References · 39
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
- 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
- 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
- 2025VACE: All-in-One Video Creation and Editingcited by 467
- 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
- 2023Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylizationcited by 251
- 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
27 more
- 2025Video models are zero-shot learners and reasonerscited by 208
- 2017Learning to Predict Indoor Illumination from a Single Imagecited by 160
- 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
- 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
- 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 63
- 2020Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illuminationcited by 62
- 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
- 2019Neural Illumination: Lighting Prediction for Indoor Environmentscited by 57
- 2016Deep Outdoor Illumination Estimationcited by 53
- 2020EMLight: Lighting Estimation via Spherical Distribution Approximationcited by 47
- 2019A Dataset of Multi-Illumination Images in the Wildcited by 45
- 2023DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimationcited by 36
- 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27
- 2024Lite2Relight: 3D-aware Single Image Portrait Relightingcited by 25
- 2025PanoWan: Lifting Diffusion Video Generation Models to 360deg with Latitude/Longitude-aware Mechanismscited by 22
- 2024Real-time 3D-aware Portrait Video Relightingcited by 21
- 2024Using Diffusion Priors for Video Amodal Segmentationcited by 15
- 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
- 2023Spatiotemporally Consistent HDR Indoor Lighting Estimationcited by 12
- 2025Exploiting Diffusion Prior for Real-World Image Dehazing with Unpaired Trainingcited by 12
- 2025GaSLight: Gaussian Splats for Spatially-Varying Lighting in HDRcited by 11
- 2025DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpaintingcited by 11
- 2024How to Use Diffusion Priors under Sparse Views?cited by 8
- 2024ComPC: Completing a 3D Point Cloud with 2D Diffusion Priorscited by 6
- 2025Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priorscited by 5
- 2024LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimationcited by 3
- 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models
- 2025Lighting in Motion: Spatiotemporal HDR Lighting Estimation
- 2026LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
- 2024GenLit: Reformulating Single-Image Relighting as Video Generation
- 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
- 2025Yesnt: Are Diffusion Relighting Models Ready for Capture Stage Compositing? A Hybrid Alternative to Bridge the Gap
- 2026CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control
- 2025ReLumix: Extending Image Relighting to Video via Video Diffusion Models