mlsanityData to 7 Oct 2026

Video Generation Models Are Inherent Lighting Estimators

arXiv 2607.04674 ↗First posted 6 Jul 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

39 of 46 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  8. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  9. 2025VACE: All-in-One Video Creation and Editingcited by 467
  10. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  11. 2023Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylizationcited by 251
  12. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
27 more
  1. 2025Video models are zero-shot learners and reasonerscited by 208
  2. 2017Learning to Predict Indoor Illumination from a Single Imagecited by 160
  3. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  4. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
  5. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 63
  6. 2020Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illuminationcited by 62
  7. 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
  8. 2019Neural Illumination: Lighting Prediction for Indoor Environmentscited by 57
  9. 2016Deep Outdoor Illumination Estimationcited by 53
  10. 2020EMLight: Lighting Estimation via Spherical Distribution Approximationcited by 47
  11. 2019A Dataset of Multi-Illumination Images in the Wildcited by 45
  12. 2023DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimationcited by 36
  13. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27
  14. 2024Lite2Relight: 3D-aware Single Image Portrait Relightingcited by 25
  15. 2025PanoWan: Lifting Diffusion Video Generation Models to 360deg with Latitude/Longitude-aware Mechanismscited by 22
  16. 2024Real-time 3D-aware Portrait Video Relightingcited by 21
  17. 2024Using Diffusion Priors for Video Amodal Segmentationcited by 15
  18. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  19. 2023Spatiotemporally Consistent HDR Indoor Lighting Estimationcited by 12
  20. 2025Exploiting Diffusion Prior for Real-World Image Dehazing with Unpaired Trainingcited by 12
  21. 2025GaSLight: Gaussian Splats for Spatially-Varying Lighting in HDRcited by 11
  22. 2025DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpaintingcited by 11
  23. 2024How to Use Diffusion Priors under Sparse Views?cited by 8
  24. 2024ComPC: Completing a 3D Point Cloud with 2D Diffusion Priorscited by 6
  25. 2025Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priorscited by 5
  26. 2024LightOctree: Lightweight 3D Spatially-Coherent Indoor Lighting Estimationcited by 3
  27. 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models
  2. 2025Lighting in Motion: Spatiotemporal HDR Lighting Estimation
  3. 2026LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
  4. 2024GenLit: Reformulating Single-Image Relighting as Video Generation
  5. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
  6. 2025Yesnt: Are Diffusion Relighting Models Ready for Capture Stage Compositing? A Hybrid Alternative to Bridge the Gap
  7. 2026CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control
  8. 2025ReLumix: Extending Image Relighting to Video via Video Diffusion Models