mlsanityData to 13 Oct 2026

DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models

arXiv 2501.18590 ↗First posted 30 Jan 2025cs.CV cs.GRcited by 55

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.

Key facts. Cited by 55 papers in the map (data through 13 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 62 of its 93 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

62 of 93 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting29 Nov
  2. 2024Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think17 Sep
  3. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis13 Sep
  4. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering19 Aug
  5. 2024Neural Gaffer: Relighting Any Object via Diffusion11 Jun
  6. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models1 May
  7. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination17 Apr
  8. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image18 Mar
  9. 2024LightIt: Illumination Modeling and Control for Diffusion Models15 Mar
  10. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation19 Feb
  11. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ball14 Dec
  12. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,213
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,170
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,023
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,550
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,529
  12. 2025Cosmos World Foundation Model Platform for Physical AIcited by 848
38 more
  1. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 594
  2. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 521
  3. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 447
  4. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 364
  5. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 333
  6. 2020State of the Art on Neural Renderingcited by 277
  7. 2023GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfacescited by 250
  8. 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 222
  9. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  10. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
  11. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 152
  12. 2021NeRF for Outdoor Scene Relightingcited by 114
  13. 2022IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Imagescited by 111
  14. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  15. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  16. 2021Real-time Neural Radiance Caching for Path Tracingcited by 91
  17. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 90
  18. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 86
  19. 2018CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Renderingcited by 85
  20. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  21. 2022Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracingcited by 78
  22. 2020OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasetscited by 73
  23. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 63
  24. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  25. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  26. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  27. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  28. 2016Deep Shading: Convolutional Neural Networks for Screen-Space Shadingcited by 46
  29. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  30. 2020Learning to Factorize and Relight a Citycited by 36
  31. 2022Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertioncited by 35
  32. 2024ColorVideoVDP: A visual difference predictor for image, video and display distortionscited by 33
  33. 2022De-rendering 3D Objects in the Wildcited by 31
  34. 2022OutCast: Outdoor Single-image Relighting with Cast Shadowscited by 30
  35. 2023UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Videocited by 22
  36. 2017Shading Annotations in the Wildcited by 19
  37. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  38. 2018Deep-learning the Latent Space of Light Transportcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 28; 2026: 27.

55 papers.

Sort:

By area:

  1. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 81
  2. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  3. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  4. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 19
  5. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  6. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
  7. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 16
  8. 2025LightSwitch: Multi-view Relighting with Material-guided Diffusioncited by 15
  9. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  10. 2026Video Generation Models are General-Purpose Vision Learnerscited by 13
  11. 2025Controllable Weather Synthesis and Removal with Video Diffusion Modelscited by 11
  12. 2025VideoMat: Extracting PBR Materials from Video Diffusion Modelscited by 10
38 more
  1. 2026DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancercited by 10
  2. 2025MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generationcited by 8
  3. 2026Programmable World Modelcited by 7
  4. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusioncited by 5
  5. 2026THFM: A Unified Video Foundation Model for 4D Human Perception and Beyondcited by 5
  6. 2025InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modelingcited by 4
  7. 2025A Real-world Display Inverse Rendering Datasetcited by 4
  8. 2025NeAR: Coupled Neural Asset-Renderer Stackcited by 4
  9. 2025ePBR: Extended PBR Materials in Image Synthesiscited by 3
  10. 2025X2Video: Adapting Diffusion Models for Multimodal Controllable Neural Video Renderingcited by 3
  11. 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 3
  12. 2025StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimationcited by 2
  13. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  14. 2025IntrinsicWeather: Controllable Weather Editing in Intrinsic Spacecited by 1
  15. 2025ReLumix: Extending Image Relighting to Video via Video Diffusion Modelscited by 1
  16. 2025Lighting in Motion: Spatiotemporal HDR Lighting Estimationcited by 1
  17. 2026HorizonRelight: Relighting Long-horizon Videos Consistently via Diffusion Transformerscited by 1
  18. 2026InvSplat: Inverse Feed-Forward Scene Splattingcited by 1
  19. 2025Spatial Polarization Multiplexing: Single-Shot Invisible Shape and Reflectance Recoverycited by 0
  20. 2025NeuralMeshing: Complete Object Mesh Extraction from Casual Capturescited by 0
  21. 2025DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generationcited by 0
  22. 2025LumiX: Structured and Coherent Text-to-Intrinsic Generationcited by 0
  23. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  24. 2026RelightAnyone: A Generalized Relightable 3D Gaussian Head Modelcited by 0
  25. 2026SyncLight: Single-Edit Multi-View Relightingcited by 0
  26. 2026Relighting as a Probe of Visual Priors via Augmented Latent Intrinsicscited by 0
  27. 2026Geo-ID: Test-Time Geometric Consensus for Cross-View Consistent Intrinsicscited by 0
  28. 2026Under One Sun: Multi-Object Generative Perception of Materials and Illuminationcited by 0
  29. 2026GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Modelscited by 0
  30. 2026TOPOS: High-Fidelity and Efficient Industry-Grade 3D Head Generationcited by 0
  31. 2026TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructionscited by 0
  32. 2026BRDFusion: Physics Meets Generation for Urban Scene Inverse Renderingcited by 0
  33. 2026Generative Relightable Avatarscited by 0
  34. 2026AEGIR: Modeling Area Emitters for Indoor Inverse Rendering using Gaussian Splattingcited by 0
  35. 2026DANTE-W: Diffuse Albedo Neural Texturing in the Wildcited by 0
  36. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Renderingcited by 0
  37. 2026LumiTokens: 3D Relighting via Token-Space Lighting Transformationcited by 0
  38. 2026HELIOS: From midnight to noon, continuous outdoor urban scene relightingcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
  2. 2024Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion
  3. 2026RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
  4. 2026Feature-Guided Diffusion for Non-Differentiable Inverse Rendering
  5. 2025Yesnt: Are Diffusion Relighting Models Ready for Capture Stage Compositing? A Hybrid Alternative to Bridge the Gap
  6. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenes
  7. 2026DiffusionShadow: Diffusion-based Shadow Caching for Neural Volume Rendering
  8. 2026PI-Light: Physics-Inspired Diffusion for Full-Image Relighting