mlsanityData to 7 Oct 2026

Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering

arXiv 2408.09702 ↗First posted 19 Aug 2024cs.CV cs.AI cs.GRcited by 27

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process. While recent large-scale diffusion models have shown strong generative and inpainting capabilities, we find that current models do not sufficiently "understand" the scene shown in a single picture to generate consistent lighting effects (shadows, bright reflections, etc.) while preserving the identity and details of the composited object. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes. Our physically based pipeline further enables automatic materials and tone-mapping refinement.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 50 of its 81 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

50 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  2. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ball14 Dec
  3. 2023Text-to-3D with Classifier Score Distillation30 Oct
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 46

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  11. 2019Neural Spline Flowscited by 590
  12. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
34 more
  1. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  2. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  3. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  4. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  5. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  6. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  7. 2017Learning to Predict Indoor Illumination from a Single Imagecited by 160
  8. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  9. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  10. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  11. 2018CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Renderingcited by 85
  12. 2019Deep Parametric Indoor Lighting Estimationcited by 82
  13. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  14. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
  15. 2019Deep Sky Modeling for Single Image Outdoor Lighting Estimationcited by 66
  16. 2019DeepLight: Learning Illumination for Unconstrained Mobile Mixed Realitycited by 63
  17. 2020Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illuminationcited by 62
  18. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  19. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  20. 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
  21. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
  22. 2019Neural Illumination: Lighting Prediction for Indoor Environmentscited by 57
  23. 2016Deep Outdoor Illumination Estimationcited by 53
  24. 2020EMLight: Lighting Estimation via Spherical Distribution Approximationcited by 47
  25. 2022Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertioncited by 35
  26. 2022Guided Co-Modulated GAN for 360deg Field of View Extrapolationcited by 32
  27. 2023EverLight: Indoor-Outdoor Editable HDR Lighting Estimationcited by 32
  28. 2022De-rendering 3D Objects in the Wildcited by 31
  29. 2017Shading Annotations in the Wildcited by 19
  30. 2020PointAR: Efficient Lighting Estimation for Mobile Augmented Realitycited by 19
  31. 2021Spatially-Varying Outdoor Lighting Estimation from Intrinsicscited by 14
  32. 2023Accidental Light Probescited by 13
  33. 2023Spatiotemporally Consistent HDR Indoor Lighting Estimationcited by 12
  34. 2023Personalized Restoration via Dual-Pivot Tuningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 17; 2026: 6.

27 papers.

Sort:

By area:

  1. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  2. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  3. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relightingcited by 37
  4. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  5. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  6. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  7. 2025R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulationcited by 17
  8. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  9. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Renderingcited by 11
  10. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  11. 2025MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generationcited by 8
  12. 20253D Object Manipulation in a Single Image using Generative Modelscited by 7
15 more
  1. 2025DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Imagecited by 6
  2. 2025Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priorscited by 5
  3. 2025MVInverse: Feed-forward Multi-view Inverse Rendering in Secondscited by 5
  4. 2025MADrive: Memory-Augmented Driving Scene Modelingcited by 3
  5. 2025Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenescited by 3
  6. 2024Monocular Facial Appearance Capture in the Wildcited by 2
  7. 2025Diffusion Models are Secretly Zero-Shot 3DGS Harmonizerscited by 2
  8. 2025SecureT2I: No More Unauthorized Manipulation on AI Generated Images from Promptscited by 1
  9. 2025ComGS: Efficient 3D Object-Scene Composition via Surface Octahedral Probescited by 1
  10. 2026Unposed-to-3D: Learning Simulation-Ready Vehicles from Real-World Imagescited by 1
  11. 2026Under One Sun: Multi-Object Generative Perception of Materials and Illuminationcited by 0
  12. 2026LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splattingcited by 0
  13. 2026BRDFusion: Physics Meets Generation for Urban Scene Inverse Renderingcited by 0
  14. 2026DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulationcited by 0
  15. 2026Video Generation Models Are Inherent Lighting Estimatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  2. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenes
  3. 2025Generative Detail Enhancement for Physically Based Materials
  4. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models
  5. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Rendering
  6. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors
  7. 2023Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering
  8. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering