mlsanityData to 7 Oct 2026

Intrinsic Image Diffusion for Indoor Single-view Material Estimation

arXiv 2312.12274 ↗First posted 19 Dec 2023cs.CV cs.AI cs.GRcited by 73

Not on the map. Why?

Abstract (arXiv)

We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Appearance decomposition poses a considerable challenge in computer vision due to the inherent ambiguity between lighting and material properties and the lack of real datasets. To address this issue, we advocate for a probabilistic formulation, where instead of attempting to directly predict the true material properties, we employ a conditional generative model to sample from the solution space. Furthermore, we show that utilizing the strong learned prior of recent diffusion models trained on large-scale real-world images can be adapted to material estimation and highly improves the generalization to real images. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by 1.5dB on PSNR and by 45% better FID score on albedo prediction. We demonstrate the effectiveness of our approach through experiments on both synthetic and real-world datasets.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

34 of 50 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  11. 2021Variational Diffusion Modelscited by 1,145
  12. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
22 more
  1. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  2. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  3. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  4. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  5. 2018CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Renderingcited by 85
  6. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  7. 2022Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracingcited by 78
  8. 2018Learning Intrinsic Image Decomposition from Watching the Worldcited by 74
  9. 2019Inverse Path Tracing for Joint Material and Lighting Estimationcited by 62
  10. 2022Physically-Based Editing of Indoor Scene Lighting from a Single Imagecited by 62
  11. 2022IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenescited by 62
  12. 2021Free-viewpoint Indoor Neural Relighting from Multi-view Stereocited by 50
  13. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  14. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  15. 2023I2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFscited by 42
  16. 2017Shading Annotations in the Wildcited by 19
  17. 2022Estimating Reflectance Layer from A Single Image: Integrating Reflectance Guidance and Shadow/Specular Aware Learningcited by 18
  18. 2020Intrinsic Image Decomposition using Paradigmscited by 17
  19. 2023Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluationcited by 16
  20. 2020Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshadingcited by 12
  21. 2023MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimationcited by 12
  22. 2023UMat: Uncertainty-Aware Single Image High Resolution Material Capturecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 37; 2026: 27.

73 papers.

Sort:

By area:

  1. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  2. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  3. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  4. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priorscited by 28
  5. 2024ScribbleLight: Single Image Indoor Relighting with Scribblescited by 22
  6. 2024IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminationscited by 21
  7. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 18
  8. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  9. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  10. 2024Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusioncited by 15
  11. 2025LightSwitch: Multi-view Relighting with Material-guided Diffusioncited by 15
  12. 2024GS-ID: Illumination Decomposition on Gaussian Splatting via Adaptive Light Aggregation and Diffusion-Guided Material Priorscited by 14
38 more
  1. 2025UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedbackcited by 12
  2. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  3. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  4. 2024SuperMat: Physically Consistent PBR Material Estimation at Interactive Ratescited by 9
  5. 2025PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modelingcited by 9
  6. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  7. 2025MuMA: 3D PBR Texturing via Multi-Channel Multi-View Generation and Agentic Post-Processingcited by 8
  8. 2026Generative World Renderercited by 8
  9. 2025SViM3D: Stable Video Material Diffusion for Single Image 3D Generationcited by 7
  10. 2025Intrinsic Image Fusion for Multi-View 3D Material Reconstructioncited by 7
  11. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 7
  12. 2025MatSwap: Light-aware material transfers in imagescited by 6
  13. 2026LiveLight: Real-time Streaming Video Relighting with Interactive Controlcited by 6
  14. 2025Jodi: Unification of Visual Generation and Understanding via Joint Modelingcited by 5
  15. 2025MVInverse: Feed-forward Multi-view Inverse Rendering in Secondscited by 5
  16. 2026PI-Light: Physics-Inspired Diffusion for Full-Image Relightingcited by 5
  17. 2025InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modelingcited by 4
  18. 2025OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenescited by 4
  19. 2026LightCity: An Urban Dataset for Outdoor Inverse Rendering and Reconstruction under Multi-illumination Conditionscited by 4
  20. 2025ePBR: Extended PBR Materials in Image Synthesiscited by 3
  21. 2025Fine-Grained Spatially Varying Material Selection in Imagescited by 3
  22. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  23. 2026UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priorscited by 3
  24. 2025MatMart: Material Reconstruction of 3D Objects via Diffusioncited by 2
  25. 2025LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Contextcited by 2
  26. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenescited by 1
  27. 2025SAIL: Self-supervised Albedo Estimation from Real Images with a Latent Diffusion Modelcited by 1
  28. 2025Generative Perception of Shape and Material from Differential Motioncited by 1
  29. 2025IntrinsicWeather: Controllable Weather Editing in Intrinsic Spacecited by 1
  30. 2025PFAvatar: Pose-Fusion 3D Personalized Avatar Reconstruction from Real-World Outfit-of-the-Day Photoscited by 1
  31. 2025LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Modelscited by 1
  32. 2025MVID: Feed-Forward Multi-View Intrinsic Image Decompositioncited by 1
  33. 2026FlowIID: Single-Step Intrinsic Image Decomposition via Latent Flow Matchingcited by 1
  34. 2026SGS-Intrinsic: Semantic-Invariant Gaussian Splatting for Sparse-View Indoor Inverse Renderingcited by 1
  35. 2026GOR-IS: 3D Gaussian Object Removal in the Intrinsic Spacecited by 1
  36. 2025Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generationcited by 0
  37. 2025VT-Intrinsic: Physics-Based Decomposition of Reflectance and Shading using a Single Visible-Thermal Image Paircited by 0
  38. 2025PhysHDR: When Lighting Meets Materials and Scene Geometry in HDR Reconstructioncited by 0

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  2. 2024MatFusion: A Generative Diffusion Model for SVBRDF Capture
  3. 2025StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimation
  4. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models
  5. 2023Alchemist: Parametric Control of Material Properties with Diffusion Models
  6. 2025DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion
  7. 2023Material Palette: Extraction of Materials from a Single Image
  8. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Rendering