# Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering

arXiv: https://arxiv.org/abs/2310.00362 · HTML: https://mlsanity.com/paper/2310.00362/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 30 Sep 2023.

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Parallel Diffusion Models of Operator and Image for Blind Inverse Problems](https://mlsanity.com/paper/2211.10656/) (2022)

## Built on by (most cited first, 6 of 6)

- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Materialist: Physically Based Editing Using Single-Image Inverse Rendering](https://mlsanity.com/paper/2501.03717/) (2025)
- [Uncertainty for SVBRDF Acquisition using Frequency Analysis](https://mlsanity.com/paper/2406.17774/) (2024)
- [Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2503.09993/) (2025)
- [DDTR: Diffusion Denoising Trace Recovery](https://mlsanity.com/paper/2510.22553/) (2025)

## Abstract (arXiv)

Inverse rendering, the process of inferring scene properties from images, is a challenging inverse problem. The task is ill-posed, as many different scene configurations can give rise to the same image. Most existing solutions incorporate priors into the inverse-rendering pipeline to encourage plausible solutions, but they do not consider the inherent ambiguities and the multi-modal distribution of possible decompositions. In this work, we propose a novel scheme that integrates a denoising diffusion probabilistic model pre-trained on natural illumination maps into an optimization framework involving a differentiable path tracer. The proposed method allows sampling from combinations of illumination and spatially-varying surface materials that are, both, natural and explain the image observations. We further conduct an extensive comparative study of different priors on illumination used in previous work on inverse rendering. Our method excels in recovering materials and producing highly realistic and diverse environment map samples that faithfully explain the illumination of the input images.