# Multistable Shape from Shading Emerges from Patch Diffusion

arXiv: https://arxiv.org/abs/2405.14530 · HTML: https://mlsanity.com/paper/2405.14530/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 62 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models](https://mlsanity.com/paper/2207.14626/) (2022)
- [Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency](https://mlsanity.com/paper/2307.08123/) (2023)
- [Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models](https://mlsanity.com/paper/2304.12526/) (2023)
- [SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions](https://mlsanity.com/paper/2306.05178/) (2023)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Restart Sampling for Improving Generative Processes](https://mlsanity.com/paper/2306.14878/) (2023)
- [Patched Denoising Diffusion Models For High-Resolution Image Synthesis](https://mlsanity.com/paper/2308.01316/) (2023)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [Universal Photometric Stereo Network using Global Lighting Contexts](https://mlsanity.com/paper/2206.02452/) (2022)
- [From Shading to Local Shape](https://mlsanity.com/paper/1310.2916/) (2013)
- [Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance](https://mlsanity.com/paper/2312.04529/) (2023)

## Built on by (most cited first, 6 of 6)

- [ResearchTown: Simulator of Human Research Community](https://mlsanity.com/paper/2412.17767/) (2024)
- [GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structure](https://mlsanity.com/paper/2512.22274/) (2025)
- [Filterless Snapshot Hyperspectral Imaging using Guided Patch Diffusion](https://mlsanity.com/paper/2412.02798/) (2024)
- [Generative Perception of Shape and Material from Differential Motion](https://mlsanity.com/paper/2506.02473/) (2025)
- [Under One Sun: Multi-Object Generative Perception of Materials and Illumination](https://mlsanity.com/paper/2603.19226/) (2026)
- [A Theory of Shape Reconstruction from Heat Conduction and Shading](https://mlsanity.com/paper/2610.04052/) (2026)

## Abstract (arXiv)

Models for inferring monocular shape of surfaces with diffuse reflection -- shape from shading -- ought to produce distributions of outputs, because there are fundamental mathematical ambiguities of both continuous (e.g., bas-relief) and discrete (e.g., convex/concave) types that are also experienced by humans. Yet, the outputs of current models are limited to point estimates or tight distributions around single modes, which prevent them from capturing these effects. We introduce a model that reconstructs a multimodal distribution of shapes from a single shading image, which aligns with the human experience of multistable perception. We train a small denoising diffusion process to generate surface normal fields from 16× 16 patches of synthetic images of everyday 3D objects. We deploy this model patch-wise at multiple scales, with guidance from inter-patch shape consistency constraints. Despite its relatively small parameter count and predominantly bottom-up structure, we show that multistable shape explanations emerge from this model for ambiguous test images that humans experience as being multistable. At the same time, the model produces veridical shape estimates for object-like images that include distinctive occluding contours and appear less ambiguous. This may inspire new architectures for stochastic 3D shape perception that are more efficient and better aligned with human experience.