# MatFusion: A Generative Diffusion Model for SVBRDF Capture

arXiv: https://arxiv.org/abs/2406.06539 · HTML: https://mlsanity.com/paper/2406.06539/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 21 of 21)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks](https://mlsanity.com/paper/1809.00886/) (2018)
- [Flexible SVBRDF Capture with a Multi-Image Deep Network](https://mlsanity.com/paper/1906.11557/) (2019)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [MaterialGAN: Reflectance Capture using a Generative SVBRDF Model](https://mlsanity.com/paper/2010.00114/) (2020)
- [SurfaceNet: Adversarial SVBRDF Estimation from a Single Image](https://mlsanity.com/paper/2107.11298/) (2021)
- [An Inverse Procedural Modeling Pipeline for SVBRDF Maps](https://mlsanity.com/paper/2109.06395/) (2021)
- [Normal Integration: A Survey](https://mlsanity.com/paper/1709.05940/) (2017)
- [MatFormer: A Generative Model for Procedural Materials](https://mlsanity.com/paper/2207.01044/) (2022)
- [A Bayesian Inference Framework for Procedural Material Parameter Estimation](https://mlsanity.com/paper/1912.01067/) (2019)
- [Controlling Material Appearance by Examples](https://mlsanity.com/paper/2206.14970/) (2022)
- [Metappearance: Meta-Learning for Visual Appearance Reproduction](https://mlsanity.com/paper/2204.08993/) (2022)

## Built on by (most cited first, 30 of 31)

- [Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details](https://mlsanity.com/paper/2506.16504/) (2025)
- [DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models](https://mlsanity.com/paper/2405.17176/) (2024)
- [MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors](https://mlsanity.com/paper/2409.15273/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [TexSliders: Diffusion-Based Texture Editing in CLIP Space](https://mlsanity.com/paper/2405.00672/) (2024)
- [Boosting 3D Object Generation through PBR Materials](https://mlsanity.com/paper/2411.16080/) (2024)
- [SAMa: Material-aware 3D Selection and Segmentation](https://mlsanity.com/paper/2411.19322/) (2024)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [ReflectanceFusion: Diffusion-based text to SVBRDF Generation](https://mlsanity.com/paper/2406.14565/) (2024)
- [Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Images](https://mlsanity.com/paper/2509.09952/) (2025)
- [A Comprehensive Survey on Diffusion Models and Their Applications](https://mlsanity.com/paper/2408.10207/) (2024)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptation](https://mlsanity.com/paper/2510.05532/) (2025)
- [MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis](https://mlsanity.com/paper/2511.16957/) (2025)
- [Content-aware Tile Generation using Exterior Boundary Inpainting](https://mlsanity.com/paper/2409.14184/) (2024)
- [LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context](https://mlsanity.com/paper/2511.19437/) (2025)
- [VideoNeuMat: Neural Material Extraction from Generative Video Models](https://mlsanity.com/paper/2602.07272/) (2026)
- [FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Images](https://mlsanity.com/paper/2410.01801/) (2024)
- [Neural Differential Appearance Equations](https://mlsanity.com/paper/2410.07128/) (2024)
- [Single-image Reflectance and Transmittance Estimation from Any Flatbed Scanner](https://mlsanity.com/paper/2502.14462/) (2025)
- [DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion](https://mlsanity.com/paper/2508.05060/) (2025)
- [RealMat: Realistic Materials with Diffusion and Reinforcement Learning](https://mlsanity.com/paper/2509.01134/) (2025)
- [Neural Appearance Modeling From Single Images](https://mlsanity.com/paper/2406.18593/) (2024)
- [HiMat: DiT-based Ultra-High Resolution SVBRDF Generation](https://mlsanity.com/paper/2508.07011/) (2025)
- [MatE: Material Extraction from Single-Image via Geometric Prior](https://mlsanity.com/paper/2512.18312/) (2025)
- [HeatMat: Simulation of City Material Impact on Urban Heat Island Effect](https://mlsanity.com/paper/2601.22796/) (2026)
- [Bridging Rendering and Generative Modeling with Monte Carlo Transport Scheduling](https://mlsanity.com/paper/2602.20725/) (2026)
- [FabricGen: Microstructure-Aware Woven Fabric Generation](https://mlsanity.com/paper/2603.07240/) (2026)
- [HAFMat: Hybrid Priors Guided Adaptive Fusion for Single-Image Human Material Estimation](https://mlsanity.com/paper/2606.16323/) (2026)

## Abstract (arXiv)

We formulate SVBRDF estimation from photographs as a diffusion task. To model the distribution of spatially varying materials, we first train a novel unconditional SVBRDF diffusion backbone model on a large set of 312,165 synthetic spatially varying material exemplars. This SVBRDF diffusion backbone model, named MatFusion, can then serve as a basis for refining a conditional diffusion model to estimate the material properties from a photograph under controlled or uncontrolled lighting. Our backbone MatFusion model is trained using only a loss on the reflectance properties, and therefore refinement can be paired with more expensive rendering methods without the need for backpropagation during training. Because the conditional SVBRDF diffusion models are generative, we can synthesize multiple SVBRDF estimates from the same input photograph from which the user can select the one that best matches the users' expectation. We demonstrate the flexibility of our method by refining different SVBRDF diffusion models conditioned on different types of incident lighting, and show that for a single photograph under colocated flash lighting our method achieves equal or better accuracy than existing SVBRDF estimation methods.