# Extracting Neural Materials from Images

arXiv: https://arxiv.org/abs/2606.26715 · HTML: https://mlsanity.com/paper/2606.26715/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 71 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 30 of 44)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces](https://mlsanity.com/paper/2311.17977/) (2023)
- [GS-IR: 3D Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2311.16473/) (2023)
- [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Native and Compact Structured Latents for 3D Generation](https://mlsanity.com/paper/2512.14692/) (2025)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material](https://mlsanity.com/paper/2506.15442/) (2025)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing](https://mlsanity.com/paper/2311.16043/) (2023)
- [MeshLRM: Large Reconstruction Model for High-Quality Meshes](https://mlsanity.com/paper/2404.12385/) (2024)
- [Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering](https://mlsanity.com/paper/2010.09125/) (2020)
- [Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering](https://mlsanity.com/paper/2312.11360/) (2023)
- [Intrinsic Image Diffusion for Indoor Single-view Material Estimation](https://mlsanity.com/paper/2312.12274/) (2023)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials](https://mlsanity.com/paper/2407.02445/) (2024)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [Two-shot Spatially-varying BRDF and Shape Estimation](https://mlsanity.com/paper/2004.00403/) (2020)
- [DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer](https://mlsanity.com/paper/2111.00140/) (2021)
- [TexVerse: A Universe of 3D Objects with High-Resolution Textures](https://mlsanity.com/paper/2508.10868/) (2025)
- [Relighting Neural Radiance Fields with Shadow and Highlight Hints](https://mlsanity.com/paper/2308.13404/) (2023)

## Abstract (arXiv)

Neural materials can represent complex specular reflections and scattering effects in a compact, universal basis. However, acquiring and authoring such materials remains challenging. We present NeuMatEx, a differentiable inverse rendering method for extracting spatially varying neural materials from images. The nonlinear structure of neural material latent spaces makes optimization with naive inverse rendering infeasible. To address this, we train a Large Material Reconstruction Model (LMRM) that directly predicts initial base color, neural material latents, and aleatoric uncertainty guides from images. This material prior provides a good initialization and better constrains our subsequent optimization using inverse path tracing. The predicted uncertainty further helps by anchoring high-confidence regions more tightly to the LMRM prediction, preventing lighting and complex specular effects from being baked into materials. Experiments on synthetic and real assets show that NeuMatEx extracts complex materials with better visual quality and material decomposition than PBR-based methods.