# MatSpray: Fusing 2D Material World Knowledge on 3D Geometry

arXiv: https://arxiv.org/abs/2512.18314 · HTML: https://mlsanity.com/paper/2512.18314/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 20 Dec 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 39)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [DepthSplat: Connecting Gaussian Splatting and Depth](https://mlsanity.com/paper/2410.13862/) (2024)
- [GS-IR: 3D Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2311.16473/) (2023)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes](https://mlsanity.com/paper/2407.07090/) (2024)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis](https://mlsanity.com/paper/2505.09358/) (2025)
- [IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing](https://mlsanity.com/paper/2412.15867/) (2024)
- [NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and Pose Annotations](https://mlsanity.com/paper/2306.09109/) (2023)
- [EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis](https://mlsanity.com/paper/2410.01804/) (2024)
- [GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2410.02619/) (2024)
- [Radiant Foam: Real-Time Differentiable Ray Tracing](https://mlsanity.com/paper/2502.01157/) (2025)
- [IllumiNeRF: 3D Relighting Without Inverse Rendering](https://mlsanity.com/paper/2406.06527/) (2024)
- [SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2504.06815/) (2025)
- [LumiGauss: Relightable Gaussian Splatting in the Wild](https://mlsanity.com/paper/2408.04474/) (2024)
- [LightSwitch: Multi-view Relighting with Material-guided Diffusion](https://mlsanity.com/paper/2508.06494/) (2025)
- [ReCap: Better Gaussian Relighting with Cross-Environment Captures](https://mlsanity.com/paper/2412.07534/) (2024)
- [GlossyGS: Inverse Rendering of Glossy Objects with 3D Gaussian Splatting](https://mlsanity.com/paper/2410.13349/) (2024)
- [PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields](https://mlsanity.com/paper/2412.09680/) (2024)
- [DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Rendering](https://mlsanity.com/paper/2507.03924/) (2025)
- [OccluGaussian: Occlusion-Aware Gaussian Splatting for Large Scene Reconstruction and Rendering](https://mlsanity.com/paper/2503.16177/) (2025)
- [3D Gaussian Inverse Rendering with Approximated Global Illumination](https://mlsanity.com/paper/2504.01358/) (2025)
- [BiGS: Bidirectional Gaussian Primitives for Relightable 3D Gaussian Splatting](https://mlsanity.com/paper/2408.13370/) (2024)

## Built on by (most cited first, 2 of 2)

- [Lighting-Consistent Object Transfer Across Radiance Fields](https://mlsanity.com/paper/2606.22481/) (2026)
- [Luce: Relightable Gaussians for 3D Asset Generation](https://mlsanity.com/paper/2608.23943/) (2026)

## Abstract (arXiv)

Manual modeling of material parameters and 3D geometry is a time consuming yet essential task in the gaming and film industries. While recent advances in 3D reconstruction have enabled accurate approximations of scene geometry and appearance, these methods often fall short in relighting scenarios due to the lack of precise, spatially varying material parameters. At the same time, diffusion models operating on 2D images have shown strong performance in predicting physically based rendering (PBR) properties such as albedo, roughness, and metallicity. However, transferring these 2D material maps onto reconstructed 3D geometry remains a significant challenge. We propose a framework for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches. We begin by reconstructing scene geometry via Gaussian Splatting. From the input images, a diffusion model generates 2D maps for albedo, roughness, and metallic parameters. Any existing diffusion model that can convert images or videos to PBR materials can be applied. The predictions are further integrated into the 3D representation either by optimizing an image-based loss or by directly projecting the material parameters onto the Gaussians using Gaussian ray tracing. To enhance fine-scale accuracy and multi-view consistency, we further introduce a light-weight neural refinement step (Neural Merger), which takes ray-traced material features as input and produces detailed adjustments. Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism. This enables more accurate, relightable, and photorealistic renderings from reconstructed scenes, significantly improving the realism and efficiency of asset creation workflows in content production pipelines.