# Generative Detail Enhancement for Physically Based Materials

arXiv: https://arxiv.org/abs/2502.13994 · HTML: https://mlsanity.com/paper/2502.13994/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 8 of 8)

- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Text-Guided Texturing by Synchronized Multi-View Diffusion](https://mlsanity.com/paper/2311.12891/) (2023)
- [ControlMat: A Controlled Generative Approach to Material Capture](https://mlsanity.com/paper/2309.01700/) (2023)
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](https://mlsanity.com/paper/2404.17569/) (2024)
- [TexSliders: Diffusion-Based Texture Editing in CLIP Space](https://mlsanity.com/paper/2405.00672/) (2024)
- [Consolidating Attention Features for Multi-view Image Editing](https://mlsanity.com/paper/2402.14792/) (2024)

## Built on by (most cited first, 2 of 2)

- [Example-Based Feature Painting on Textures](https://mlsanity.com/paper/2511.01513/) (2025)
- [Texture Space Material Diffusion](https://mlsanity.com/paper/2609.37654/) (2026)

## Abstract (arXiv)

We present a tool for enhancing the detail of physically based materials using an off-the-shelf diffusion model and inverse rendering. Our goal is to enhance the visual fidelity of materials with detail that is often tedious to author, by adding signs of wear, aging, weathering, etc. As these appearance details are often rooted in real-world processes, we leverage a generative image model trained on a large dataset of natural images with corresponding visuals in context. Starting with a given geometry, UV mapping, and basic appearance, we render multiple views of the object. We use these views, together with an appearance-defining text prompt, to condition a diffusion model. The details it generates are then backpropagated from the enhanced images to the material parameters via inverse differentiable rendering. For inverse rendering to be successful, the generated appearance has to be consistent across all the images. We propose two priors to address the multi-view consistency of the diffusion model. First, we ensure that the initial noise that seeds the diffusion process is itself consistent across views by integrating it from a view-independent UV space. Second, we enforce geometric consistency by biasing the attention mechanism via a projective constraint so that pixels attend strongly to their corresponding pixel locations in other views. Our approach does not require any training or finetuning of the diffusion model, is agnostic of the material model used, and the enhanced material properties, i.e., 2D PBR textures, can be further edited by artists. This project is available at https://generative-detail.github.io.