# PBR3DGen: A VLM-guided Mesh Generation with High-quality PBR Texture

arXiv: https://arxiv.org/abs/2503.11368 · HTML: https://mlsanity.com/paper/2503.11368/

## Key facts

- Cited by 1 papers in the map (data through 7 Oct 2026).
- Builds on 39 of its 48 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [TripoSR: Fast 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/2403.02151/) (2024)
- [MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion](https://mlsanity.com/paper/2307.01097/) (2023)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Flexible Isosurface Extraction for Gradient-Based Mesh Optimization](https://mlsanity.com/paper/2308.05371/) (2023)
- [MeshLRM: Large Reconstruction Model for High-Quality Meshes](https://mlsanity.com/paper/2404.12385/) (2024)
- [Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models](https://mlsanity.com/paper/2312.13913/) (2023)
- [3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion](https://mlsanity.com/paper/2409.12957/) (2024)
- [Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention](https://mlsanity.com/paper/2405.11616/) (2024)
- [TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models](https://mlsanity.com/paper/2310.13772/) (2023)

## Built on by (most cited first, 1 of 1)

- [MatLat: Material Latent Space for PBR Texture Generation](https://mlsanity.com/paper/2512.17302/) (2025)

## Abstract (arXiv)

Generating high-quality physically based rendering (PBR) materials is important to achieve realistic rendering in the downstream tasks, yet it remains challenging due to the intertwined effects of materials and lighting. While existing methods have made breakthroughs by incorporating material decomposition in the 3D generation pipeline, they tend to bake highlights into albedo and ignore spatially varying properties of metallicity and roughness. In this work, we present PBR3DGen, a two-stage mesh generation method with high-quality PBR materials that integrates the novel multi-view PBR material estimation model and a 3D PBR mesh reconstruction model. Specifically, PBR3DGen leverages vision language models (VLM) to guide multi-view diffusion, precisely capturing the spatial distribution and inherent attributes of reflective-metalness material. Additionally, we incorporate view-dependent illumination-aware conditions as pixel-aware priors to enhance spatially varying material properties. Furthermore, our reconstruction model reconstructs high-quality mesh with PBR materials. Experimental results demonstrate that PBR3DGen significantly outperforms existing methods, achieving new state-of-the-art results for PBR estimation and mesh generation. More results and visualization can be found on our project page: https://pbr3dgen1218.github.io/.