# MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

arXiv: https://arxiv.org/abs/2308.09278 · HTML: https://mlsanity.com/paper/2308.09278/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer](https://mlsanity.com/paper/1807.07543/) (2018)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Two-shot Spatially-varying BRDF and Shape Estimation](https://mlsanity.com/paper/2004.00403/) (2020)

## Built on by (most cited first, 30 of 30)

- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D](https://mlsanity.com/paper/2311.16918/) (2023)
- [Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details](https://mlsanity.com/paper/2506.16504/) (2025)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials](https://mlsanity.com/paper/2407.02445/) (2024)
- [FlashTex: Fast Relightable Mesh Texturing with LightControlNet](https://mlsanity.com/paper/2402.13251/) (2024)
- [DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models](https://mlsanity.com/paper/2405.17176/) (2024)
- [UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation](https://mlsanity.com/paper/2312.08754/) (2023)
- [Posterior Distillation Sampling](https://mlsanity.com/paper/2311.13831/) (2023)
- [Material Anything: Generating Materials for Any 3D Object via Diffusion](https://mlsanity.com/paper/2411.15138/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance](https://mlsanity.com/paper/2312.08889/) (2023)
- [Collaborative Control for Geometry-Conditioned PBR Image Generation](https://mlsanity.com/paper/2402.05919/) (2024)
- [InTeX: Interactive Text-to-texture Synthesis via Unified Depth-aware Inpainting](https://mlsanity.com/paper/2403.11878/) (2024)
- [Diffusion Time-step Curriculum for One Image to 3D Generation](https://mlsanity.com/paper/2404.04562/) (2024)
- [A Survey On Text-to-3D Contents Generation In The Wild](https://mlsanity.com/paper/2405.09431/) (2024)
- [Edify 3D: Scalable High-Quality 3D Asset Generation](https://mlsanity.com/paper/2411.07135/) (2024)
- [Boosting 3D Object Generation through PBR Materials](https://mlsanity.com/paper/2411.16080/) (2024)
- [SuperMat: Physically Consistent PBR Material Estimation at Interactive Rates](https://mlsanity.com/paper/2411.17515/) (2024)
- [GetMesh: A Controllable Model for High-quality Mesh Generation and Manipulation](https://mlsanity.com/paper/2403.11990/) (2024)
- [ARM: Appearance Reconstruction Model for Relightable 3D Generation](https://mlsanity.com/paper/2411.10825/) (2024)
- [Birth and Death of a Rose](https://mlsanity.com/paper/2412.05278/) (2024)
- [SViM3D: Stable Video Material Diffusion for Single Image 3D Generation](https://mlsanity.com/paper/2510.08271/) (2025)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [Meta 3D Gen](https://mlsanity.com/paper/2407.02599/) (2024)
- [Content-aware Tile Generation using Exterior Boundary Inpainting](https://mlsanity.com/paper/2409.14184/) (2024)
- [MVLight: Relightable Text-to-3D Generation via Light-conditioned Multi-View Diffusion](https://mlsanity.com/paper/2411.11475/) (2024)
- [NI-Tex: Non-isometric Image-based Garment Texture Generation](https://mlsanity.com/paper/2511.18765/) (2025)
- [LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context](https://mlsanity.com/paper/2511.19437/) (2025)
- [DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation](https://mlsanity.com/paper/2509.07435/) (2025)

## Abstract (arXiv)

Based on powerful text-to-image diffusion models, text-to-3D generation has made significant progress in generating compelling geometry and appearance. However, existing methods still struggle to recover high-fidelity object materials, either only considering Lambertian reflectance, or failing to disentangle BRDF materials from the environment lights. In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (MATLABER) that leverages a novel latent BRDF auto-encoder for material generation. We train this auto-encoder with large-scale real-world BRDF collections and ensure the smoothness of its latent space, which implicitly acts as a natural distribution of materials. During appearance modeling in text-to-3D generation, the latent BRDF embeddings, rather than BRDF parameters, are predicted via a material network. Through exhaustive experiments, our approach demonstrates the superiority over existing ones in generating realistic and coherent object materials. Moreover, high-quality materials naturally enable multiple downstream tasks such as relighting and material editing. Code and model will be publicly available at urlhttps://sheldontsui.github.io/projects/Matlaber.