# MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models

arXiv: https://arxiv.org/abs/2501.15981 · HTML: https://mlsanity.com/paper/2501.15981/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering](https://mlsanity.com/paper/2312.11360/) (2023)
- [FlashTex: Fast Relightable Mesh Texturing with LightControlNet](https://mlsanity.com/paper/2402.13251/) (2024)
- [Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects](https://mlsanity.com/paper/2407.02430/) (2024)
- [ControlMat: A Controlled Generative Approach to Material Capture](https://mlsanity.com/paper/2309.01700/) (2023)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [MatFuse: Controllable Material Generation with Diffusion Models](https://mlsanity.com/paper/2308.11408/) (2023)
- [MaterialGAN: Reflectance Capture using a Generative SVBRDF Model](https://mlsanity.com/paper/2010.00114/) (2020)
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](https://mlsanity.com/paper/2404.17569/) (2024)
- [SurfaceNet: Adversarial SVBRDF Estimation from a Single Image](https://mlsanity.com/paper/2107.11298/) (2021)
- [TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling](https://mlsanity.com/paper/2408.01291/) (2024)
- [PSDR-Room: Single Photo to Scene using Differentiable Rendering](https://mlsanity.com/paper/2307.03244/) (2023)
- [PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes](https://mlsanity.com/paper/2207.00757/) (2022)
- [PhotoShape: Photorealistic Materials for Large-Scale Shape Collections](https://mlsanity.com/paper/1809.09761/) (2018)
- [3DCoMPaT++: An improved Large-scale 3D Vision Dataset for Compositional Recognition](https://mlsanity.com/paper/2310.18511/) (2023)
- [MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment](https://mlsanity.com/paper/2404.02899/) (2024)
- [StyleTex: Style Image-Guided Texture Generation for 3D Models](https://mlsanity.com/paper/2411.00399/) (2024)
- [SAMa: Material-aware 3D Selection and Segmentation](https://mlsanity.com/paper/2411.19322/) (2024)
- [FlexiTex: Enhancing Texture Generation via Visual Guidance](https://mlsanity.com/paper/2409.12431/) (2024)
- [The Visual Language of Fabrics](https://mlsanity.com/paper/2307.13681/) (2023)

## Built on by (most cited first, 2 of 2)

- [LaFiTe: A Generative Latent Field for 3D Native Texturing](https://mlsanity.com/paper/2512.04786/) (2025)
- [Phieat: Physically Grounded Material Feature Representation](https://mlsanity.com/paper/2511.11270/) (2025)

## Abstract (arXiv)

Assigning realistic materials to 3D models remains a significant challenge in computer graphics. We propose MatCLIP, a novel method that extracts shape- and lighting-insensitive descriptors of Physically Based Rendering (PBR) materials to assign plausible textures to 3D objects based on images, such as the output of Latent Diffusion Models (LDMs) or photographs. Matching PBR materials to static images is challenging because the PBR representation captures the dynamic appearance of materials under varying viewing angles, shapes, and lighting conditions. By extending an Alpha-CLIP-based model on material renderings across diverse shapes and lighting, and encoding multiple viewing conditions for PBR materials, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings, including LDM outputs. This enables consistent material assignments without requiring explicit knowledge of material relationships between different parts of an object. MatCLIP achieves a top-1 classification accuracy of 76.6%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas by over 15 percentage points on publicly available datasets. Our method can be used to construct material assignments for 3D shape datasets such as ShapeNet, 3DCoMPaT++, and Objaverse. All code and data will be released.