# Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering

arXiv: https://arxiv.org/abs/2312.11360 · HTML: https://mlsanity.com/paper/2312.11360/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 66 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)

## Built on by (most cited first, 30 of 84)

- [Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details](https://mlsanity.com/paper/2506.16504/) (2025)
- [Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects](https://mlsanity.com/paper/2407.02430/) (2024)
- [DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models](https://mlsanity.com/paper/2405.17176/) (2024)
- [Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models](https://mlsanity.com/paper/2406.08475/) (2024)
- [Rethinking Score Distillation as a Bridge Between Image Distributions](https://mlsanity.com/paper/2406.09417/) (2024)
- [Material Anything: Generating Materials for Any 3D Object via Diffusion](https://mlsanity.com/paper/2411.15138/) (2024)
- [TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting](https://mlsanity.com/paper/2411.19654/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [SyncTweedies: A General Generative Framework Based on Synchronized Diffusions](https://mlsanity.com/paper/2403.14370/) (2024)
- [Collaborative Control for Geometry-Conditioned PBR Image Generation](https://mlsanity.com/paper/2402.05919/) (2024)
- [3D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation](https://mlsanity.com/paper/2410.18974/) (2024)
- [Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy](https://mlsanity.com/paper/2412.06698/) (2024)
- [MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignment](https://mlsanity.com/paper/2404.02899/) (2024)
- [RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting](https://mlsanity.com/paper/2406.02461/) (2024)
- [StyleTex: Style Image-Guided Texture Generation for 3D Models](https://mlsanity.com/paper/2411.00399/) (2024)
- [Ultraman: Single Image 3D Human Reconstruction with Ultra Speed and Detail](https://mlsanity.com/paper/2403.12028/) (2024)
- [Make-A-Texture: Fast Shape-Aware Texture Generation in 3 Seconds](https://mlsanity.com/paper/2412.07766/) (2024)
- [Multimodal Latent Diffusion Model for Complex Sewing Pattern Generation](https://mlsanity.com/paper/2412.14453/) (2024)
- [Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation](https://mlsanity.com/paper/2312.14124/) (2023)
- [JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers](https://mlsanity.com/paper/2505.00482/) (2025)
- [Boosting 3D Object Generation through PBR Materials](https://mlsanity.com/paper/2411.16080/) (2024)
- [RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturing](https://mlsanity.com/paper/2412.16778/) (2024)
- [VideoMat: Extracting PBR Materials from Video Diffusion Models](https://mlsanity.com/paper/2506.09665/) (2025)
- [SuperMat: Physically Consistent PBR Material Estimation at Interactive Rates](https://mlsanity.com/paper/2411.17515/) (2024)
- [FlexiTex: Enhancing Texture Generation via Visual Guidance](https://mlsanity.com/paper/2409.12431/) (2024)
- [GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation](https://mlsanity.com/paper/2409.18401/) (2024)
- [StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces](https://mlsanity.com/paper/2501.15445/) (2025)
- [MuMA: 3D PBR Texturing via Multi-Channel Multi-View Generation and Agentic Post-Processing](https://mlsanity.com/paper/2503.18461/) (2025)
- [Elevating 3D Models: High-Quality Texture and Geometry Refinement from a Low-Quality Model](https://mlsanity.com/paper/2507.11465/) (2025)
- [RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image](https://mlsanity.com/paper/2410.08181/) (2024)

## Abstract (arXiv)

We present Paint-it, a text-driven high-fidelity texture map synthesis method for 3D meshes via neural re-parameterized texture optimization. Paint-it synthesizes texture maps from a text description by synthesis-through-optimization, exploiting the Score-Distillation Sampling (SDS). We observe that directly applying SDS yields undesirable texture quality due to its noisy gradients. We reveal the importance of texture parameterization when using SDS. Specifically, we propose Deep Convolutional Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes the physically-based rendering (PBR) texture maps with randomly initialized convolution-based neural kernels, instead of a standard pixel-based parameterization. We show that DC-PBR inherently schedules the optimization curriculum according to texture frequency and naturally filters out the noisy signals from SDS. In experiments, Paint-it obtains remarkable quality PBR texture maps within 15 min., given only a text description. We demonstrate the generalizability and practicality of Paint-it by synthesizing high-quality texture maps for large-scale mesh datasets and showing test-time applications such as relighting and material control using a popular graphics engine. Project page: https://kim-youwang.github.io/paint-it