# Consistent Mesh Diffusion

arXiv: https://arxiv.org/abs/2312.00971 · HTML: https://mlsanity.com/paper/2312.00971/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Builds on (most cited first, 30 of 34)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [Texture Fields: Learning Texture Representations in Function Space](https://mlsanity.com/paper/1905.07259/) (2019)
- [3DGen: Triplane Latent Diffusion for Textured Mesh Generation](https://mlsanity.com/paper/2303.05371/) (2023)
- [GAUDI: A Neural Architect for Immersive 3D Scene Generation](https://mlsanity.com/paper/2207.13751/) (2022)
- [TextMesh: Generation of Realistic 3D Meshes From Text Prompts](https://mlsanity.com/paper/2304.12439/) (2023)
- [Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation](https://mlsanity.com/paper/2303.07937/) (2023)
- [3D Shape Induction from 2D Views of Multiple Objects](https://mlsanity.com/paper/1612.05872/) (2016)
- [TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models](https://mlsanity.com/paper/2310.13772/) (2023)
- [Texture Generation on 3D Meshes with Point-UV Diffusion](https://mlsanity.com/paper/2308.10490/) (2023)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)

## Built on by (most cited first, 3 of 3)

- [Collaborative Control for Geometry-Conditioned PBR Image Generation](https://mlsanity.com/paper/2402.05919/) (2024)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [StyleCity: Large-Scale 3D Urban Scenes Stylization](https://mlsanity.com/paper/2404.10681/) (2024)

## Abstract (arXiv)

Given a 3D mesh with a UV parameterization, we introduce a novel approach to generating textures from text prompts. While prior work uses optimization from Text-to-Image Diffusion models to generate textures and geometry, this is slow and requires significant compute resources. Alternatively, there are projection based approaches that use the same Text-to-Image models that paint images onto a mesh, but lack consistency at different viewing angles, we propose a method that uses a single Depth-to-Image diffusion network, and generates a single consistent texture when rendered on the 3D surface by first unifying multiple 2D image's diffusion paths, and hoisting that to 3D with MultiDiffusion~citemultidiffusion. We demonstrate our approach on a dataset containing 30 meshes, taking approximately 5 minutes per mesh. To evaluate the quality of our approach, we use CLIP-score~citeclipscore and Frechet Inception Distance (FID)~citefrechet to evaluate the quality of the rendering, and show our improvement over prior work.