# 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models arXiv: https://arxiv.org/abs/2311.05464 ยท HTML: https://mlsanity.com/paper/2311.05464/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. - Builds on 36 of its 50 references that are in the map. - First posted on arXiv 9 Nov 2023. ## Placed in - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 30 of 36) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014) - [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022) - [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016) - [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022) - [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021) - [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022) - [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017) - [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021) - [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021) - [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022) - [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022) - [Contextual Transformer Networks for Visual Recognition](https://mlsanity.com/paper/2107.12292/) (2021) - [To Create What You Tell: Generating Videos from Captions](https://mlsanity.com/paper/1804.08264/) (2018) ## Built on by (most cited first, 17 of 17) - [Boosting Diffusion Models with Moving Average Sampling in Frequency Domain](https://mlsanity.com/paper/2403.17870/) (2024) - [SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer](https://mlsanity.com/paper/2403.17004/) (2024) - [BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing](https://mlsanity.com/paper/2504.01786/) (2025) - [Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models](https://mlsanity.com/paper/2409.07452/) (2024) - [VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation](https://mlsanity.com/paper/2403.17001/) (2024) - [BlenderAlchemy: Editing 3D Graphics with Vision-Language Models](https://mlsanity.com/paper/2404.17672/) (2024) - [Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing](https://mlsanity.com/paper/2405.04496/) (2024) - [DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation](https://mlsanity.com/paper/2409.07454/) (2024) - [Advances in 3D Neural Stylization: A Survey](https://mlsanity.com/paper/2311.18328/) (2023) - [X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation](https://mlsanity.com/paper/2312.00085/) (2023) - [UVMap-ID: A Controllable and Personalized UV Map Generative Model](https://mlsanity.com/paper/2404.14568/) (2024) - [FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process](https://mlsanity.com/paper/2409.07451/) (2024) - [ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing](https://mlsanity.com/paper/2501.02064/) (2025) - [StyleCity: Large-Scale 3D Urban Scenes Stylization](https://mlsanity.com/paper/2404.10681/) (2024) - [DreamJourney: Perpetual View Generation with Video Diffusion Models](https://mlsanity.com/paper/2506.17705/) (2025) - [ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splatting](https://mlsanity.com/paper/2602.10278/) (2026) - [Decompose Radicals, Then Reward: Fine-Grained Inspection for Accurate Chinese Text Rendering](https://mlsanity.com/paper/2609.37569/) (2026) ## Abstract (arXiv) 3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. Recent advances of cross-modal foundation models (e.g., CLIP) have made this problem feasible. Those approaches commonly leverage CLIP to align the holistic semantics of stylized mesh with the given text prompt. Nevertheless, it is not trivial to enable more controllable stylization of fine-grained details in 3D meshes solely based on such semantic-level cross-modal supervision. In this work, we propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models. Technically, 3DStyle-Diffusion first parameterizes the texture of 3D mesh into reflectance properties and scene lighting using implicit MLP networks. Meanwhile, an accurate depth map of each sampled view is achieved conditioned on 3D mesh. Then, 3DStyle-Diffusion leverages a pre-trained controllable 2D Diffusion model to guide the learning of rendered images, encouraging the synthesized image of each view semantically aligned with text prompt and geometrically consistent with depth map. This way elegantly integrates both image rendering via implicit MLP networks and diffusion process of image synthesis in an end-to-end fashion, enabling a high-quality fine-grained stylization of 3D meshes. We also build a new dataset derived from Objaverse and the evaluation protocol for this task. Through both qualitative and quantitative experiments, we validate the capability of our 3DStyle-Diffusion. Source code and data are available at urlhttps://github.com/yanghb22-fdu/3DStyle-Diffusion-Official.