# Control3D: Towards Controllable Text-to-3D Generation

arXiv: https://arxiv.org/abs/2311.05461 · HTML: https://mlsanity.com/paper/2311.05461/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 33)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](https://mlsanity.com/paper/2104.00677/) (2021)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering](https://mlsanity.com/paper/2112.15399/) (2021)
- [Sketch-Guided Text-to-Image Diffusion Models](https://mlsanity.com/paper/2211.13752/) (2022)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis](https://mlsanity.com/paper/1912.05237/) (2019)
- [Contextual Transformer Networks for Visual Recognition](https://mlsanity.com/paper/2107.12292/) (2021)
- [To Create What You Tell: Generating Videos from Captions](https://mlsanity.com/paper/1804.08264/) (2018)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning](https://mlsanity.com/paper/2207.04978/) (2022)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)

## Built on by (most cited first, 30 of 46)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Boosting Diffusion Models with Moving Average Sampling in Frequency Domain](https://mlsanity.com/paper/2403.17870/) (2024)
- [SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer](https://mlsanity.com/paper/2403.17004/) (2024)
- [DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior](https://mlsanity.com/paper/2312.06439/) (2023)
- [Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images](https://mlsanity.com/paper/2407.06191/) (2024)
- [View Selection for 3D Captioning via Diffusion Ranking](https://mlsanity.com/paper/2404.07984/) (2024)
- [Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models](https://mlsanity.com/paper/2409.07452/) (2024)
- [Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning](https://mlsanity.com/paper/2405.08054/) (2024)
- [VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation](https://mlsanity.com/paper/2403.17001/) (2024)
- [MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusion](https://mlsanity.com/paper/2402.14253/) (2024)
- [3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting](https://mlsanity.com/paper/2412.01583/) (2024)
- [Improving Virtual Try-On with Garment-focused Diffusion Models](https://mlsanity.com/paper/2409.08258/) (2024)
- [DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation](https://mlsanity.com/paper/2505.18078/) (2025)
- [Edit360: 2D Image Edits to 3D Assets from Any Angle](https://mlsanity.com/paper/2506.10507/) (2025)
- [Improving Text-guided Object Inpainting with Semantic Pre-inpainting](https://mlsanity.com/paper/2409.08260/) (2024)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [Prompt Refinement with Image Pivot for Text-to-Image Generation](https://mlsanity.com/paper/2407.00247/) (2024)
- [Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing](https://mlsanity.com/paper/2405.04496/) (2024)
- [DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation](https://mlsanity.com/paper/2409.07454/) (2024)
- [TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt](https://mlsanity.com/paper/2410.21299/) (2024)
- [Incorporating Visual Correspondence into Diffusion Model for Virtual Try-On](https://mlsanity.com/paper/2505.16977/) (2025)
- [X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation](https://mlsanity.com/paper/2312.00085/) (2023)
- [BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion](https://mlsanity.com/paper/2401.16764/) (2024)
- [When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability](https://mlsanity.com/paper/2403.00467/) (2024)
- [FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process](https://mlsanity.com/paper/2409.07451/) (2024)
- [X-Oscar: A Progressive Framework for High-quality Text-guided 3D Animatable Avatar Generation](https://mlsanity.com/paper/2405.00954/) (2024)
- [Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation](https://mlsanity.com/paper/2408.05938/) (2024)
- [Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation](https://mlsanity.com/paper/2404.01843/) (2024)
- [SD-πXL: Generating Low-Resolution Quantized Imagery via Score Distillation](https://mlsanity.com/paper/2410.06236/) (2024)
- [Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models](https://mlsanity.com/paper/2501.05839/) (2025)

## Abstract (arXiv)

Recent remarkable advances in large-scale text-to-image diffusion models have inspired a significant breakthrough in text-to-3D generation, pursuing 3D content creation solely from a given text prompt. However, existing text-to-3D techniques lack a crucial ability in the creative process: interactively control and shape the synthetic 3D contents according to users' desired specifications (e.g., sketch). To alleviate this issue, we present the first attempt for text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D, which enhances controllability for users. In particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF, encouraging each view of 3D scene aligned with the given text prompt and hand-drawn sketch. Moreover, we exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene. Such estimated sketch along with each sampled view is further enforced to be geometrically consistent with the given sketch, pursuing better controllable text-to-3D generation. Through extensive experiments, we demonstrate that our proposal can generate accurate and faithful 3D scenes that align closely with the input text prompts and sketches.