mlsanityData to 7 Oct 2026

Control3D: Towards Controllable Text-to-3D Generation

arXiv 2311.05461 ↗First posted 9 Nov 2023cs.CV cs.MMcited by 46

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent remarkable advances in large-scale text-to-image diffusion models have inspired a significant breakthrough in text-to-3D generation, pursuing 3D content creation solely from a given text prompt. However, existing text-to-3D techniques lack a crucial ability in the creative process: interactively control and shape the synthetic 3D contents according to users' desired specifications (e.g., sketch). To alleviate this issue, we present the first attempt for text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D, which enhances controllability for users. In particular, a 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF, encouraging each view of 3D scene aligned with the given text prompt and hand-drawn sketch. Moreover, we exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene. Such estimated sketch along with each sampled view is further enforced to be geometrically consistent with the given sketch, pursuing better controllable text-to-3D generation. Through extensive experiments, we demonstrate that our proposal can generate accurate and faithful 3D scenes that align closely with the input text prompts and sketches.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 33 of its 45 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

33 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 31

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  12. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
19 more
  1. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  2. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  3. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  4. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  5. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  6. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  7. 2021InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Renderingcited by 180
  8. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  9. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  10. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  11. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  12. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  13. 2018SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesiscited by 81
  14. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  15. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  16. 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 67
  17. 2021Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketchescited by 50
  18. 2019Mocycle-GAN: Unpaired Video-to-Video Translationcited by 37
  19. 2017Image Generation from Sketch Constraint Using Contextual GANcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 27; 2025: 10; 2026: 6.

46 papers.

Sort:

By area:

  1. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  2. 2024Boosting Diffusion Models with Moving Average Sampling in Frequency Domaincited by 53
  3. 2024SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformercited by 47
  4. 2023DreamControl: Control-Based Text-to-3D Generation with 3D Self-Priorcited by 39
  5. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 34
  6. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  7. 2024Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Modelscited by 27
  8. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  9. 2024VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generationcited by 25
  10. 2024MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusioncited by 22
  11. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 20
  12. 2024Improving Virtual Try-On with Garment-focused Diffusion Modelscited by 15
34 more
  1. 2025Edit360: 2D Image Edits to 3D Assets from Any Anglecited by 13
  2. 2024Improving Text-guided Object Inpainting with Semantic Pre-inpaintingcited by 12
  3. 2025Neural Radiance Fields for the Real World: A Surveycited by 12
  4. 2025DanceTogether! Identity-Preserving Multi-Person Interactive Video Generationcited by 12
  5. 2024Prompt Refinement with Image Pivot for Text-to-Image Generationcited by 11
  6. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  7. 2024DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generationcited by 9
  8. 2024TV-3DG: Mastering Text-to-3D Customized Generation with Visual Promptcited by 8
  9. 2025Incorporating Visual Correspondence into Diffusion Model for Virtual Try-Oncited by 8
  10. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generationcited by 7
  11. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusioncited by 6
  12. 2024When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Abilitycited by 6
  13. 2024FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Processcited by 6
  14. 2024X-Oscar: A Progressive Framework for High-quality Text-guided 3D Animatable Avatar Generationcited by 5
  15. 2024Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generationcited by 5
  16. 2024Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generationcited by 4
  17. 2024SD-πXL: Generating Low-Resolution Quantized Imagery via Score Distillationcited by 3
  18. 2025Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Modelscited by 3
  19. 2025PlantDreamer: Achieving Realistic 3D Plant Models with Diffusion-Guided Gaussian Splattingcited by 3
  20. 2026Deep Sketch-Based 3D Modeling: A Surveycited by 3
  21. 2024COMOGen: A Controllable Text-to-3D Multi-object Generation Frameworkcited by 2
  22. 2024SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generationcited by 2
  23. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generationcited by 1
  24. 2025DreamJourney: Perpetual View Generation with Video Diffusion Modelscited by 1
  25. 2025Structural Energy-Guided Sampling for View-Consistent Text-to-3Dcited by 1
  26. 2025Controllable 3D Object Generation with Single Image Promptcited by 1
  27. 2026ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splattingcited by 1
  28. 2026Beyond Prompts: Unconditional 3D Inversion for Out-of-Distribution Shapescited by 1
  29. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  30. 2024YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animalscited by 0
  31. 2024ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplarcited by 0
  32. 2025SplatFont3D: Structure-Aware Text-to-3D Artistic Font Generation with Part-Level Style Controlcited by 0
  33. 2026Reconstruction of a 3D wireframe from a single line drawing via generative depth estimationcited by 0
  34. 2026Structural Energy Guidance for View-Consistent Text-to-3D Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation
  2. 2024SketchDream: Sketch-based Text-to-3D Generation and Editing
  3. 2024Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting
  4. 2023ControlDreamer: Blending Geometry and Style in Text-to-3D
  5. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  6. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creation
  7. 2024Training-Free Sketch-Guided Diffusion with Latent Optimization
  8. 2023MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generation