# SketchDream: Sketch-based Text-to-3D Generation and Editing

arXiv: https://arxiv.org/abs/2405.06461 · HTML: https://mlsanity.com/paper/2405.06461/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2024.

## Built on by (most cited first, 25 of 25)

- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks](https://mlsanity.com/paper/2510.15019/) (2025)
- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [MeshPad: Interactive Sketch-Conditioned Artist-Reminiscent Mesh Generation and Editing](https://mlsanity.com/paper/2503.01425/) (2025)
- [Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model](https://mlsanity.com/paper/2409.16938/) (2024)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [Sketch-guided Cage-based 3D Gaussian Splatting Deformation](https://mlsanity.com/paper/2411.12168/) (2024)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [COMOGen: A Controllable Text-to-3D Multi-object Generation Framework](https://mlsanity.com/paper/2409.00590/) (2024)
- [SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generation](https://mlsanity.com/paper/2410.07658/) (2024)
- [Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing](https://mlsanity.com/paper/2508.13797/) (2025)
- [Beyond Voxel 3D Editing: Learning from 3D Masks and Self-Constructed Data](https://mlsanity.com/paper/2604.13688/) (2026)
- [Text2VDM: Text to Vector Displacement Maps for Expressive and Interactive 3D Sculpting](https://mlsanity.com/paper/2502.20045/) (2025)
- [Mono4DEditor: Text-Driven 4D Scene Editing from Monocular Video via Point-Level Localization of Language-Embedded Gaussians](https://mlsanity.com/paper/2510.09438/) (2025)
- [ScribbleSense: Generative Scribble-Based Texture Editing with Intent Prediction](https://mlsanity.com/paper/2601.22455/) (2026)
- [DancingBox: A Lightweight MoCap System for Character Animation from Physical Proxies](https://mlsanity.com/paper/2603.17704/) (2026)
- [EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learning](https://mlsanity.com/paper/2607.07187/) (2026)
- [ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar](https://mlsanity.com/paper/2412.16253/) (2024)
- [Dragen3D: Multiview Geometry Consistent 3D Gaussian Generation with Drag-Based Control](https://mlsanity.com/paper/2502.16475/) (2025)
- [SplatFont3D: Structure-Aware Text-to-3D Artistic Font Generation with Part-Level Style Control](https://mlsanity.com/paper/2512.00413/) (2025)
- [SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching](https://mlsanity.com/paper/2512.22016/) (2025)
- [Reconstruction of a 3D wireframe from a single line drawing via generative depth estimation](https://mlsanity.com/paper/2604.13549/) (2026)
- [Geometrically Consistent Multi-View Scene Generation from Freehand Sketches](https://mlsanity.com/paper/2604.14302/) (2026)
- [Learning 3D Editing without Paired Supervision via Generative Prior Distillation](https://mlsanity.com/paper/2609.04942/) (2026)

## Abstract (arXiv)

Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.