mlsanityData to 7 Oct 2026

SKED: Sketch-guided Text-based 3D Editing

arXiv 2303.10735 ↗First posted 19 Mar 2023cs.CV cs.GRcited by 72

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Text-to-image diffusion models are gradually introduced into computer graphics, recently enabling the development of Text-to-3D pipelines in an open domain. However, for interactive editing purposes, local manipulations of content through a simplistic textual interface can be arduous. Incorporating user guided sketches with Text-to-image pipelines offers users more intuitive control. Still, as state-of-the-art Text-to-3D pipelines rely on optimizing Neural Radiance Fields (NeRF) through gradients from arbitrary rendering views, conditioning on sketches is not straightforward. In this paper, we present SKED, a technique for editing 3D shapes represented by NeRFs. Our technique utilizes as few as two guiding sketches from different views to alter an existing neural field. The edited region respects the prompt semantics through a pre-trained diffusion model. To ensure the generated output adheres to the provided sketches, we propose novel loss functions to generate the desired edits while preserving the density and radiance of the base instance. We demonstrate the effectiveness of our proposed method through several qualitative and quantitative experiments. https://sked-paper.github.io/

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 89 references that are in the map. First posted on arXiv 19 Mar 2023.

Builds on

53 of 89 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models28 Dec
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  3. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  4. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  5. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models25 Nov
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  7. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
33 more
  1. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  2. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  4. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  8. 2020Neural Sparse Voxel Fieldscited by 959
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2020Convolutional Occupancy Networkscited by 760
  12. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  13. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  14. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  15. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  16. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  17. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  18. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  19. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  20. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  21. 2021Editing Conditional Radiance Fieldscited by 222
  22. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 216
  23. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  24. 2022NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editingcited by 141
  25. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  26. 2022Variable Bitrate Neural Fieldscited by 116
  27. 2021NeRF-SR: High-Quality Neural Radiance Fields using Supersamplingcited by 95
  28. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Modelcited by 62
  29. 20173D Shape Reconstruction from Sketches via Multi-view Convolutional Networkscited by 44
  30. 2022PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fieldscited by 43
  31. 2022Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Modelcited by 39
  32. 2022LaTeRF: Label and Text Driven Object Radiance Fieldscited by 38
  33. 20173D Sketching using Multi-View Deep Volumetric Predictioncited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 30; 2025: 13; 2026: 10.

72 papers.

Sort:

By area:

  1. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  2. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 331
  3. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  4. 2024GES: Generalized Exponential Splatting for Efficient Radiance Field Renderingcited by 92
  5. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  6. 2023FocalDreamer: Text-driven 3D Editing via Focal-fusion Assemblycited by 75
  7. 2024View-Consistent 3D Editing with Gaussian Splattingcited by 67
  8. 2024Advances in 3D Generation: A Surveycited by 62
  9. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  10. 2023PIE-NeRF: Physics-based Interactive Elastodynamics with NeRFcited by 51
  11. 2023Watch Your Steps: Local Image and Scene Editing by Text Instructionscited by 50
  12. 2024Instant3dit: Multiview Inpainting for Fast Editing of 3D Objectscited by 43
38 more
  1. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
  2. 2023OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fieldscited by 34
  3. 2025NANO3D: A Training-Free Approach for Efficient 3D Editing Without Maskscited by 29
  4. 2024It's All About Your Sketch: Democratising Sketch Control in Diffusion Modelscited by 28
  5. 2024Make-A-Shape: a Ten-Million-scale 3D Shape Modelcited by 25
  6. 2023Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fieldscited by 23
  7. 20243Doodle: Compact Abstraction of Objects with 3D Strokescited by 21
  8. 2025CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editingcited by 19
  9. 2025EditP23: 3D Editing via Propagation of Image Prompts to Multi-Viewcited by 19
  10. 2023Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fieldscited by 18
  11. 2023DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editingcited by 17
  12. 2023LatentEditor: Text Driven Local Editing of 3D Scenescited by 17
  13. 2025CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generationcited by 14
  14. 2024Text-to-3D Shape Generationcited by 13
  15. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  16. 2024Neural Radiance Field-based Visual Rendering: A Comprehensive Reviewcited by 12
  17. 2024GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Imagecited by 12
  18. 2024GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformerscited by 11
  19. 2024Diffusion Models in 3D Vision: A Surveycited by 11
  20. 2023SketchDreamer: Interactive Text-Augmented Creative Sketch Ideationcited by 10
  21. 2024Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodingscited by 10
  22. 2024External Knowledge Enhanced 3D Scene Generation from Sketchcited by 9
  23. 2023SENS: Part-Aware Sketch-based Implicit Neural Shape Modelingcited by 8
  24. 2024CNS-Edit: 3D Shape Editing via Coupled Neural Shape Optimizationcited by 8
  25. 2024Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakerscited by 8
  26. 2023ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priorscited by 7
  27. 20243D Gaussian Editing with A Single Imagecited by 7
  28. 2024Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generationcited by 6
  29. 2024Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Modelcited by 6
  30. 2024GaussianBlock: Building Part-Aware Compositional and Editable 3D Scene by Primitives and Gaussianscited by 6
  31. 2025MeshPad: Interactive Sketch-Conditioned Artist-Reminiscent Mesh Generation and Editingcited by 6
  32. 2024Freditor: High-Fidelity and Transferable NeRF Editing by Frequency Decompositioncited by 5
  33. 2024DATENeRF: Depth-Aware Text-based Editing of NeRFscited by 5
  34. 2024LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Modelcited by 5
  35. 2025SketchFusion: Learning Universal Sketch Features through Fusing Foundation Modelscited by 4
  36. 2023DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priorscited by 3
  37. 2023MaTe3D: Mask-guided Text-based 3D-aware Portrait Editingcited by 3
  38. 2024SyncNoise: Geometrically Consistent Noise Prediction for Text-based 3D Scene Editingcited by 3

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models
  2. 2024SketchDream: Sketch-based Text-to-3D Generation and Editing
  3. 2023ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF
  4. 2023Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model
  5. 2024ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
  6. 2024Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories
  7. 2024GSEdit: Efficient Text-Guided Editing of 3D Objects via Gaussian Splatting
  8. 2023Control3D: Towards Controllable Text-to-3D Generation