mlsanityData to 7 Oct 2026

InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions

arXiv 2306.07154 ↗First posted 12 Jun 2023cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Enhancing AI systems to perform tasks following human instructions can significantly boost productivity. In this paper, we present InstructP2P, an end-to-end framework for 3D shape editing on point clouds, guided by high-level textual instructions. InstructP2P extends the capabilities of existing methods by synergizing the strengths of a text-conditioned point cloud diffusion model, Point-E, and powerful language models, enabling color and geometry editing using language instructions. To train InstructP2P, we introduce a new shape editing dataset, constructed by integrating a shape segmentation dataset, off-the-shelf shape programs, and diverse edit instructions generated by a large language model, ChatGPT. Our proposed method allows for editing both color and geometry of specific regions in a single forward pass, while leaving other regions unaffected. In our experiments, InstructP2P shows generalization capabilities, adapting to novel shape categories and instructions, despite being trained on a limited amount of data.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 55 of its 70 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

55 of 70 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  8. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  9. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  11. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  12. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  13. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  14. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  15. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  16. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  17. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  18. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  19. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  20. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  21. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  22. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  23. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  24. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  25. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  26. 2023TextDeformer: Geometry Manipulation using Text Guidancecited by 94
  27. 2023PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstructioncited by 84
  28. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  29. 2020DualSDF: Semantic Shape Manipulation using a Two-Level Representationcited by 82
  30. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  31. 2023Shape-aware Text-driven Layered Video Editingcited by 54
  32. 2023Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversioncited by 47
  33. 2023Controllable Mesh Generation Through Sparse Latent Point Diffusion Modelscited by 40
  34. 2023X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidancecited by 37
  35. 2021DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinatescited by 36
  36. 2023DiffFacto: Controllable Part-Based 3D Point Cloud Generation with Cross Diffusioncited by 32
  37. 2023Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and Manipulationcited by 27
  38. 2023DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulationcited by 27

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 4; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 166
  2. 2024Large Multimodal Agents: A Surveycited by 99
  3. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  4. 2023Alchemist: Parametric Control of Material Properties with Diffusion Modelscited by 38
  5. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  6. 2025NANO3D: A Training-Free Approach for Efficient 3D Editing Without Maskscited by 29
  7. 2024PrEditor3D: Fast and Precise 3D Shape Editingcited by 27
  8. 2024Geometry Cloak: Preventing TGS-based 3D Reconstruction from Copyrighted Imagescited by 15
  9. 2024Diffusion-Based Attention Warping for Consistent 3D Scene Editingcited by 7
  10. 2025MeshPad: Interactive Sketch-Conditioned Artist-Reminiscent Mesh Generation and Editingcited by 6
  11. 2024HANDI: Hand-Centric Text-and-Image Conditioned Video Generationcited by 5
  12. 2025DisCo3D: Distilling Multi-View Consistency for 3D Scene Editingcited by 3
2 more
  1. 2026Vinedresser3D: Agentic Text-guided 3D Editingcited by 3
  2. 2025InStyle: Instant Appearance Stylization of 3D Shapescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Blended Point Cloud Diffusion for Localized Text-guided Shape Editing
  2. 2026ShapeUP: Scalable Image-Conditioned 3D Editing
  3. 20253D-LATTE: Latent Space 3D Editing from Textual Instructions
  4. 2026PatchAlign3D: Local Feature Alignment for Dense 3D Shape Understanding
  5. 2024SegPoint: Segment Any Point Cloud via Large Language Model
  6. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  7. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
  8. 2026SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision