mlsanityData to 7 Oct 2026

Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversion

arXiv 2303.15780 ↗First posted 28 Mar 2023cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

We propose a high-quality 3D-to-3D conversion method, Instruct 3D-to-3D. Our method is designed for a novel task, which is to convert a given 3D scene to another scene according to text instructions. Instruct 3D-to-3D applies pretrained Image-to-Image diffusion models for 3D-to-3D conversion. This enables the likelihood maximization of each viewpoint image and high-quality 3D generation. In addition, our proposed method explicitly inputs the source 3D scene as a condition, which enhances 3D consistency and controllability of how much of the source 3D scene structure is reflected. We also propose dynamic scaling, which allows the intensity of the geometry transformation to be adjusted. We performed quantitative and qualitative evaluations and showed that our proposed method achieves higher quality 3D-to-3D conversions than baseline methods.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

34 of 40 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
22 more
  1. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  2. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  8. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  9. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  10. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  11. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  12. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  13. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  14. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  15. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  16. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  17. 2022ARF: Artistic Radiance Fieldscited by 147
  18. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  19. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  20. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121
  21. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  22. 2022CLIP-based Neural Neighbor Style Transfer for 3D Assetscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 16; 2026: 3.

47 papers.

Sort:

By area:

  1. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  2. 2024ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fieldscited by 68
  3. 2024View-Consistent 3D Editing with Gaussian Splattingcited by 67
  4. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  5. 2023Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Promptscited by 51
  6. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
  7. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  8. 2024HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splattingcited by 33
  9. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimizationcited by 27
  10. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  11. 2024Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusioncited by 24
  12. 2025InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Priorcited by 19
35 more
  1. 2024ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editingcited by 18
  2. 2024Geometry Transfer for Stylizing Radiance Fieldscited by 16
  3. 2024Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Imagescited by 16
  4. 2023InstructP2P: Learning to Edit 3D Point Clouds with Text Instructionscited by 14
  5. 2025Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generationcited by 14
  6. 2024Text-to-3D Shape Generationcited by 13
  7. 2024ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editingcited by 12
  8. 2025Fast Multi-view Consistent 3D Editing with Video Priorscited by 12
  9. 2024TrAME: Trajectory-Anchored Multi-View Editing for Text-Guided 3D Gaussian Splatting Manipulationcited by 10
  10. 2025Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusioncited by 9
  11. 2023O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Modelcited by 8
  12. 2024PEGASUS: Personalized Generative 3D Avatars with Composable Attributescited by 7
  13. 2024Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generationcited by 6
  14. 2024ConRF: Zero-shot Stylization of 3D Scenes with Conditioned Radiation Fieldscited by 5
  15. 2025SplatPainter: Interactive Authoring of 3D Gaussians from 2D Edits via Test-Time Trainingcited by 5
  16. 2023Scene-Conditional 3D Object Stylization and Compositioncited by 4
  17. 2024InstructHumans: Editing Animated 3D Human Textures with Instructionscited by 4
  18. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  19. 2024DragTex: Generative Point-Based Texture Editing on 3D Meshcited by 3
  20. 2024SyncNoise: Geometrically Consistent Noise Prediction for Text-based 3D Scene Editingcited by 3
  21. 2025GaussianAvatar-Editor: Photorealistic Animatable Gaussian Head Avatar Editorcited by 3
  22. 2025DisCo3D: Distilling Multi-View Consistency for 3D Scene Editingcited by 3
  23. 2026Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editingcited by 3
  24. 2024Towards a Training Free Approach for 3D Scene Editingcited by 2
  25. 2025Personalize Your Gaussian: Consistent 3D Scene Personalization from a Single Imagecited by 2
  26. 2024EVLM: Self-Reflective Multimodal Reasoning for Cross-Dimensional Visual Editingcited by 1
  27. 2025Instruct-4DGS: Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separationcited by 1
  28. 2025Towards Generalized and Training-Free Text-Guided Semantic Manipulationcited by 1
  29. 2025InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalizationcited by 1
  30. 20253D-Consistent Multi-View Editing by Correspondence Guidancecited by 1
  31. 2026RelaxFlow: Text-Driven Amodal 3D Generationcited by 1
  32. 2024Exploration and Improvement of Nerf-based 3D Scene Editing Techniquescited by 0
  33. 2025PSF-4D: A Progressive Sampling Framework for View Consistent 4D Editingcited by 0
  34. 2025InStyle: Instant Appearance Stylization of 3D Shapescited by 0
  35. 2026CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
  2. 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
  3. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  4. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
  5. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
  6. 2023Control3D: Towards Controllable Text-to-3D Generation
  7. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
  8. 2023ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors