mlsanityData to 7 Oct 2026

CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language

arXiv 2211.01427 ↗First posted 2 Nov 2022cs.CV cs.AIcited by 46

Not on the map. Why?

Abstract (arXiv)

Recent works have demonstrated that natural language can be used to generate and edit 3D shapes. However, these methods generate shapes with limited fidelity and diversity. We introduce CLIP-Sculptor, a method to address these constraints by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs during training. CLIP-Sculptor achieves this in a multi-resolution approach that first generates in a low-dimensional latent space and then upscales to a higher resolution for improved shape fidelity. For improved shape diversity, we use a discrete latent space which is modeled using a transformer conditioned on CLIP's image-text embedding space. We also present a novel variant of classifier-free guidance, which improves the accuracy-diversity trade-off. Finally, we perform extensive experiments demonstrating that CLIP-Sculptor outperforms state-of-the-art baselines. The code is available at https://ivl.cs.brown.edu/#/projects/clip-sculptor.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 39 of its 42 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

39 of 42 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
27 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  3. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  4. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  7. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  9. 2020Jukebox: A Generative Model for Musiccited by 535
  10. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  11. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  14. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  15. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  16. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  17. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  18. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  19. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  20. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  21. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  22. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Modelcited by 62
  23. 2019ShapeGlot: Learning Language for Shape Differentiationcited by 54
  24. 2020Robust Training of Vector Quantized Bottleneck Modelscited by 53
  25. 20213D Shape Generation with Grid-based Implicit Functionscited by 48
  26. 20223DALL-E: Integrating Text-to-Image AI in 3D Design Workflowscited by 41
  27. 2022Opal: Multimodal Image Generation for News Illustrationcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 21; 2025: 8; 2026: 3.

46 papers.

Sort:

By area:

  1. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  2. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  3. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  4. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  5. 2024Advances in 3D Generation: A Surveycited by 62
  6. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  7. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Modelscited by 51
  8. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  9. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  10. 2024Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databasescited by 43
  11. 2024Scaling Diffusion Models to Real-World 3D LiDAR Scene Completioncited by 40
  12. 2023SHAP-EDITOR: Instruction-guided Latent 3D Editing in Secondscited by 36
34 more
  1. 2024Make-A-Shape: a Ten-Million-scale 3D Shape Modelcited by 25
  2. 2024TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resamplingcited by 24
  3. 2024Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Promptscited by 22
  4. 2024BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesiscited by 20
  5. 2024GaussianSR: 3D Gaussian Super-Resolution with 2D Diffusion Priorscited by 20
  6. 2023A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shotcited by 19
  7. 2024GM-DF: Generalized Multi-Scenario Deepfake Detectioncited by 16
  8. 2023VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representationcited by 14
  9. 2024A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIPcited by 14
  10. 2024Text-to-3D Shape Generationcited by 13
  11. 2025VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modelingcited by 12
  12. 2024JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillationcited by 11
  13. 2024Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodingscited by 10
  14. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 10
  15. 2023TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fieldscited by 9
  16. 2023Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Modelscited by 8
  17. 2023Text-Guided 3D Face Synthesis -- From Generation to Editingcited by 8
  18. 2024HyperSDFusion: Bridging Hierarchical Structures in Language and Geometry for Enhanced 3D Text2Shape Generationcited by 8
  19. 2025Register Any Point: Scaling 3D Point Cloud Registration by Flow Matchingcited by 8
  20. 2023Spice-E : Structural Priors in 3D Diffusion using Cross-Entity Attentioncited by 7
  21. 2023TPA3D: Triplane Attention for Fast Text-to-3D Generationcited by 6
  22. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusioncited by 6
  23. 2025Towards Generating Realistic 3D Semantic Training Data for Autonomous Drivingcited by 5
  24. 2023DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priorscited by 3
  25. 2024Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conceptioncited by 3
  26. 2024GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representationcited by 3
  27. 2025PASTA: Part-Aware Sketch-to-3D Shape Generation with Text-Aligned Priorcited by 2
  28. 2026SldprtNet: A Large-Scale Multimodal Dataset for CAD Generation in Language-Driven 3D Designcited by 2
  29. 2022LoopDraw: a Loop-Based Autoregressive Model for Shape Synthesis and Editingcited by 1
  30. 2024HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generationcited by 1
  31. 2025Dive3D: Diverse Distillation-based Text-to-3D Generation via Score Implicit Matchingcited by 1
  32. 2026Spherical Geometry Diffusion: Generating High-quality 3D Face Geometry via Sphere-anchored Representationscited by 1
  33. 2024GenHeld: Generating and Editing Handheld Objectscited by 0
  34. 2026CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Controlcited by 0