mlsanityData to 7 Oct 2026

CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation

arXiv 2110.02624 ↗First posted 6 Oct 2021cs.CV cs.AIcited by 254

Not on the map. Why?

Cited by 67 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape generation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive comparative evaluations to better understand its behavior.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

46 of 69 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  12. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
34 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2015Variational Inference with Normalizing Flowscited by 2,161
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  6. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  7. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2020Convolutional Occupancy Networkscited by 760
  10. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  11. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  12. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  13. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  14. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  15. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  16. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  17. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  20. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  21. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  22. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  23. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  24. 2020Learning Visual Representations with Caption Annotationscited by 123
  25. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  26. 2018Point Cloud GANcited by 79
  27. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  28. 2018FloWaveNet : A Generative Flow for Raw Audiocited by 54
  29. 2019C-Flow: Conditional Generative Flow Models for Images and 3D Point Cloudscited by 50
  30. 2018Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequencescited by 26
  31. 2019Noise Regularization for Conditional Density Estimationcited by 23
  32. 2021Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIPcited by 21
  33. 2019An Acceleration Framework for High Resolution Image Synthesiscited by 6
  34. 2020Static and Animated 3D Scene Generation from Free-form Text Descriptionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 35; 2023: 109; 2024: 62; 2025: 36; 2026: 8.

254 papers.

Sort:

By area:

  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  4. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  5. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  8. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  11. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  12. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
38 more
  1. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  2. 2023Text-to-3D using Gaussian Splattingcited by 299
  3. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  4. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  5. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  6. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  7. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  8. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  9. 2023Text-To-4D Dynamic Scene Generationcited by 204
  10. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  11. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  12. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  13. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  14. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  15. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  16. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  17. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  18. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  19. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  20. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  21. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  22. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  23. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  24. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  25. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  26. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  27. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  28. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  29. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  30. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114
  31. 2023TADA! Text to Animatable Digital Avatarscited by 109
  32. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  33. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  34. 2023TextDeformer: Geometry Manipulation using Text Guidancecited by 94
  35. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  36. 2023HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generationcited by 88
  37. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  38. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87

50 most cited of 254 All 254 (JSON)