mlsanityData to 14 Oct 2026

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

arXiv 1803.08495 ↗First posted 22 Mar 2018cs.CV cs.AI cs.GRcited by 169

Not on the map. Why?

Cited by 40 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a method for generating colored 3D shapes from natural language. To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes. Our model combines and extends learning by association and metric learning approaches to learn implicit cross-modal connections, and produces a joint representation that captures the many-to-many relations between language and physical properties of 3D shapes such as color and shape. To evaluate our approach, we collect a large dataset of natural language descriptions for physical 3D objects in the ShapeNet dataset. With this learned joint embedding we demonstrate text-to-shape retrieval that outperforms baseline approaches. Using our embeddings with a novel conditional Wasserstein GAN framework, we generate colored 3D shapes from text. Our method is the first to connect natural language text with realistic 3D objects exhibiting rich variations in color, texture, and shape detail. See video at https://youtu.be/zraPvRdl13Q

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 30; 2023: 52; 2024: 50; 2025: 18; 2026: 8.

169 papers.

Sort:

By area:

  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  4. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  5. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  6. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  7. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  8. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  9. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  10. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  11. 2023Text-To-4D Dynamic Scene Generationcited by 205
  12. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
38 more
  1. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  2. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  3. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  5. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  6. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  7. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 130
  8. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  10. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 123
  11. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  12. 2023TextDeformer: Geometry Manipulation using Text Guidancecited by 94
  13. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 87
  14. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 82
  15. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  16. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  17. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 74
  18. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  19. 2022Towards Implicit Text-Guided 3D Shape Generationcited by 68
  20. 2023Texture Generation on 3D Meshes with Point-UV Diffusioncited by 63
  21. 2024Advances in 3D Generation: A Surveycited by 63
  22. 2022ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Modelcited by 62
  23. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  24. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 60
  25. 2022Understanding Pure CLIP Guidance for Voxel Grid NeRF Modelscited by 56
  26. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  27. 2023Large-Vocabulary 3D Diffusion Model with Transformercited by 48
  28. 2022CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Languagecited by 46
  29. 2024TEXGen: a Generative Diffusion Model for Mesh Texturescited by 46
  30. 2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
  31. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 44
  32. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  33. 2024OctFusion: Octree-based Diffusion Models for 3D Shape Generationcited by 43
  34. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  35. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  36. 2022AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesiscited by 39
  37. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 39
  38. 2023Sketch and Text Guided Diffusion Model for Colored Point Cloud Generationcited by 36

50 most cited of 169 All 169 (JSON)