mlsanityData to 7 Oct 2026

Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program

arXiv 2212.12952 ↗First posted 25 Dec 2022cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: textitText Longleftrightarrow textitPoint Cloud Longleftrightarrow textitProgram. We propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed textitShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed textitPointVQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in textitText Longrightarrow textitPoint Cloud, textitPoint Cloud Longrightarrow textitText, textitPoint Cloud Longrightarrow textitProgram, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 87 of its 130 references that are in the map. First posted on arXiv 25 Dec 2022.

Builds on

87 of 130 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2017Mask R-CNNcited by 7,259
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  13. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  14. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  15. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  17. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  18. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  19. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  20. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  21. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  22. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  23. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  24. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  25. 2022A Generalist Agentcited by 802
  26. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  27. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  28. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  29. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  30. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  31. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  32. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 549
  33. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  34. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  35. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  36. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  37. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  38. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  2. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  3. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  4. 2024Text-to-3D Shape Generationcited by 13
  5. 2024MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creationcited by 7
  6. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0