mlsanityData to 7 Oct 2026

3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models

arXiv 2301.11445 ↗First posted 26 Jan 2023cs.CV cs.GRcited by 449

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce 3DShape2VecSet, a novel shape representation for neural fields designed for generative diffusion models. Our shape representation can encode 3D shapes given as surface models or point clouds, and represents them as neural fields. The concept of neural fields has previously been combined with a global latent vector, a regular grid of latent vectors, or an irregular grid of latent vectors. Our new representation encodes neural fields on top of a set of vectors. We draw from multiple concepts, such as the radial basis function representation and the cross attention and self-attention function, to design a learnable representation that is especially suitable for processing with transformers. Our results show improved performance in 3D shape encoding and 3D shape generative modeling tasks. We demonstrate a wide variety of generative applications: unconditioned generation, category-conditioned generation, text-conditioned generation, point-cloud completion, and image-conditioned generation.

Key facts. Cited by 449 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 72 of its 94 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

72 of 94 references.

Closest ancestors · 6

Same direction, newest first

  1. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  2. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions24 Nov
  3. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan
  6. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
38 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  6. 2015Variational Inference with Normalizing Flowscited by 2,161
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2020Point Transformercited by 1,462
  9. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  12. 2020Convolutional Occupancy Networkscited by 760
  13. 2020PCT: Point cloud transformercited by 730
  14. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  15. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  16. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  17. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  18. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  19. 2018Assessing Generative Models via Precision and Recallcited by 401
  20. 2020Local Implicit Grid Representations for 3D Scenescited by 372
  21. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  22. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  23. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  24. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  25. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  26. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  27. 2016Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesiscited by 219
  28. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  29. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  30. 2016A Theory of Generative ConvNetcited by 174
  31. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  32. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132
  33. 2016Generative and Discriminative Voxel Modeling with Convolutional Neural Networkscited by 128
  34. 2020Points2Surf: Learning Implicit Surfaces from Point Cloud Patchescited by 122
  35. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  36. 2018PointGrow: Autoregressively Learned Point Cloud Generation with Self-Attentioncited by 105
  37. 20223DILG: Irregular Latent Grids for 3D Generative Modelingcited by 92
  38. 2022SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generationcited by 92

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 82; 2025: 154; 2026: 175.

449 papers.

Sort:

By area:

  1. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  3. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  5. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  6. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  7. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  8. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  9. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  10. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  11. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  12. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
38 more
  1. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  2. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  3. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  4. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  5. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  6. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 157
  7. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  8. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  10. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  11. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  12. 2024MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generationcited by 113
  13. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  14. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  15. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  16. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 93
  17. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  18. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  19. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  20. 2025Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modelingcited by 84
  21. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  22. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  23. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  24. 2024Scaling Mesh Generation via Compressive Tokenizationcited by 71
  25. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioningcited by 68
  26. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  27. 2024Advances in 3D Generation: A Surveycited by 62
  28. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  29. 2025OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesioncited by 60
  30. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  31. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 57
  32. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  33. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  34. 2023Mosaic-SDF for 3D Generative Modelscited by 48
  35. 2024PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidancecited by 47
  36. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  37. 2025Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attentioncited by 43
  38. 2025X-Part: high fidelity and structure coherent shape decompositioncited by 42

50 most cited of 449 All 449 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
  2. 2022Deep Generative Models on 3D Representations: A Survey
  3. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
  4. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data
  5. 2025Point-Based Shape Representation Generation with a Correspondence-Preserving Diffusion Model
  6. 2023Shap-E: Generating Conditional 3D Implicit Functions
  7. 2025Repurposing 2D Diffusion Models for 3D Shape Completion
  8. 2023Breathing New Life into 3D Assets with Generative Repainting