mlsanityData to 7 Oct 2026

SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation

arXiv 2212.04493 ↗First posted 8 Dec 2022cs.CV cs.LGcited by 287

Not on the map. Why?

Cited by 97 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we present a novel framework built to simplify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, including images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multi-modal inputs, we employ task-specific encoders with dropout followed by a cross-attention mechanism. Due to its flexibility, our model naturally supports a variety of tasks, outperforming prior works on shape completion, image-based 3D reconstruction, and text-to-3D. Most interestingly, our model can combine all these tasks into one swiss-army-knife tool, enabling the user to perform shape generation using incomplete shapes, images, and textual descriptions at the same time, providing the relative weights for each input and facilitating interactivity. Despite our approach being shape-only, we further show an efficient method to texture the generated shape using large-scale text-to-image models.

Key facts. Cited by 287 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

45 of 59 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
33 more
  1. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  2. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  3. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  4. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  7. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  12. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  13. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  14. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  15. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  16. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  17. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  18. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  19. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  20. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  21. 2019Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Imagescited by 164
  22. 2019SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimizationcited by 159
  23. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  24. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  25. 2017Improved Adversarial Systems for 3D Object Generation and Reconstructioncited by 66
  26. 2020Multimodal Shape Completion via Conditional Generative Adversarial Networkscited by 61
  27. 2022Generative Multiplane Images: Making a 2D GAN 3D-Awarecited by 57
  28. 2018Learning Descriptor Networks for 3D Shape Synthesis and Analysiscited by 56
  29. 2019ShapeGlot: Learning Language for Shape Differentiationcited by 54
  30. 2021Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketchescited by 54
  31. 2022Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Modelcited by 39
  32. 2021BuildingNet: Learning to Label 3D Buildingscited by 33
  33. 2022Cross-Modal 3D Shape Generation and Manipulationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 72; 2024: 117; 2025: 63; 2026: 35.

287 papers.

Sort:

By area:

  1. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  3. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  5. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  6. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  7. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  8. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  9. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  10. 2023Text-to-3D using Gaussian Splattingcited by 299
  11. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  12. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
38 more
  1. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  2. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  3. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  4. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  5. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  6. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  7. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  8. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  9. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  10. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  11. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 109
  12. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  13. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  14. 2024GES: Generalized Exponential Splatting for Efficient Radiance Field Renderingcited by 92
  15. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  16. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  17. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  18. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  19. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  20. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  21. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  22. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  23. 2024Scaling Mesh Generation via Compressive Tokenizationcited by 71
  24. 2023InfiniCity: Infinite-Scale City Synthesiscited by 68
  25. 2024Towards Realistic Scene Generation with LiDAR Diffusion Modelscited by 67
  26. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  27. 2023SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulationcited by 66
  28. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  29. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Modelscited by 64
  30. 2024Advances in 3D Generation: A Surveycited by 62
  31. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  32. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 57
  33. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  34. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  35. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  36. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 54
  37. 2024LT3SD: Latent Trees for 3D Scene Diffusioncited by 53
  38. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52

50 most cited of 287 All 287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusion
  2. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
  3. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models
  4. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models
  5. 2022IC3D: Image-Conditioned 3D Diffusion for Shape Generation
  6. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions
  7. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
  8. 2025Collaborative Multi-Modal Coding for High-Quality 3D Generation