mlsanityData to 7 Oct 2026

AutoDecoding Latent 3D Diffusion Models

arXiv 2307.05445 ↗First posted 7 Jul 2023cs.CVcited by 50

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation for rendering view-consistent appearance and geometry. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations to learn a 3D diffusion from 2D images or monocular videos of rigid or articulated objects. Our approach is flexible enough to use either existing camera supervision or no camera information at all -- instead efficiently learning it during training. Our evaluations demonstrate that our generation results outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and a large-scale, real video dataset of static objects.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 75 of its 95 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

75 of 95 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  2. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  3. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  4. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  5. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  7. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2022Classifier-Free Diffusion Guidancecited by 5,717
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  9. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  13. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  14. 2022Video Diffusion Modelscited by 2,280
  15. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  16. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  17. 2018Demystifying MMD GANscited by 1,268
  18. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  19. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  20. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  21. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  22. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  23. 2020First Order Motion Model for Image Animationcited by 698
  24. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  25. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  26. 2021Score-based Generative Modeling in Latent Spacecited by 634
  27. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  28. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  29. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  30. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  31. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  32. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  33. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  34. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  35. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  36. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  37. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  38. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 24; 2025: 12; 2026: 2.

50 papers.

Sort:

By area:

  1. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  3. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  5. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  6. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  7. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  8. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  9. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  10. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  11. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  12. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
38 more
  1. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  2. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  3. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52
  4. 2024GVGEN: Text-to-3D Generation with Volumetric Representationcited by 51
  5. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  6. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  7. 2024RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Modelscited by 36
  8. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  9. 2024Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstructioncited by 32
  10. 2023HumanLiff: Layer-wise 3D Human Generation with Diffusion Modelcited by 23
  11. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  12. 2024StructLDM: Structured Latent Diffusion for 3D Human Generationcited by 17
  13. 2024ViewFusion: Towards Multi-View Consistency via Interpolated Denoisingcited by 14
  14. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  15. 2024Envision3D: One Image to 3D with Anchor Views Interpolationcited by 12
  16. 2024MVImgNet2.0: A Larger-scale Dataset of Multi-view Imagescited by 12
  17. 2023NVS-Adapter: Plug-and-Play Novel View Synthesis from a Single Imagecited by 11
  18. 2024InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenescited by 10
  19. 2024E3Gen: Efficient, Expressive and Editable Avatars Generationcited by 10
  20. 2025Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Imagecited by 8
  21. 2025AnchorFlow: Training-Free 3D Editing via Latent Anchor-Aligned Flowscited by 8
  22. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  23. 2025Towards Learning to Complete Anything in Lidarcited by 7
  24. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusioncited by 6
  25. 2024Multi-View Large Reconstruction Model via Geometry-Aware Positional Encoding and Attentioncited by 6
  26. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6
  27. 2024NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Modelcited by 6
  28. 2023High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Priorcited by 5
  29. 2023BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generationcited by 4
  30. 2026GenAssets: Generating in-the-wild 3D Assets in Latent Spacecited by 4
  31. 2025DirectTriGS: Triplane-based Gaussian Splatting Field Representation for 3D Generationcited by 3
  32. 2024Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priorscited by 2
  33. 2024GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Spacecited by 1
  34. 2025Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Modelscited by 1
  35. 2024Diverse Score Distillationcited by 0
  36. 2025Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumescited by 0
  37. 2025LoomNet: Enhancing Multi-View Image Generation via Latent Space Weavingcited by 0
  38. 2026Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models
  2. 2024Compress3D: a Compressed Latent Space for 3D Generation from a Single Image
  3. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  4. 2025FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion
  5. 2025Representing 3D Shapes With 64 Latent Vectors for 3D Diffusion Models
  6. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
  7. 2025Drive Any Mesh: 4D Latent Diffusion for Mesh Deformation from Video
  8. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale