mlsanityData to 7 Oct 2026

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

arXiv 2304.09787 ↗First posted 19 Apr 2023cs.CVcited by 76

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Automatically generating high-quality real world 3D scenes is of enormous interest for applications such as virtual reality and robotics simulation. Towards this goal, we introduce NeuralField-LDM, a generative model capable of synthesizing complex 3D environments. We leverage Latent Diffusion Models that have been successfully utilized for efficient high-quality 2D content creation. We first train a scene auto-encoder to express a set of image and pose pairs as a neural field, represented as density and feature voxel grids that can be projected to produce novel views of the scene. To further compress this representation, we train a latent-autoencoder that maps the voxel grids to a set of latent representations. A hierarchical diffusion model is then fit to the latents to complete the scene generation pipeline. We achieve a substantial improvement over existing state-of-the-art scene generation models. Additionally, we show how NeuralField-LDM can be used for a variety of 3D content creation applications, including conditional scene generation, scene inpainting and scene style manipulation.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 75 of its 95 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

75 of 95 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  3. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 2022Video Diffusion Modelscited by 2,280
  13. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  14. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  15. 2018Group Normalizationcited by 1,514
  16. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  17. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  18. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  19. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  20. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  21. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  22. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  23. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  24. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  25. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  26. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  27. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  28. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  29. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  30. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  31. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  32. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  33. 2018Which Training Methods for GANs do actually Converge?cited by 672
  34. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  35. 2021Score-based Generative Modeling in Latent Spacecited by 634
  36. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  37. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  38. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 37; 2025: 13; 2026: 12.

76 papers.

Sort:

By area:

  1. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  2. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  3. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  4. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  5. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  6. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  7. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  8. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  9. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  10. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  11. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  12. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
38 more
  1. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  2. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  3. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  4. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  5. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  6. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  7. 2024LT3SD: Latent Trees for 3D Scene Diffusioncited by 53
  8. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  9. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  10. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  11. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  12. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  13. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generationcited by 32
  14. 2024A Comprehensive Survey on 3D Content Generationcited by 28
  15. 2024Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusioncited by 27
  16. 2024HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generationcited by 21
  17. 2024RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpaintingcited by 21
  18. 2024DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latentscited by 20
  19. 2023NViST: In the Wild New View Synthesis from a Single Image with Transformerscited by 19
  20. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  21. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  22. 2024Urban Architect: Steerable 3D Urban Scene Generation with Layout Priorcited by 17
  23. 2024Denoising Diffusion via Image-Based Renderingcited by 15
  24. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Datacited by 15
  25. 2024ViewFusion: Towards Multi-View Consistency via Interpolated Denoisingcited by 14
  26. 2024Urban Scene Diffusion through Semantic Occupancy Mapcited by 14
  27. 2024HPC: Hierarchical Progressive Coding Framework for Volumetric Videocited by 13
  28. 2024Envision3D: One Image to 3D with Anchor Views Interpolationcited by 12
  29. 2024Diffusion Models in 3D Vision: A Surveycited by 11
  30. 2024PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesiscited by 10
  31. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Modelscited by 10
  32. 2024MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detectioncited by 10
  33. 2023WildFusion: Learning 3D-Aware Latent Diffusion Models in View Spacecited by 8
  34. 2024Coherent 3D Scene Diffusion From a Single RGB Imagecited by 8
  35. 2025Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Imagecited by 8
  36. 2023Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Modelscited by 7
  37. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  38. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253D Scene Generation: A Survey
  2. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
  3. 20223D Neural Field Generation using Triplane Diffusion
  4. 2025Recent Advances in 3D Object and Scene Generation: A Survey
  5. 2024Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields
  6. 2024L3DG: Latent 3D Gaussian Diffusion
  7. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models
  8. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation