mlsanityData to 7 Oct 2026

LDM3D: Latent Diffusion Model for 3D

arXiv 2305.10853 ↗First posted 18 May 2023cs.CVcited by 53

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences. A short video summarizing the approach can be found at https://t.ly/tdi2.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 27 of its 31 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

27 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Monocular Depth Estimation using Diffusion Models28 Feb

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
14 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  6. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  7. 2018A Note on the Inception Scorecited by 225
  8. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  9. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 121
  10. 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
  11. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  12. 2018Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimationcited by 78
  13. 2017Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimationcited by 70
  14. 2021GCNDepth: Self-supervised Monocular Depth Estimation based on Graph Convolutional Networkcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 18; 2025: 20; 2026: 5.

53 papers.

Sort:

By area:

  1. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  2. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  3. 2023SparseGS: Sparse View Synthesis using 3D Gaussian Splattingcited by 134
  4. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  5. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  6. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  7. 2024LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generationcited by 53
  8. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  9. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  10. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  11. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  12. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
38 more
  1. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  2. 2023Exploiting the Signal-Leak Bias in Diffusion Modelscited by 36
  3. 20243DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generationcited by 36
  4. 2023Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generationcited by 33
  5. 2025SynCity: Training-Free Generation of 3D Worldscited by 30
  6. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  7. 20253D Scene Generation: A Surveycited by 24
  8. 2023JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modelingcited by 23
  9. 2024Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpaintingcited by 23
  10. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  11. 2023iNVS: Repurposing Diffusion Inpainters for Novel View Synthesiscited by 20
  12. 2024Diffusion Priors for Dynamic View Synthesis from Monocular Videoscited by 18
  13. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  14. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 14
  15. 2024A Simple Approach to Unifying Diffusion-based Conditional Generationcited by 13
  16. 2025DreamCube: 3D Panorama Generation via Multi-plane Synchronizationcited by 12
  17. 2024IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generationcited by 11
  18. 2025JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformerscited by 11
  19. 2024Blind Inversion using Latent Diffusion Priorscited by 10
  20. 2024Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priorscited by 8
  21. 2026360Anything: Geometry-Free Lifting of Images and Videos to 360degcited by 7
  22. 2024L-MAGIC: Language Model Assisted Generation of Images with Coherencecited by 6
  23. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6
  24. 2024G3DR: Generative 3D Reconstruction in ImageNetcited by 5
  25. 2024CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Modelcited by 5
  26. 2024SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blendingcited by 3
  27. 2025Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generationcited by 3
  28. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  29. 2026World Tracing: Generative Pixel-Aligned Geometry Beyond the Visiblecited by 3
  30. 2024MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Textcited by 2
  31. 2025Constructing a 3D Scene from a Single Imagecited by 2
  32. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  33. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2
  34. 2025Joint Learning of Depth and Appearance for Portrait Image Animationcited by 1
  35. 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformercited by 1
  36. 2025Nested Annealed Training Scheme for Generative Adversarial Networkscited by 0
  37. 2025Towards Spatially Consistent Image Generation: On Incorporating Intrinsic Scene Properties into Diffusion Modelscited by 0
  38. 2025Diffusion^2: Turning 3D Environments into Radio Frequency Heatmapscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LDM3D-VR: Latent Diffusion Model for 3D VR
  2. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
  3. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion
  5. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  6. 2024Diffusion Models in 3D Vision: A Survey
  7. 2024VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models
  8. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation