mlsanityData to 7 Oct 2026

D4D: An RGBD diffusion model to boost monocular depth estimation

arXiv 2403.07516 ↗First posted 12 Mar 2024cs.CVcited by 3

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Ground-truth RGBD data are fundamental for a wide range of computer vision applications; however, those labeled samples are difficult to collect and time-consuming to produce. A common solution to overcome this lack of data is to employ graphic engines to produce synthetic proxies; however, those data do not often reflect real-world images, resulting in poor performance of the trained models at the inference step. In this paper we propose a novel training pipeline that incorporates Diffusion4D (D4D), a customized 4-channels diffusion model able to generate realistic RGBD samples. We show the effectiveness of the developed solution in improving the performances of deep learning models on the monocular depth estimation task, where the correspondence between RGB and depth map is crucial to achieving accurate measurements. Our supervised training pipeline, enriched by the generated samples, outperforms synthetic and original data performances achieving an RMSE reduction of (8.2%, 11.9%) and (8.1%, 6.1%) respectively on the indoor NYU Depth v2 and the outdoor KITTI dataset.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 35 of its 68 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

35 of 68 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
23 more
  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  3. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  5. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  6. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  7. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  8. 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 292
  9. 2018Unity: A General Platform for Intelligent Agentscited by 232
  10. 2018DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistencycited by 148
  11. 2018Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundariescited by 144
  12. 2018High Quality Monocular Depth Estimation via Transfer Learningcited by 133
  13. 2019FastDepth: Fast Monocular Depth Estimation on Embedded Systemscited by 93
  14. 2019AttentionGAN: Unpaired Image-to-Image Translation using Attention-Guided Generative Adversarial Networkscited by 77
  15. 2021Improved Transformer for High-Resolution GANscited by 59
  16. 2017From source to target and back: symmetric bi-directional adaptive GANcited by 53
  17. 2021DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenescited by 45
  18. 2022UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translationcited by 42
  19. 2018Learning Single-Image Depth from Videos using Quality Assessment Networkscited by 25
  20. 2019Translate-to-Recognize Networks for RGB-D Scene Recognitioncited by 20
  21. 2021MLReal: Bridging the gap between training on synthetic data and real data applications in machine learningcited by 19
  22. 2017Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challengescited by 12
  23. 2022A Survey on RGB-D Datasetscited by 11

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024SGFormer: Spherical Geometry Transformer for 360 Depth Estimationcited by 9
  2. 2025Towards Consistent and Controllable Image Synthesis for Face Editingcited by 3
  3. 2025A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robotcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation
  2. 2025GeoDiff: Geometry-Guided Diffusion for Metric Depth Estimation
  3. 2023Monocular Depth Estimation using Diffusion Models
  4. 2026Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
  5. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
  6. 2023MonoDiffusion: Self-Supervised Monocular Depth Estimation Using Diffusion Model
  7. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model
  8. 2024Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions