mlsanityData to 14 Oct 2026

Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

arXiv 2312.02145 ↗First posted 4 Dec 2023cs.CVcited by 327

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page: https://marigoldmonodepth.github.io.

Key facts. Cited by 327 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 63 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

53 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Breathing New Life into 3D Assets with Generative Repainting15 Sep
  2. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation2 Jun
  3. 2023Monocular Depth Estimation using Diffusion Models28 Feb

Other references · 50

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2014Conditional Generative Adversarial Netscited by 2,162
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
38 more
  1. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  3. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  4. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  5. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  6. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  7. 2021Extract Free Dense Labels from CLIPcited by 583
  8. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  9. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 382
  10. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 365
  11. 2020Virtual KITTI 2cited by 353
  12. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  13. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  14. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  15. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  16. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 240
  17. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  18. 2016Single-Image Depth Perception in the Wildcited by 180
  19. 2022BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimationcited by 168
  20. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  21. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  22. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  23. 2022DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimationcited by 118
  24. 2022P3Depth: Monocular Depth Estimation with a Piecewise Planarity Priorcited by 107
  25. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 101
  26. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 82
  27. 2023Neural LiDAR Fields for Novel View Synthesiscited by 67
  28. 2020OASIS: A Large-Scale Dataset for Single Image 3D in the Wildcited by 62
  29. 2020DiverseDepth: Affine-invariant Depth Prediction Using Diverse Datacited by 61
  30. 2023VA-DepthNet: A Variational Approach to Single Image Depth Predictioncited by 54
  31. 2022Hierarchical Normalization for Robust Monocular Depth Estimationcited by 52
  32. 2023All in Tokens: Unifying Output Space of Visual Tasks via Soft Tokencited by 52
  33. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  34. 2020Bidirectional Attention Network for Monocular Depth Estimationcited by 48
  35. 2021Non Gaussian Denoising Diffusion Modelscited by 45
  36. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  37. 2022SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Modelscited by 34
  38. 20233DiffTection: 3D Object Detection with Geometry-Aware Diffusion Featurescited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 85; 2025: 145; 2026: 97.

327 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,422
  2. 2025Qwen-Image Technical Reportcited by 1,044
  3. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 692
  4. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  5. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  6. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
  7. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  8. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 286
  9. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 261
  10. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 173
  11. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 166
  12. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
38 more
  1. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  2. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  3. 2025Cameras as Relative Positional Encodingcited by 105
  4. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  5. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 82
  6. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  7. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  8. 2024PF3plat: Pose-Free Feed-Forward 3D Gaussian Splattingcited by 62
  9. 2024Depth Any Video with Scalable Synthetic Datacited by 57
  10. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 56
  11. 2024MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priorscited by 51
  12. 2025Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformerscited by 51
  13. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 50
  14. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 45
  15. 2026Image Generators are Generalist Vision Learnerscited by 43
  16. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 40
  17. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 40
  18. 2024GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removalcited by 39
  19. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  20. 2024Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentationcited by 38
  21. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 38
  22. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 38
  23. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  24. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  25. 2024Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assemblycited by 35
  26. 2025CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generationcited by 35
  27. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 35
  28. 2024DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Modelscited by 34
  29. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  30. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  31. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  32. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  33. 2024BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimationcited by 28
  34. 2024DepthLab: From Partial to Completecited by 27
  35. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  36. 2024LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priorscited by 25
  37. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 25
  38. 2025DA2: Depth Anything in Any Directioncited by 25

50 most cited of 327 All 327 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
  2. 2024Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion
  3. 2024Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions
  4. 2024Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation
  5. 2026Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation
  6. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage
  7. 2024Unsupervised Monocular Depth Estimation Based on Hierarchical Feature-Guided Diffusion
  8. 2024DepthFM: Fast Monocular Depth Estimation with Flow Matching