mlsanityData to 7 Oct 2026

Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think

arXiv 2409.11355 ↗First posted 17 Sep 2024cs.CVcited by 82

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results, high computational demands due to multi-step inference limited its use in many scenarios. In this paper, we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200× faster. To optimize for downstream task performance, we perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models on common zero-shot benchmarks. We surprisingly find that this fine-tuning protocol also works directly on Stable Diffusion and achieves comparable performance to current state-of-the-art diffusion-based depth and normal estimation models, calling into question some of the conclusions drawn from prior works.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

47 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation24 May
  2. 2024DepthFM: Fast Monocular Depth Estimation with Flow Matching20 Mar
  3. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image18 Mar
  4. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D28 Nov
  5. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  6. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation2 Jun
  7. 2023Monocular Depth Estimation using Diffusion Models28 Feb

Other references · 40

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  9. 2021Vision Transformers for Dense Predictioncited by 1,789
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2024Depth Anything V2cited by 1,385
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
28 more
  1. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  2. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  3. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  4. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  5. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  6. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  7. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  8. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 375
  9. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 357
  10. 2020Virtual KITTI 2cited by 345
  11. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  12. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  13. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  14. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 236
  15. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  16. 2016Single-Image Depth Perception in the Wildcited by 179
  17. 2018Evaluation of CNN-based Single-Image Depth Estimation Methodscited by 137
  18. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  19. 20223D Common Corruptions and Data Augmentationcited by 120
  20. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  21. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  22. 2020OASIS: A Large-Scale Dataset for Single Image 3D in the Wildcited by 62
  23. 2014Designing Deep Networks for Surface Normal Estimationcited by 59
  24. 2022Hierarchical Normalization for Robust Monocular Depth Estimationcited by 51
  25. 2021Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Predictioncited by 48
  26. 2016Marr Revisited: 2D-3D Alignment via Surface Normal Predictioncited by 34
  27. 2023Tame a Wild Camera: In-the-Wild Monocular Camera Calibrationcited by 34
  28. 2023D2NT: A High-Performing Depth-to-Normal Translatorcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 44; 2026: 29.

82 papers.

Sort:

By area:

  1. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
  2. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  3. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  4. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  5. 2024Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusioncited by 47
  6. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  7. 2026Image Generators are Generalist Vision Learnerscited by 40
  8. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 37
  9. 2025LBM: Latent Bridge Matching for Fast Image-to-Image Translationcited by 36
  10. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 36
  11. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  12. 2024Video Depth without Video Modelscited by 28
38 more
  1. 2025Diff2Flow: Training Flow Matching Models via Diffusion Model Alignmentcited by 27
  2. 2025NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion Priorscited by 25
  3. 2025Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimationcited by 20
  4. 2025Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstructioncited by 20
  5. 2025From Editor to Dense Geometry Estimatorcited by 20
  6. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
  7. 2025AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backendcited by 16
  8. 2025Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Modelcited by 16
  9. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  10. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  11. 2026Video Generation Models are General-Purpose Vision Learnerscited by 12
  12. 2024Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priorscited by 11
  13. 2025GS-2DGS: Geometrically Supervised 2DGS for Reflective Object Reconstructioncited by 11
  14. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Renderingcited by 11
  15. 2025RePoseD: Efficient Relative Pose Estimation With Known Depth Informationcited by 10
  16. 2025M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversioncited by 10
  17. 2024SuperMat: Physically Consistent PBR Material Estimation at Interactive Ratescited by 9
  18. 2025UniGeo: Taming Video Diffusion for Unified Consistent Geometry Estimationcited by 9
  19. 2025Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimationcited by 9
  20. 2025GCC: Generative Color Constancy via Diffusing a Color Checkercited by 8
  21. 2025The Fourth Monocular Depth Estimation Challengecited by 8
  22. 2024Boost 3D Reconstruction using Diffusion-based Monocular Camera Calibrationcited by 7
  23. 2024DPBridge: Latent Diffusion Bridge for Dense Predictioncited by 6
  24. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 6
  25. 2025StereoDiff: Stereo-Diffusion Synergy for Video Depth Estimationcited by 6
  26. 2024FiffDepth: Feed-forward Transformation of Diffusion-Based Generators for Detailed Depth Estimationcited by 5
  27. 2025Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Imagescited by 5
  28. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusioncited by 5
  29. 2025MVInverse: Feed-forward Multi-view Inverse Rendering in Secondscited by 5
  30. 2026PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimationcited by 5
  31. 2025Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceiverscited by 4
  32. 2026THFM: A Unified Video Foundation Model for 4D Human Perception and Beyondcited by 4
  33. 2026Diffusion Model as a Generalist Segmentation Learnercited by 4
  34. 2025Elastic3D: Controllable Stereo Video Conversion with Guided Latent Decodingcited by 3
  35. 2025StereoPilot: Learning Unified and Efficient Stereo Conversion via Generative Priorscited by 3
  36. 2026VideoMaMa: Mask-Guided Video Matting via Generative Priorcited by 3
  37. 2026MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAEcited by 3
  38. 2025gen2seg: Generative Models Enable Generalizable Instance Segmentationcited by 2

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Simple Approach to Unifying Diffusion-based Conditional Generation
  2. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimage
  3. 2024Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions
  4. 2025DepthMaster: Taming Diffusion Models for Monocular Depth Estimation
  5. 2023A Unified Conditional Framework for Diffusion-based Image Restoration
  6. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning
  7. 2022DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models
  8. 2022Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors