mlsanityData to 14 Oct 2026

Exploiting Diffusion Prior for Generalizable Dense Prediction

arXiv 2311.18832 ↗First posted 30 Nov 2023cs.CVcited by 38

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Contents generated by recent advanced Text-to-Image (T2I) diffusion models are sometimes too imaginative for existing off-the-shelf dense predictors to estimate due to the immitigable domain gap. We introduce DMP, a pipeline utilizing pre-trained T2I models as a prior for dense prediction tasks. To address the misalignment between deterministic prediction tasks and stochastic T2I models, we reformulate the diffusion process through a sequence of interpolations, establishing a deterministic mapping between input RGB images and output prediction distributions. To preserve generalizability, we use low-rank adaptation to fine-tune pre-trained models. Extensive experiments across five tasks, including 3D property estimation, semantic segmentation, and intrinsic image decomposition, showcase the efficacy of the proposed method. Despite limited-domain training data, the approach yields faithful estimations for arbitrary images, surpassing existing state-of-the-art algorithms.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 71 of its 86 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

71 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Matryoshka Diffusion Models23 Oct
  2. 2023Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model5 May
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Models1 Jun
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  6. 2020Denoising Diffusion Implicit Models6 Oct
  7. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 64

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  9. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  11. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  12. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  13. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  14. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  15. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  16. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  17. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
  18. 2023Zero-shot Image-to-Image Translationcited by 505
  19. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  20. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  21. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  22. 2017Universal Style Transfer via Feature Transformscited by 354
  23. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  24. 2017Sparsity Invariant CNNscited by 326
  25. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  26. 2017Toward Multimodal Image-to-Image Translationcited by 296
  27. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  28. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  29. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 291
  30. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  31. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  32. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  33. 2019Large Scale Adversarial Representation Learningcited by 239
  34. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 213
  35. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  36. 2019Mode Seeking Generative Adversarial Networks for Diverse Image Synthesiscited by 138
  37. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  38. 20223D Common Corruptions and Data Augmentationcited by 119

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 11.

38 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 1,044
  2. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 166
  3. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  4. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?cited by 74
  5. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 39
  6. 2024PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructionscited by 31
  7. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  8. 2025Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Modelcited by 18
  9. 2024Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentationcited by 16
  10. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimagecited by 12
  11. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Renderingcited by 11
  12. 2025UniGeo: Taming Video Diffusion for Unified Consistent Geometry Estimationcited by 9
26 more
  1. 2024Birth and Death of a Rosecited by 7
  2. 2025Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Priorcited by 6
  3. 2025Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Modelscited by 6
  4. 2024GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Modelscited by 5
  5. 2024Generative Edge Detection with Stable Diffusioncited by 5
  6. 2025Jodi: Unification of Visual Generation and Understanding via Joint Modelingcited by 5
  7. 2026LightCity: An Urban Dataset for Outdoor Inverse Rendering and Reconstruction under Multi-illumination Conditionscited by 4
  8. 2026DVD: Deterministic Video Depth Estimation with Generative Priorscited by 4
  9. 2025From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarioscited by 3
  10. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
  11. 2025SuperCarver: Texture-Consistent 3D Geometry Super-Resolution for High-Fidelity Surface Detail Generationcited by 2
  12. 2025VRMDiff: Text-Guided Video Referring Matting Generation of Diffusioncited by 2
  13. 2025DIVE: Inverting Conditional Diffusion Models for Discriminative Taskscited by 2
  14. 2025LEAF: Latent Diffusion with Efficient Encoder Distillation for Aligned Features in Medical Image Segmentationcited by 1
  15. 2025LawDIS: Language-Window-based Controllable Dichotomous Image Segmentationcited by 1
  16. 2026Self-transcendence: Is External Feature Guidance Indispensable for Accelerating Diffusion Transformer Training?cited by 1
  17. 2024Urban Air Temperature Prediction using Conditional Diffusion Modelscited by 0
  18. 2025FreeDNA: Endowing Domain Adaptation of Diffusion-Based Dense Prediction with Training-Free Domain Noise Alignmentcited by 0
  19. 2025D3-Predictor: Noise-Free Deterministic Diffusion for Dense Predictioncited by 0
  20. 2026GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detectioncited by 0
  21. 2026Rethinking Vector Field Learning for Generative Segmentationcited by 0
  22. 2026SD-FSMIS: Adapting Stable Diffusion for Few-Shot Medical Image Segmentationcited by 0
  23. 2026FlowDIS: Language-Guided Dichotomous Image Segmentation with Flow Matchingcited by 0
  24. 2026MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Predictioncited by 0
  25. 2026TransNormal-2: Geometry-Grounded Rectified Flow with Edge-Aware Decoding for Precise Normal Estimationcited by 0
  26. 2026Why DiT Models Underperform as Representation Learners without Long Skip Connectionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DPBridge: Latent Diffusion Bridge for Dense Prediction
  2. 2024Toward a Diffusion-Based Generalist for Dense Vision Tasks
  3. 2023DDP: Diffusion Model for Dense Visual Prediction
  4. 2024Diffusion Models Need Visual Priors for Image Generation
  5. 2023Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
  6. 2026An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models
  7. 2025Residual Prior Diffusion: A Probabilistic Framework Integrating Coarse Latent Priors with Diffusion Models
  8. 2023Diffusion Models Without Attention