mlsanityData to 14 Oct 2026

Text-image Alignment for Diffusion-based Perception

arXiv 2310.00031 ↗First posted 29 Sep 2023cs.CVcited by 42

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models are generative models with impressive text-to-image synthesis capabilities and have spurred a new wave of creative methods for classical machine learning tasks. However, the best way to harness the perceptual knowledge of these generative models for visual tasks is still an open question. Specifically, it is unclear how to use the prompting interface when applying diffusion backbones to vision tasks. We find that automatically generated captions can improve text-image alignment and significantly enhance a model's cross-attention maps, leading to better perceptual performance. Our approach improves upon the current state-of-the-art (SOTA) in diffusion-based semantic segmentation on ADE20K and the current overall SOTA for depth estimation on NYUv2. Furthermore, our method generalizes to the cross-domain setting. We use model personalization and caption modifications to align our model to the target domain and find improvements over unaligned baselines. Our cross-domain object detection model, trained on Pascal VOC, achieves SOTA results on Watercolor2K. Our cross-domain segmentation method, trained on Cityscapes, achieves SOTA results on Dark Zurich-val and Nighttime Driving. Project page: https://www.vision.caltech.edu/tadp/. Code: https://github.com/damaggu/TADP.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 43 of its 59 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

43 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Emergent Correspondence from Image Diffusion6 Jun
  2. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence23 May
  3. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
24 more
  1. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  3. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  6. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  7. 2019Panoptic Feature Pyramid Networkscited by 645
  8. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  9. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  10. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  11. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  12. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 300
  13. 2018Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptationcited by 218
  14. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  15. 2020Unbiased Mean Teacher for Cross-domain Object Detectioncited by 159
  16. 2018Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttimecited by 122
  17. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  18. 2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87
  19. 2023All in Tokens: Unifying Output Space of Visual Tasks via Soft Tokencited by 52
  20. 2021Decoupled Adaptation for Cross-Domain Object Detectioncited by 37
  21. 2023One-shot Unsupervised Domain Adaptation with Personalized Diffusion Modelscited by 35
  22. 2022Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditionscited by 31
  23. 2023Prompting Diffusion Representations for Cross-Domain Semantic Segmentationcited by 30
  24. 2020Adaptive Object Detection with Dual Multi-Label Predictioncited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 19; 2025: 17; 2026: 4.

42 papers.

Sort:

By area:

  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  2. 2023DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Controlcited by 45
  3. 2024ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimationcited by 45
  4. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  5. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  6. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  7. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 25
  8. 2025Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimationcited by 21
  9. 2024Digging into contrastive learning for robust depth estimation with diffusion modelscited by 19
  10. 2025DepthMaster: Taming Diffusion Models for Monocular Depth Estimationcited by 19
  11. 2024ODGEN: Domain-specific Object Detection Data Generation with Diffusion Modelscited by 17
  12. 2025Knowledge-Aligned Counterfactual-Enhancement Diffusion Perception for Unsupervised Cross-Domain Visual Emotion Recognitioncited by 17
30 more
  1. 2023Harnessing Diffusion Models for Visual Perception with Meta Promptscited by 13
  2. 2024CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusioncited by 12
  3. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimagecited by 12
  4. 2024iSeg: An Iterative Refinement-based Framework for Training-free Segmentationcited by 10
  5. 2024SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimationcited by 8
  6. 2024MedDiff-FM: A Diffusion-based Foundation Model for Versatile Medical Image Applicationscited by 7
  7. 2024Understanding Implosion in Text-to-Image Generative Modelscited by 6
  8. 2025TextSplat: Text-Guided Semantic Fusion for Generalizable Gaussian Splattingcited by 6
  9. 2024Generative Edge Detection with Stable Diffusioncited by 5
  10. 2024On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimationcited by 4
  11. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentationcited by 4
  12. 2024Implicit and Explicit Language Guidance for Diffusion-based Visual Perceptioncited by 3
  13. 2025Segment Any-Quality Images with Generative Latent Space Enhancementcited by 3
  14. 2024Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniquescited by 2
  15. 2024No Annotations for Object Detection in Art through Stable Diffusioncited by 2
  16. 2025DIVE: Inverting Conditional Diffusion Models for Discriminative Taskscited by 2
  17. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIPcited by 2
  18. 2025Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervisioncited by 2
  19. 2025Diffusion-Based Action Recognition Generalizes to Untrained Domainscited by 2
  20. 2025Beyond Synthetic Replays: Turning Diffusion Features into Few-Shot Class-Incremental Learning Knowledgecited by 1
  21. 2025Stable Diffusion-Based Approach for Human De-Occlusioncited by 1
  22. 2025Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learningcited by 1
  23. 2024EDADepth: Enhanced Data Augmentation for Monocular Depth Estimationcited by 0
  24. 2025VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recoverycited by 0
  25. 2025Exploring Conditions for Diffusion models in Robotic Controlcited by 0
  26. 2025Autonomous Source Knowledge Selection in Multi-Domain Adaptationcited by 0
  27. 2026Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancementcited by 0
  28. 2026Rethinking Vector Field Learning for Generative Segmentationcited by 0
  29. 2026Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Modelscited by 0
  30. 2026Beyond Future Prediction: Denoising as Generative Adaptation for Robot Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Diffusion for Unified Image Generation and Understanding
  2. 2026Making Training-Free Diffusion Segmentors Scale with the Generative Power
  3. 2023Are Diffusion Models Vision-And-Language Reasoners?
  4. 2024Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
  5. 2024Improving Long-Text Alignment for Text-to-Image Diffusion Models
  6. 2025UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception
  7. 2024Diffusion Features to Bridge Domain Gap for Semantic Segmentation
  8. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching