mlsanityData to 14 Oct 2026

Unsupervised Semantic Correspondence Using Stable Diffusion

arXiv 2305.15581 ↗First posted 24 May 2023cs.CVcited by 120

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models are now capable of generating images that are often indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they are asked to generate. In this work we show that, without any training, one can leverage this semantic knowledge within diffusion models to find semantic correspondences - locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention on the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing so we obtain results on par with the strongly supervised state of the art on the PF-Willow dataset and significantly outperform (20.9% relative for the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 75 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

52 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence23 May
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
33 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  8. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  9. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  10. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  11. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  12. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  13. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 495
  14. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  15. 2016LIFT: Learned Invariant Feature Transformcited by 306
  16. 2020Image Matching across Wide Baselines: From Paper to Practicecited by 225
  17. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  18. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  19. 2017Learning to Find Good Correspondencescited by 176
  20. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 146
  21. 2019GLU-Net: Global-Local Universal Network for Dense Flow and Correspondencescited by 139
  22. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  23. 2015Proposal Flowcited by 57
  24. 2020GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Networkcited by 53
  25. 2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
  26. 2017Proposal Flow: Semantic Correspondences from Object Proposalscited by 47
  27. 2021FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modellingcited by 47
  28. 2023Gradient-Free Textual Inversioncited by 29
  29. 2023ASIC: Aligning Sparse in-the-wild Image Collectionscited by 27
  30. 2022Semi-Supervised Learning of Semantic Correspondence with Pseudo-Labelscited by 16
  31. 2021Convolutional Hough Matching Networks for Robust and Efficient Visual Correspondencecited by 12
  32. 2022MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentationcited by 10
  33. 2019ACNe: Attentive Context Normalization for Robust Permutation-Equivariant Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 46; 2025: 43; 2026: 14.

120 papers.

Sort:

By area:

  1. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  2. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  3. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  4. 2024Training-Free Consistent Text-to-Image Generationcited by 125
  5. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 86
  6. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 71
  7. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  8. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  9. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  10. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Modelscited by 58
  11. 2024OmniGlue: Generalizable Feature Matching with Foundation Model Guidancecited by 56
  12. 2024Diffusion Models and Representation Learning: A Surveycited by 54
38 more
  1. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  2. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  3. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  4. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  5. 2024CleanDIFT: Diffusion Features without Noisecited by 44
  6. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  7. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 43
  8. 2024Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignmentcited by 43
  9. 2026Image Generators are Generalist Vision Learnerscited by 43
  10. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  11. 2023SLiMe: Segment Like Mecited by 34
  12. 2023Improving Semantic Correspondence with Viewpoint-Guided Spherical Mapscited by 34
  13. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  14. 2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
  15. 2023FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimatorscited by 31
  16. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 31
  17. 2024Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Modelscited by 29
  18. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 22
  19. 2023CLiC: Concept Learning in Contextcited by 20
  20. 2024Generative Image as Action Modelscited by 18
  21. 2024Distillation of Diffusion Features for Semantic Correspondencecited by 17
  22. 2025MangaNinja: Line Art Colorization with Precise Reference Followingcited by 17
  23. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  24. 2023Unsupervised Keypoints from Pretrained Diffusion Modelscited by 15
  25. 2024Generative Enhancement for 3D Medical Imagescited by 14
  26. 2024Can Visual Foundation Models Achieve Long-term Point Tracking?cited by 14
  27. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 14
  28. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  29. 2024SHIC: Shape-Image Correspondences with no Keypoint Supervisioncited by 12
  30. 2024PrimeDepth: Efficient Monocular Depth Estimation with a Stable Diffusion Preimagecited by 12
  31. 2025MATCHA:Towards Matching Anythingcited by 12
  32. 2024NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFscited by 11
  33. 2024Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptorscited by 11
  34. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  35. 2025How Much 3D Do Video Foundation Models Encode?cited by 11
  36. 2024Zero-Shot Image Feature Consensus with Deep Functional Mapscited by 10
  37. 20243x2: 3D Object Part Segmentation by 2D Semantic Correspondencescited by 10
  38. 2023Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representationscited by 9

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Emergent Correspondence from Image Diffusion
  2. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
  3. 2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching
  4. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  5. 2026A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers
  6. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models
  7. 2024Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variations
  8. 2026MARCO: Navigating the Unseen Space of Semantic Correspondence