Data to 7 Oct 2026

A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence

arXiv 2305.15347 ↗First posted 24 May 2023cs.CVcited by 297

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences can enable interesting applications such as instance swapping in two images.

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 59 of its 75 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

59 of 75 references.

Closest ancestors · 13

Same direction, newest first

1 more

2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 46

2015Deep Residual Learning for Image Recognitioncited by 51,809
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2020Denoising Diffusion Implicit Modelscited by 9,655
2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437

34 more

2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
2021Deep ViT Features as Dense Visual Descriptorscited by 327
2019D2-Net: A Trainable CNN for Joint Detection and Description of Local Featurescited by 324
2020DISK: Learning local features with policy gradientcited by 310
2016LIFT: Learned Invariant Feature Transformcited by 305
2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
2018LF-Net: Learning Local Features from Imagescited by 151
2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
2019GLU-Net: Global-Local Universal Network for Dense Flow and Correspondencescited by 139
2017Convolutional neural network architecture for geometric matchingcited by 138
2021Learning Accurate Dense Correspondences and When to Trust Themcited by 113
2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
2023Monocular Depth Estimation using Diffusion Modelscited by 87
2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
2021CATs: Cost Aggregation Transformers for Visual Correspondencecited by 77
2021GAN-Supervised Dense Visual Alignmentcited by 61
2017End-to-end weakly-supervised semantic alignmentcited by 60
2015Proposal Flowcited by 57
2020GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Networkcited by 53
2022CATs++: Boosting Cost Aggregation with Convolutions and Transformerscited by 53
2021Warp Consistency for Unsupervised Learning of Dense Correspondencescited by 42
2018Recurrent Transformer Networks for Semantic Correspondencecited by 32
2018Attentive Semantic Alignment with Offset-Aware Correlation Kernelscited by 31
2022Learning Semantic Correspondence with Sparse Annotationscited by 31
2022Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondencescited by 30
2023Semantic Diffusion Network for Semantic Segmentationcited by 30
2023Neural Congealing: Aligning Images to a Joint Semantic Atlascited by 30
2018PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondencecited by 26
2023ASIC: Aligning Sparse in-the-wild Image Collectionscited by 26
2018DifNet: Semantic Segmentation by Diffusion Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 94; 2025: 113; 2026: 60.

297 papers.

Sort:

By area:

2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
2023SeeSR: Towards Semantics-Aware Real-World Image Super-Resolutioncited by 311
2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 194
2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
2024Training-Free Consistent Text-to-Image Generationcited by 123
2023FoundPose: Unseen Object Pose Estimation with Foundation Featurescited by 109
2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 85
2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70

38 more

2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
2023LEAP: Liberate Sparse-view 3D Modeling from Camera Posescited by 65
2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
2024Zero-shot Image Editing with Reference Imitationcited by 64
2024SPAD : Spatially Aware Multiview Diffuserscited by 62
2024SpotlessSplats: Ignoring Distractors in 3D Gaussian Splattingcited by 61
2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 60
2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
2023Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Featurescited by 55
2024OmniGlue: Generalizable Feature Matching with Foundation Model Guidancecited by 55
2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
2024Diffusion Models and Representation Learning: A Surveycited by 53
2023SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimationcited by 52
2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generationcited by 43
2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
2024Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignmentcited by 42
2024CleanDIFT: Diffusion Features without Noisecited by 42
2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
2026Image Generators are Generalist Vision Learnerscited by 40
2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matchingcited by 38
2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38
2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36
2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 35
2023Do text-free diffusion models learn discriminative visual representations?cited by 33
2023Improving Semantic Correspondence with Viewpoint-Guided Spherical Mapscited by 33
2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
2023FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimatorscited by 29

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way