mlsanityData to 14 Oct 2026

Emergent Correspondence from Image Diffusion

arXiv 2306.03881 ↗First posted 6 Jun 2023cs.CVcited by 446

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io

Key facts. Cited by 446 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 79 of its 108 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

79 of 108 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov

Other references · 77

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Generative Adversarial Networkscited by 5,627
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  9. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  10. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  11. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  12. 2023Zero-shot Image-to-Image Translationcited by 505
  13. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  14. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  15. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  16. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  17. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  18. 2020DISK: Learning local features with policy gradientcited by 312
  19. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  20. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 264
  21. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  22. 2022Unsupervised Semantic Segmentation by Distilling Feature Correspondencescited by 241
  23. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  24. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  25. 2019R2D2: Repeatable and Reliable Detector and Descriptorcited by 207
  26. 2018Neighbourhood Consensus Networkscited by 174
  27. 2018Tracking Emerges by Colorizing Videoscited by 166
  28. 2017A Survey of Structure from Motioncited by 158
  29. 2018LF-Net: Learning Local Features from Imagescited by 151
  30. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 146
  31. 2017Convolutional neural network architecture for geometric matchingcited by 138
  32. 2022AiATrack: Attention in Attention for Transformer Visual Trackingcited by 137
  33. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  34. 2020Learning Feature Descriptors using Camera Pose Supervisioncited by 104
  35. 2022Towards Grand Unification of Object Trackingcited by 93
  36. 2021Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspectivecited by 81
  37. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  38. 2019ContextDesc: Local Descriptor Augmentation with Cross-Modality Contextcited by 79

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 42; 2024: 142; 2025: 148; 2026: 114.

446 papers.

Sort:

By area:

  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  2. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  3. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  4. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  5. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  6. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  7. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  8. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  10. 2024Training-Free Consistent Text-to-Image Generationcited by 125
  11. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  12. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
38 more
  1. 2023Diffusion Model as Representation Learnercited by 95
  2. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  3. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 86
  4. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  5. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 71
  6. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  7. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 69
  8. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  9. 2024Zero-shot Image Editing with Reference Imitationcited by 65
  10. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  11. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  12. 2024SpotlessSplats: Ignoring Distractors in 3D Gaussian Splattingcited by 61
  13. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  14. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 60
  15. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
  16. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 59
  17. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  18. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 58
  19. 2024OmniGlue: Generalizable Feature Matching with Foundation Model Guidancecited by 56
  20. 2023Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Featurescited by 55
  21. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  22. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  23. 2024Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagationcited by 51
  24. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  25. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  26. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  27. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  28. 2024LaRE2: Latent Reconstruction Error Based Method for Diffusion-Generated Image Detectioncited by 47
  29. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 47
  30. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  31. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 44
  32. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 43
  33. 2024Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignmentcited by 43
  34. 2026Image Generators are Generalist Vision Learnerscited by 43
  35. 2023Text-image Alignment for Diffusion-based Perceptioncited by 42
  36. 2023DragVideo: Interactive Drag-style Video Editingcited by 40
  37. 2023SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matchingcited by 38
  38. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38

50 most cited of 446 All 446 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
  2. 2023Unsupervised Semantic Correspondence Using Stable Diffusion
  3. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
  4. 2025Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations
  5. 2024Diffusion Models in Low-Level Vision: A Survey
  6. 2024CleanDIFT: Diffusion Features without Noise
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  8. 2023DiffuGen: Adaptable Approach for Generating Labeled Image Datasets using Stable Diffusion Models