mlsanityData to 7 Oct 2026

SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching

arXiv 2310.17569 ↗First posted 26 Oct 2023cs.CV cs.LGcited by 38

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In this paper, we address the challenge of matching semantically similar keypoints across image pairs. Existing research indicates that the intermediate output of the UNet within the Stable Diffusion (SD) can serve as robust image feature maps for such a matching task. We demonstrate that by employing a basic prompt tuning technique, the inherent potential of Stable Diffusion can be harnessed, resulting in a significant enhancement in accuracy over previous approaches. We further introduce a novel conditional prompting module that conditions the prompt on the local details of the input image pairs, leading to a further improvement in performance. We designate our approach as SD4Match, short for Stable Diffusion for Semantic Matching. Comprehensive evaluations of SD4Match on the PF-Pascal, PF-Willow, and SPair-71k datasets show that it sets new benchmarks in accuracy across all these datasets. Particularly, SD4Match outperforms the previous state-of-the-art by a margin of 12 percentage points on the challenging SPair-71k dataset.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

48 of 59 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Emergent Correspondence from Image Diffusion6 Jun
  2. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence24 May
  3. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence23 May
  4. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  6. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug

Other references · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
29 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  3. 2019How Can We Know What Language Models Know?cited by 953
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  7. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  8. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  9. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  10. 2018Neighbourhood Consensus Networkscited by 173
  11. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
  12. 2016Universal Correspondence Networkcited by 82
  13. 2020Deep Graph Matching via Blackbox Differentiation of Combinatorial Solverscited by 81
  14. 2021CATs: Cost Aggregation Transformers for Visual Correspondencecited by 77
  15. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  16. 2019SFNet: Learning Object-aware Semantic Correspondencecited by 60
  17. 2015Proposal Flowcited by 57
  18. 2019Hyperpixel Flow: Semantic Correspondence with Multi-layer Neural Featurescited by 56
  19. 2020Learning to Compose Hypercolumns for Visual Correspondencecited by 53
  20. 2017SCNet: Learning Semantic Correspondencecited by 48
  21. 2017Proposal Flow: Semantic Correspondences from Object Proposalscited by 46
  22. 2021Convolutional Hough Matching Networkscited by 43
  23. 2020Correspondence Networks with Adaptive Neighbourhood Consensuscited by 42
  24. 2022Learning Semantic Correspondence with Sparse Annotationscited by 31
  25. 2022Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondencescited by 30
  26. 2022TransforMatcher: Match-to-Match Attention for Semantic Correspondencecited by 29
  27. 2022Semi-Supervised Learning of Semantic Correspondence with Pseudo-Labelscited by 16
  28. 2021Joint Graph Learning and Matching for Semantic Feature Correspondencecited by 8
  29. 2023SimSC: A Simple Framework for Semantic Correspondence with Temperature Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 16; 2026: 7.

38 papers.

Sort:

By area:

  1. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  2. 2024CleanDIFT: Diffusion Features without Noisecited by 42
  3. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  5. 2026Image Generators are Generalist Vision Learnerscited by 40
  6. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Featurescited by 23
  7. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  8. 2024Distillation of Diffusion Features for Semantic Correspondencecited by 17
  9. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  10. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  11. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  12. 2025MATCHA:Towards Matching Anythingcited by 12
26 more
  1. 2024SHIC: Shape-Image Correspondences with no Keypoint Supervisioncited by 11
  2. 2024CorrespondentDream: Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondencescited by 6
  3. 2025SemAlign3D: Semantic Correspondence between RGB-Images through Aligning 3D Object-Class Representationscited by 6
  4. 2024Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentationcited by 4
  5. 2024Medical Manifestation-Aware De-Identificationcited by 4
  6. 2025GenColor: Generative Color-Concept Association in Visual Designcited by 3
  7. 2025Semantic Correspondence: Unified Benchmarking and a Strong Baselinecited by 3
  8. 2025Domain Generalizable Portrait Style Transfercited by 3
  9. 2025Towards Geometry-Grounded Dense Semantic Matching with VGGT Priorscited by 3
  10. 2025Probing and Leveraging Video Diffusion Transformer Features for Robust Point Trackingcited by 3
  11. 2026MARCO: Navigating the Unseen Space of Semantic Correspondencecited by 3
  12. 2024Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniquescited by 2
  13. 2025Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classificationcited by 2
  14. 2025ICE: Intrinsic Concept Extraction from a Single Image via Diffusion Modelscited by 1
  15. 2025Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomaliescited by 1
  16. 2025Similarity-Aware Selective State-Space Modeling for Semantic Correspondencecited by 1
  17. 2025V-Warper: Appearance-Consistent Video Diffusion Personalization via Value Warpingcited by 1
  18. 2026Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondencecited by 1
  19. 2024Matching Semantically Similar Non-Identical Objectscited by 0
  20. 2025Towards Robust Semantic Correspondence: A Benchmark and Insightscited by 0
  21. 2025MSDNet: Efficient 4D Radar Super-Resolution via Multi-Stage Distillationcited by 0
  22. 2025Exploring Conditions for Diffusion models in Robotic Controlcited by 0
  23. 2026SimpleMatch: A Simple and Strong Baseline for Semantic Correspondencecited by 0
  24. 2026AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifierscited by 0
  25. 2026Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulationcited by 0
  26. 2026Diffusion Trajectory Modeling for Semantic Correspondencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unsupervised Semantic Correspondence Using Stable Diffusion
  2. 2025Mind the Gap: Aligning Vision Foundation Models to Image Feature Matching
  3. 2024Bridging the Gap: Aligning Text-to-Image Diffusion Models with Specific Feedback
  4. 2023Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners
  5. 2026DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Models
  6. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  7. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models
  8. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models