mlsanityData to 7 Oct 2026

Unsupervised Keypoints from Pretrained Diffusion Models

arXiv 2312.00065 ↗First posted 29 Nov 2023cs.CVcited by 15

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 55 of its 71 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

55 of 71 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Emergent Correspondence from Image Diffusion6 Jun
  2. 2023Unsupervised Semantic Correspondence Using Stable Diffusion24 May
  3. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence24 May
  4. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence23 May
  5. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  6. 2023Text-to-Image Diffusion Models are Zero-Shot Classifiers27 Mar
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  8. 2022DiffusionDet: Diffusion Model for Object Detection17 Nov
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 45

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  10. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
33 more
  1. 2020First Order Motion Model for Image Animationcited by 698
  2. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  3. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  4. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  5. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  6. 2020Deep Learning-Based Human Pose Estimation: A Surveycited by 205
  7. 2023Tracking Everything Everywhere All at Oncecited by 190
  8. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  9. 2018PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Modelcited by 133
  10. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  11. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
  12. 2019SCOPS: Self-Supervised Co-Part Segmentationcited by 100
  13. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  14. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  15. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  16. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  17. 2018Deep Feature Factorization For Concept Discoverycited by 64
  18. 2018Facial Landmark Detection: a Literature Surveycited by 63
  19. 2021Unsupervised Part Discovery from Contrastive Reconstructioncited by 57
  20. 2021Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspectivecited by 49
  21. 2023SLiMe: Segment Like Mecited by 34
  22. 2020Deep Keypoint-Based Camera Pose Estimation with Geometric Constraintscited by 29
  23. 2022Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysiscited by 29
  24. 2020Motion-supervised Co-Part Segmentationcited by 28
  25. 2021Unsupervised Human Pose Estimation through Transforming Shape Templatescited by 26
  26. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  27. 2022ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wildcited by 23
  28. 2021GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generationcited by 19
  29. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  30. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  31. 2021T-LEAP: Occlusion-robust pose estimation of walking cows using temporal informationcited by 15
  32. 2021Audacity of huge: overcoming challenges of data scarcity and data quality for machine learning in computational materials discoverycited by 10
  33. 2022TUSK: Task-Agnostic Unsupervised Keypointscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024SpotlessSplats: Ignoring Distractors in 3D Gaussian Splattingcited by 61
  2. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  3. 2024Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Modelscited by 12
  4. 2025FaceShot: Bring Any Character into Lifecited by 12
  5. 2025RUBIK: A Structured Benchmark for Image Matching across Geometric Challengescited by 8
  6. 2025Stable Diffusion Models are Secretly Good at Visual In-Context Learningcited by 8
  7. 2025Cora: Correspondence-aware image editing using few step diffusioncited by 5
  8. 2024Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimationcited by 3
  9. 2025Symmetry Understanding of 3D Shapes via Chirality Disentanglementcited by 3
  10. 2025Diffusion-Based Action Recognition Generalizes to Untrained Domainscited by 2
  11. 2026Symmetry Informative and Agnostic Feature Disentanglement for 3D Shapescited by 2
  12. 2025Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Modelscited by 1
3 more
  1. 2024Matching Semantically Similar Non-Identical Objectscited by 0
  2. 2025Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priorscited by 0
  3. 2026KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks
  2. 2024Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts
  3. 2023NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models
  4. 2025KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models
  5. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models
  6. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  7. 2024Diffusion Models in Low-Level Vision: A Survey
  8. 2023Do text-free diffusion models learn discriminative visual representations?