# Unsupervised Keypoints from Pretrained Diffusion Models

arXiv: https://arxiv.org/abs/2312.00065 · HTML: https://mlsanity.com/paper/2312.00065/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 55 of its 71 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 55)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence](https://mlsanity.com/paper/2305.14334/) (2023)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Text-to-Image Diffusion Models are Zero-Shot Classifiers](https://mlsanity.com/paper/2303.15233/) (2023)
- [PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model](https://mlsanity.com/paper/1803.08225/) (2018)

## Built on by (most cited first, 15 of 15)

- [SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting](https://mlsanity.com/paper/2406.20055/) (2024)
- [From Image to Video: An Empirical Study of Diffusion Representations](https://mlsanity.com/paper/2502.07001/) (2025)
- [FaceShot: Bring Any Character into Life](https://mlsanity.com/paper/2503.00740/) (2025)
- [Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models](https://mlsanity.com/paper/2405.16947/) (2024)
- [RUBIK: A Structured Benchmark for Image Matching across Geometric Challenges](https://mlsanity.com/paper/2502.19955/) (2025)
- [Stable Diffusion Models are Secretly Good at Visual In-Context Learning](https://mlsanity.com/paper/2508.09949/) (2025)
- [Cora: Correspondence-aware image editing using few step diffusion](https://mlsanity.com/paper/2505.23907/) (2025)
- [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024)
- [Symmetry Understanding of 3D Shapes via Chirality Disentanglement](https://mlsanity.com/paper/2508.05505/) (2025)
- [Diffusion-Based Action Recognition Generalizes to Untrained Domains](https://mlsanity.com/paper/2509.08908/) (2025)
- [Symmetry Informative and Agnostic Feature Disentanglement for 3D Shapes](https://mlsanity.com/paper/2601.14804/) (2026)
- [Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models](https://mlsanity.com/paper/2506.10633/) (2025)
- [Matching Semantically Similar Non-Identical Objects](https://mlsanity.com/paper/2403.08227/) (2024)
- [Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors](https://mlsanity.com/paper/2507.12336/) (2025)
- [KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization](https://mlsanity.com/paper/2609.28818/) (2026)

## Abstract (arXiv)

Unsupervised learning of keypoints and landmarks has seen significant progress with the help of modern neural network architectures, but performance is yet to match the supervised counterpart, making their practicability questionable. We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images (i.e. keypoints). To do so, we simply optimize the text embedding such that the cross-attention maps within the denoising network are localized as Gaussians with small standard deviations. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets. We achieve significantly improved accuracy, sometimes even outperforming supervised ones, particularly for data that is non-aligned and less curated. Our code is publicly available and can be found through our project page: https://ubc-vision.github.io/StableKeypoints/