# CoKe: Localized Contrastive Learning for Robust Keypoint Detection

arXiv: https://arxiv.org/abs/2009.14115 · HTML: https://mlsanity.com/paper/2009.14115/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2020.

## Built on by (most cited first, 12 of 12)

- [Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion](https://mlsanity.com/paper/2306.04633/) (2023)
- [3D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.10123/) (2023)
- [Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features](https://mlsanity.com/paper/2209.05624/) (2022)
- [Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering](https://mlsanity.com/paper/2406.00622/) (2024)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose](https://mlsanity.com/paper/2110.14213/) (2021)
- [Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space](https://mlsanity.com/paper/2504.21749/) (2025)
- [Towards Robust and Expressive Whole-body Human Pose and Shape Estimation](https://mlsanity.com/paper/2312.08730/) (2023)
- [Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis](https://mlsanity.com/paper/2306.00118/) (2023)
- [Unified Category-Level Object Detection and Pose Estimation from RGB Images using 3D Prototypes](https://mlsanity.com/paper/2508.02157/) (2025)
- [SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapes](https://mlsanity.com/paper/2408.02291/) (2024)
- [Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection](https://mlsanity.com/paper/2507.07994/) (2025)

## Abstract (arXiv)

In this paper, we introduce a contrastive learning framework for keypoint detection (CoKe). Keypoint detection differs from other visual tasks where contrastive learning has been applied because the input is a set of images in which multiple keypoints are annotated. This requires the contrastive learning to be extended such that the keypoints are represented and detected independently, which enables the contrastive loss to make the keypoint features different from each other and from the background. Our approach has two benefits: It enables us to exploit contrastive learning for keypoint detection, and by detecting each keypoint independently the detection becomes more robust to occlusion compared to holistic methods, such as stacked hourglass networks, which attempt to detect all keypoints jointly. Our CoKe framework introduces several technical innovations. In particular, we introduce: (i) A clutter bank to represent non-keypoint features; (ii) a keypoint bank that stores prototypical representations of keypoints to approximate the contrastive loss between keypoints; and (iii) a cumulative moving average update to learn the keypoint prototypes while training the feature extractor. Our experiments on a range of diverse datasets (PASCAL3D+, MPII, ObjectNet3D) show that our approach works as well, or better than, alternative methods for keypoint detection, even for human keypoints, for which the literature is vast. Moreover, we observe that CoKe is exceptionally robust to partial occlusion and previously unseen object poses.