# Unsupervised Object Keypoint Learning using Local Spatial Predictability

arXiv: https://arxiv.org/abs/2011.12930 · HTML: https://mlsanity.com/paper/2011.12930/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming](https://mlsanity.com/paper/2202.04178/) (2022)
- [Systematic Visual Reasoning through Object-Centric Relational Abstraction](https://mlsanity.com/paper/2306.02500/) (2023)
- [3D Implicit Transporter for Temporally Consistent Keypoint Discovery](https://mlsanity.com/paper/2309.05098/) (2023)
- [Unsupervised Image Representation Learning with Deep Latent Particles](https://mlsanity.com/paper/2205.15821/) (2022)
- [Contrastive Training of Complex-Valued Autoencoders for Object Discovery](https://mlsanity.com/paper/2305.15001/) (2023)
- [Vision-based system identification and 3D keypoint discovery using dynamics constraints](https://mlsanity.com/paper/2109.05928/) (2021)
- [Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention](https://mlsanity.com/paper/2211.06027/) (2022)
- [Learning of feature points without additional supervision improves reinforcement learning from images](https://mlsanity.com/paper/2106.07995/) (2021)
- [Revealing Human Attention Patterns from Gameplay Analysis for Reinforcement Learning](https://mlsanity.com/paper/2504.11118/) (2025)
- [Entropy-driven Unsupervised Keypoint Representation Learning in Videos](https://mlsanity.com/paper/2209.15404/) (2022)
- [Learning Geometric Representations of Objects via Interaction](https://mlsanity.com/paper/2309.05346/) (2023)
- [Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)](https://mlsanity.com/paper/2312.00592/) (2023)
- [Unsupervised Salient Patch Selection for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2402.03329/) (2024)

## Abstract (arXiv)

We propose PermaKey, a novel approach to representation learning based on object keypoints. It leverages the predictability of local image regions from spatial neighborhoods to identify salient regions that correspond to object parts, which are then converted to keypoints. Unlike prior approaches, it utilizes predictability to discover object keypoints, an intrinsic property of objects. This ensures that it does not overly bias keypoints to focus on characteristics that are not unique to objects, such as movement, shape, colour etc. We demonstrate the efficacy of PermaKey on Atari where it learns keypoints corresponding to the most salient object parts and is robust to certain visual distractors. Further, on downstream RL tasks in the Atari domain we demonstrate how agents equipped with our keypoints outperform those using competing alternatives, even on challenging environments with moving backgrounds or distractor objects.