# S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency

arXiv: https://arxiv.org/abs/2009.14711 · HTML: https://mlsanity.com/paper/2009.14711/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 28 of 28)

- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation](https://mlsanity.com/paper/2308.15975/) (2023)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/2211.09325/) (2022)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models](https://mlsanity.com/paper/2402.13181/) (2024)
- [Reinforcement Learning with Neural Radiance Fields](https://mlsanity.com/paper/2206.01634/) (2022)
- [Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds](https://mlsanity.com/paper/2210.07442/) (2022)
- [USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulation](https://mlsanity.com/paper/2209.13864/) (2022)
- [Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2207.05480/) (2022)
- [One-shot Imitation Learning via Interaction Warping](https://mlsanity.com/paper/2306.12392/) (2023)
- [Reinforcement Learning with Generalizable Gaussian Splatting](https://mlsanity.com/paper/2404.07950/) (2024)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulation](https://mlsanity.com/paper/2305.04718/) (2023)
- [Sim2Real Object-Centric Keypoint Detection and Description](https://mlsanity.com/paper/2202.00448/) (2022)
- [Supervised Training of Dense Object Nets using Optimal Descriptors for Industrial Robotic Applications](https://mlsanity.com/paper/2102.08096/) (2021)
- [Learning Generalizable Agents via Saliency-Guided Features Decorrelation](https://mlsanity.com/paper/2310.05086/) (2023)
- [Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings](https://mlsanity.com/paper/2112.04910/) (2021)
- [Learning Keypoints from Synthetic Data for Robotic Cloth Folding](https://mlsanity.com/paper/2205.06714/) (2022)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Efficient and Robust Training of Dense Object Nets for Multi-Object Robot Manipulation](https://mlsanity.com/paper/2206.12145/) (2022)
- [On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learning](https://mlsanity.com/paper/2605.26649/) (2026)
- [Semi-automatic 3D Object Keypoint Annotation and Detection for the Masses](https://mlsanity.com/paper/2201.07665/) (2022)
- [Robot Skill Generalization via Keypoint Integrated Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2310.15059/) (2023)
- [Cycle-Correspondence Loss: Learning Dense View-Invariant Visual Features from Unlabeled and Unordered RGB Images](https://mlsanity.com/paper/2406.12441/) (2024)
- [KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images](https://mlsanity.com/paper/2206.11030/) (2022)
- [Semi-Supervised Neural Processes for Articulated Object Interactions](https://mlsanity.com/paper/2412.00145/) (2024)
- [LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields](https://mlsanity.com/paper/2606.31941/) (2026)

## Abstract (arXiv)

A robot's ability to act is fundamentally constrained by what it can perceive. Many existing approaches to visual representation learning utilize general-purpose training criteria, e.g. image reconstruction, smoothness in latent space, or usefulness for control, or else make use of large datasets annotated with specific features (bounding boxes, segmentations, etc.). However, both approaches often struggle to capture the fine-detail required for precision tasks on specific objects, e.g. grasping and mating a plug and socket. We argue that these difficulties arise from a lack of geometric structure in these models. In this work we advocate semantic 3D keypoints as a visual representation, and present a semi-supervised training objective that can allow instance or category-level keypoints to be trained to 1-5 millimeter-accuracy with minimal supervision. Furthermore, unlike local texture-based approaches, our model integrates contextual information from a large area and is therefore robust to occlusion, noise, and lack of discernible texture. We demonstrate that this ability to locate semantic keypoints enables high level scripting of human understandable behaviours. Finally we show that these keypoints provide a good way to define reward functions for reinforcement learning and are a good representation for training agents.