mlsanityData to 7 Oct 2026

Unsupervised Learning of Object Keypoints for Perception and Control

arXiv 1906.11883 ↗First posted 19 Jun 2019cs.CV cs.LGcited by 109

Not on the map. Why?

Abstract (arXiv)

The study of object representations in computer vision has primarily focused on developing representations that are useful for image classification, object detection, or semantic segmentation as downstream tasks. In this work we aim to learn object representations that are useful for control and reinforcement learning (RL). To this end, we introduce Transporter, a neural network architecture for discovering concise geometric object representations in terms of keypoints or image-space coordinates. Our method learns from raw video frames in a fully unsupervised manner, by transporting learnt image features between video frames using a keypoint bottleneck. The discovered keypoints track objects and object parts across long time-horizons more accurately than recent similar methods. Furthermore, consistent long-term tracking enables two notable results in control domains -- (1) using the keypoint co-ordinates and corresponding image features as inputs enables highly sample-efficient reinforcement learning; (2) learning to explore by controlling keypoint locations drastically reduces the search space, enabling deep exploration (leading to states unreachable through random action exploration) without any extrinsic rewards.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 28; 2022: 27; 2023: 26; 2024: 14; 2025: 10; 2026: 3.

109 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  5. 2021Towards Causal Representation Learningcited by 207
  6. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  7. 2022Human-to-Robot Imitation in the Wildcited by 157
  8. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  9. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  10. 2021Learning Temporally Causal Latent Processes from General Temporal Datacited by 96
  11. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  12. 2021CoNeRF: Controllable Neural Radiance Fieldscited by 65
38 more
  1. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  2. 2022Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learningcited by 54
  3. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  4. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  5. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  6. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  7. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  8. 2021Test-Time Personalization with a Transformer for Human Pose Estimationcited by 33
  9. 2021Unsupervised Visual Attention and Invariance for Reinforcement Learningcited by 30
  10. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  11. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Controlcited by 24
  12. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  13. 2022Is an Object-Centric Video Representation Beneficial for Transfer?cited by 24
  14. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  15. 2023ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environmentcited by 22
  16. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  17. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  18. 2022Self-supervised video pretraining yields robust and more human-aligned visual representationscited by 20
  19. 20233D Implicit Transporter for Temporally Consistent Keypoint Discoverycited by 20
  20. 2021GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generationcited by 19
  21. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  22. 2022Unsupervised Image Representation Learning with Deep Latent Particlescited by 19
  23. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 18
  24. 2021Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objectscited by 17
  25. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  26. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  27. 2023Contrastive Training of Complex-Valued Autoencoders for Object Discoverycited by 16
  28. 2023Model-Based Control with Sparse Neural Dynamicscited by 16
  29. 2024Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Modelscited by 16
  30. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  31. 2021Pushing it out of the Way: Interactive Visual Navigationcited by 14
  32. 2022Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Spacecited by 14
  33. 2021Reasoning-Modulated Representationscited by 13
  34. 2023Detect Everything with Few Examplescited by 13
  35. 2023The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulationcited by 12
  36. 2021A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulationcited by 11
  37. 2023Object-Centric Learning with Slot Mixture Modulecited by 11
  38. 2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 11

50 most cited of 109 All 109 (JSON)