mlsanityData to 7 Oct 2026

KETO: Learning Keypoint Representations for Tool Manipulation

arXiv 1910.11977 ↗First posted 26 Oct 2019cs.ROcited by 68

Not on the map. Why?

Abstract (arXiv)

We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals. An efficient and informative representation would facilitate the effectiveness and generalization of such algorithms. For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation. For each task, a set of task-specific keypoints is jointly predicted from 3D point clouds of the tool object by a deep neural network. These keypoints offer a concise and informative description of the object to determine grasps and subsequent manipulation actions. The model is learned from self-supervised robot interactions in the task environment without the need for explicit human annotations. We evaluate our framework in three manipulation tasks with tool use. Our model consistently outperforms state-of-the-art methods in terms of task success rates. Qualitative results of keypoint prediction and tool generation are shown to visualize the learned representations.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 7; 2022: 14; 2023: 14; 2024: 12; 2025: 12; 2026: 8.

68 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  2. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  3. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  4. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  5. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  6. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  7. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  8. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  9. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  10. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  11. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  12. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
38 more
  1. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  2. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  3. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  4. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  6. 2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
  7. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Controlcited by 24
  8. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  9. 20233D Implicit Transporter for Temporally Consistent Keypoint Discoverycited by 20
  10. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  11. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 18
  12. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  13. 2022SABLAS: Learning Safe Control for Black-box Dynamical Systemscited by 17
  14. 2024Empowering Large Language Models on Robotic Manipulation with Affordance Promptingcited by 17
  15. 2024Composable Part-Based Manipulationcited by 17
  16. 2023One-shot Imitation Learning via Interaction Warpingcited by 16
  17. 2022Transformers are Adaptable Task Plannerscited by 15
  18. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignmentcited by 14
  19. 2023The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulationcited by 12
  20. 2020A Geometric Perspective on Visual Imitation Learningcited by 10
  21. 2023Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspectivecited by 10
  22. 2022Where To Start? Transferring Simple Skills to Complex Environmentscited by 9
  23. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 9
  24. 2021OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimationcited by 8
  25. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 8
  26. 2022Semantic keypoint-based pose estimation from single RGB framescited by 7
  27. 2022Understanding Physical Effects for Effective Tool-usecited by 7
  28. 2023Learning to Design and Use Tools for Robotic Manipulationcited by 7
  29. 2023Learning Generalizable Tool-use Skills through Trajectory Generationcited by 6
  30. 2024One-Shot Manipulation Strategy Learning by Making Contact Analogiescited by 6
  31. 2025Constraint-Preserving Data Generation for Visuomotor Policy Learningcited by 6
  32. 2022Generalizable task representation learning from human demonstration videos: a geometric approachcited by 5
  33. 2023Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Renderingcited by 5
  34. 2022Creative Problem Solving in Artificially Intelligent Agents: A Survey and Frameworkcited by 4
  35. 2025Multi-Keypoint Affordance Representation for Functional Dexterous Graspingcited by 4
  36. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  37. 2021Generalizing Object-Centric Task-Axes Controllers using Keypointscited by 3
  38. 2022Learning Keypoints from Synthetic Data for Robotic Cloth Foldingcited by 3

50 most cited of 68 All 68 (JSON)