mlsanityData to 14 Oct 2026

kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation

arXiv 1903.06684 ↗First posted 15 Mar 2019cs.ROcited by 153

Not on the map. Why?

Abstract (arXiv)

We would like robots to achieve purposeful manipulation by placing any instance from a category of objects into a desired set of goal states. Existing manipulation pipelines typically specify the desired configuration as a target 6-DOF pose and rely on explicitly estimating the pose of the manipulated objects. However, representing an object with a parameterized transformation defined on a fixed template cannot capture large intra-category shape variation, and specifying a target pose at a category level can be physically infeasible or fail to accomplish the task -- e.g. knowing the pose and size of a coffee mug relative to some canonical mug is not sufficient to successfully hang it on a rack by its handle. Hence we propose a novel formulation of category-level manipulation that uses semantic 3D keypoints as the object representation. This keypoint representation enables a simple and interpretable specification of the manipulation target as geometric costs and constraints on the keypoints, which flexibly generalizes existing pose-based manipulation methods. Using this formulation, we factor the manipulation policy into instance segmentation, 3D keypoint detection, optimization-based robot action planning and local dense-geometry-based action execution. This factorization allows us to leverage advances in these sub-problems and combine them into a general and effective perception-to-action manipulation pipeline. Our pipeline is robust to large intra-category shape variation and topology changes as the keypoint representation ignores task-irrelevant geometric details. Extensive hardware experiments demonstrate our method can reliably accomplish tasks with never-before seen objects in a category, such as placing shoes and mugs with significant shape variation into category level target configurations.

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 20; 2022: 29; 2023: 31; 2024: 32; 2025: 17; 2026: 23.

153 papers.

Sort:

By area:

  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  3. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  4. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  5. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
  6. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  7. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 201
  8. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  9. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 136
  10. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  11. 2023GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondencecited by 114
  12. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
38 more
  1. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  2. 2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
  3. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  4. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 81
  5. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  6. 2022Zero-Shot Category-Level Object Pose Estimationcited by 57
  7. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  8. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  9. 2021SAR-Net: Shape Alignment and Recovery Network for Category-level 6D Object Pose and Size Estimationcited by 53
  10. 2022HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarioscited by 53
  11. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  12. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  13. 2021KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Controlcited by 51
  14. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  15. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  16. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  17. 2022PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objectscited by 45
  18. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 44
  19. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  20. 2021Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Imagecited by 42
  21. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  22. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  23. 2021StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimationcited by 40
  24. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 40
  25. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  26. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 37
  27. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  28. 2023Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagationcited by 33
  29. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  30. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  31. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  32. 2022USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulationcited by 27
  33. 2021ZePHyR: Zero-shot Pose Hypothesis Ratingcited by 26
  34. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Controlcited by 24
  35. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  36. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  37. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  38. 2023Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulationcited by 21

50 most cited of 153 All 153 (JSON)