# Semi-automatic 3D Object Keypoint Annotation and Detection for the Masses

arXiv: https://arxiv.org/abs/2201.07665 · HTML: https://mlsanity.com/paper/2201.07665/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 31 references that are in the map.
- First posted on arXiv 19 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints](https://mlsanity.com/paper/1910.10750/) (2019)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes](https://mlsanity.com/paper/1707.04796/) (2017)
- [S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency](https://mlsanity.com/paper/2009.14711/) (2020)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [CornerNet-Lite: Efficient Keypoint Based Object Detection](https://mlsanity.com/paper/1904.08900/) (2019)
- [YCB-M: A Multi-Camera RGB-D Dataset for Object Recognition and 6DoF Pose Estimation](https://mlsanity.com/paper/2004.11657/) (2020)
- [SALT: A Semi-automatic Labeling Tool for RGB-D Video Sequences](https://mlsanity.com/paper/2102.10820/) (2021)
- [VersaVIS: An Open Versatile Multi-Camera Visual-Inertial Sensor Suite](https://mlsanity.com/paper/1912.02469/) (2019)
- [Dynamical Pose Estimation](https://mlsanity.com/paper/2103.06182/) (2021)

## Built on by (most cited first, 1 of 1)

- [Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps](https://mlsanity.com/paper/2209.12744/) (2022)

## Abstract (arXiv)

Creating computer vision datasets requires careful planning and lots of time and effort. In robotics research, we often have to use standardized objects, such as the YCB object set, for tasks such as object tracking, pose estimation, grasping and manipulation, as there are datasets and pre-learned methods available for these objects. This limits the impact of our research since learning-based computer vision methods can only be used in scenarios that are supported by existing datasets. In this work, we present a full object keypoint tracking toolkit, encompassing the entire process from data collection, labeling, model learning and evaluation. We present a semi-automatic way of collecting and labeling datasets using a wrist mounted camera on a standard robotic arm. Using our toolkit and method, we are able to obtain a working 3D object keypoint detector and go through the whole process of data collection, annotation and learning in just a couple hours of active time.