# HOnnotate: A method for 3D Annotation of Hand and Object Poses

arXiv: https://arxiv.org/abs/1907.01481 · HTML: https://mlsanity.com/paper/1907.01481/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2019.

## Built on by (most cited first, 7 of 7)

- [ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis](https://mlsanity.com/paper/2109.05488/) (2021)
- [Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild](https://mlsanity.com/paper/2209.12009/) (2022)
- [Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation](https://mlsanity.com/paper/2106.05954/) (2021)
- [Fast Monocular Hand Pose Estimation on Embedded Systems](https://mlsanity.com/paper/2102.07067/) (2021)
- [Parallel mesh reconstruction streams for pose estimation of interacting hands](https://mlsanity.com/paper/2104.12123/) (2021)
- [Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos](https://mlsanity.com/paper/2605.14462/) (2026)
- [VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation](https://mlsanity.com/paper/2610.10197/) (2026)

## Abstract (arXiv)

We propose a method for annotating images of a hand manipulating an object with the 3D poses of both the hand and the object, together with a dataset created using this method. Our motivation is the current lack of annotated real images for this problem, as estimating the 3D poses is challenging, mostly because of the mutual occlusions between the hand and the object. To tackle this challenge, we capture sequences with one or several RGB-D cameras and jointly optimize the 3D hand and object poses over all the frames simultaneously. This method allows us to automatically annotate each frame with accurate estimates of the poses, despite large mutual occlusions. With this method, we created HO-3D, the first markerless dataset of color images with 3D annotations for both the hand and object. This dataset is currently made of 77,558 frames, 68 sequences, 10 persons, and 10 objects. Using our dataset, we develop a single RGB image-based method to predict the hand pose when interacting with objects under severe occlusions and show it generalizes to objects not seen in the dataset.