# Imitrob: Imitation Learning Dataset for Training and Evaluating 6D Object Pose Estimators

arXiv: https://arxiv.org/abs/2209.07976 · HTML: https://mlsanity.com/paper/2209.07976/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [DPOD: 6D Pose Object Detector and Refiner](https://mlsanity.com/paper/1902.11020/) (2019)
- [EPOS: Estimating 6D Pose of Objects with Symmetries](https://mlsanity.com/paper/2004.00605/) (2020)
- [F, B, Alpha Matting](https://mlsanity.com/paper/2003.07711/) (2020)
- [LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes](https://mlsanity.com/paper/1707.04796/) (2017)
- [Learning Task-Oriented Grasping from Human Activity Datasets](https://mlsanity.com/paper/1910.11669/) (2019)
- [A Review on Object Pose Recovery: from 3D Bounding Box Detectors to Full 6D Pose Estimators](https://mlsanity.com/paper/2001.10609/) (2020)
- [CorNet: Generic 3D Corners for 6D Pose Estimation of New Objects without Retraining](https://mlsanity.com/paper/1908.11457/) (2019)
- [YCB-M: A Multi-Camera RGB-D Dataset for Object Recognition and 6DoF Pose Estimation](https://mlsanity.com/paper/2004.11657/) (2020)
- [Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos](https://mlsanity.com/paper/2111.01591/) (2021)
- [Teaching robots to imitate a human with no on-teacher sensors. What are the key challenges?](https://mlsanity.com/paper/1901.08335/) (2019)

## Built on by (most cited first, 3 of 3)

- [Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness](https://mlsanity.com/paper/2310.18626/) (2023)
- [MonoVisual3DFilter: 3D tomatoes' localisation with monocular cameras using histogram filters](https://mlsanity.com/paper/2310.05762/) (2023)
- [Long-Horizon Visual Imitation Learning via Plan and Code Reflection](https://mlsanity.com/paper/2509.05368/) (2025)

## Abstract (arXiv)

This paper introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera. Despite the significant progress of 6D pose estimation methods, their performance is usually limited for heavily occluded objects, which is a common case in imitation learning, where the object is typically partially occluded by the manipulating hand. Currently, there is a lack of datasets that would enable the development of robust 6D pose estimation methods for these conditions. To overcome this problem, we collect a new dataset (Imitrob) aimed at 6D pose estimation in imitation learning and other applications where a human holds a tool and performs a task. The dataset contains image sequences of nine different tools and twelve manipulation tasks with two camera viewpoints, four human subjects, and left/right hand. Each image is accompanied by an accurate ground truth measurement of the 6D object pose obtained by the HTC Vive motion tracking device. The use of the dataset is demonstrated by training and evaluating a recent 6D object pose estimation method (DOPE) in various setups.