# Virtual Training for a Real Application: Accurate Object-Robot Relative Localization without Calibration

arXiv: https://arxiv.org/abs/1902.02711 · HTML: https://mlsanity.com/paper/1902.02711/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2019.

## Built on by (most cited first, 5 of 5)

- [NOPE: Novel Object Pose Estimation from a Single Image](https://mlsanity.com/paper/2303.13612/) (2023)
- [Single-view robot pose and joint angle estimation via render & compare](https://mlsanity.com/paper/2104.09359/) (2021)
- [PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation](https://mlsanity.com/paper/2211.14125/) (2022)
- [Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos](https://mlsanity.com/paper/2111.01591/) (2021)
- [6D Robotic Assembly Based on RGB-only Object Pose Estimation](https://mlsanity.com/paper/2208.12986/) (2022)

## Abstract (arXiv)

Localizing an object accurately with respect to a robot is a key step for autonomous robotic manipulation. In this work, we propose to tackle this task knowing only 3D models of the robot and object in the particular case where the scene is viewed from uncalibrated cameras -- a situation which would be typical in an uncontrolled environment, e.g., on a construction site. We demonstrate that this localization can be performed very accurately, with millimetric errors, without using a single real image for training, a strong advantage since acquiring representative training data is a long and expensive process. Our approach relies on a classification Convolutional Neural Network (CNN) trained using hundreds of thousands of synthetically rendered scenes with randomized parameters. To evaluate our approach quantitatively and make it comparable to alternative approaches, we build a new rich dataset of real robot images with accurately localized blocks.