# A Self-supervised Learning System for Object Detection using Physics Simulation and Multi-view Pose Estimation

arXiv: https://arxiv.org/abs/1703.03347 · HTML: https://mlsanity.com/paper/1703.03347/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2017.

## Built on by (most cited first, 11 of 11)

- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds](https://mlsanity.com/paper/2103.01977/) (2021)
- [Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction](https://mlsanity.com/paper/2302.03793/) (2023)
- [OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation](https://mlsanity.com/paper/2201.07309/) (2022)
- [RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features](https://mlsanity.com/paper/2403.01731/) (2024)
- [Synthetic Image Data for Deep Learning](https://mlsanity.com/paper/2212.06232/) (2022)
- [Good Grasps Only: A data engine for self-supervised fine-tuning of pose estimation using grasp poses for verification](https://mlsanity.com/paper/2409.11512/) (2024)
- [Self-supervised Pre-training for Semantic Segmentation in an Indoor Scene](https://mlsanity.com/paper/2210.01884/) (2022)
- [A generic self-supervised learning (SSL) framework for representation learning from spectra-spatial feature of unlabeled remote sensing imagery](https://mlsanity.com/paper/2306.15836/) (2023)
- [rt-RISeg: Real-Time Model-Free Robot Interactive Segmentation for Active Instance-Level Object Understanding](https://mlsanity.com/paper/2507.10776/) (2025)

## Abstract (arXiv)

Progress has been achieved recently in object detection given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. This work proposes an autonomous process for training a Convolutional Neural Network (CNN) for object detection and pose estimation in robotic setups. The focus is on detecting objects placed in cluttered, tight environments, such as a shelf with multiple objects. In particular, given access to 3D object models, several aspects of the environment are physically simulated. The models are placed in physically realistic poses with respect to their environment to generate a labeled synthetic dataset. To further improve object detection, the network self-trains over real images that are labeled using a robust multi-view pose estimation process. The proposed training process is evaluated on several existing datasets and on a dataset collected for this paper with a Motoman robotic arm. Results show that the proposed approach outperforms popular training processes relying on synthetic - but not physically realistic - data and manual annotation. The key contributions are the incorporation of physical reasoning in the synthetic data generation process and the automation of the annotation process over real images.