# Quickly Inserting Pegs into Uncertain Holes using Multi-view Images and Deep Network Trained on Synthetic Data

arXiv: https://arxiv.org/abs/1902.09157 · HTML: https://mlsanity.com/paper/1902.09157/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2019.

## Built on by (most cited first, 6 of 6)

- [ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility](https://mlsanity.com/paper/2309.16909/) (2023)
- [Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces](https://mlsanity.com/paper/2105.11283/) (2021)
- [6D Robotic Assembly Based on RGB-only Object Pose Estimation](https://mlsanity.com/paper/2208.12986/) (2022)
- [Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes](https://mlsanity.com/paper/2205.04297/) (2022)
- [Economical Precise Manipulation and Auto Eye-Hand Coordination with Binocular Visual Reinforcement Learning](https://mlsanity.com/paper/2205.05963/) (2022)
- [Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Models](https://mlsanity.com/paper/2503.06026/) (2025)

## Abstract (arXiv)

This paper uses robots to assemble pegs into holes on surfaces with different colors and textures. It especially targets at the problem of peg-in-hole assembly with initial position uncertainty. Two in-hand cameras and a force-torque sensor are used to account for the position uncertainty. A program sequence comprising learning-based visual servoing, spiral search, and impedance control is implemented to perform the peg-in-hole task with feedback from the above sensors. Contributions are mainly made in the learning-based visual servoing of the sequence, where a deep neural network is trained with various sets of synthetic data generated using the concept of domain randomization to predict where a hole is. In the experiments and analysis section, the network is analyzed and compared, and a real-world robotic system to insert pegs to holes using the proposed method is implemented. The results show that the implemented peg-in-hole assembly system can perform successful peg-in-hole insertions on surfaces with various colors and textures. It can generally speed up the entire peg-in-hole process.