# Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditions

arXiv: https://arxiv.org/abs/2312.16438 · HTML: https://mlsanity.com/paper/2312.16438/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Scaled-YOLOv4: Scaling Cross Stage Partial Network](https://mlsanity.com/paper/2011.08036/) (2020)
- [Deep Spatial Autoencoders for Visuomotor Learning](https://mlsanity.com/paper/1509.06113/) (2015)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Deep Reinforcement Learning for High Precision Assembly Tasks](https://mlsanity.com/paper/1708.04033/) (2017)
- [Unsupervised Learning of Visual 3D Keypoints for Control](https://mlsanity.com/paper/2106.07643/) (2021)
- [Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2008.10224/) (2020)
- [Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks](https://mlsanity.com/paper/1602.02672/) (2016)
- [Contact-Rich Manipulation of a Flexible Object based on Deep Predictive Learning using Vision and Tactility](https://mlsanity.com/paper/2112.06442/) (2021)
- [Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/2012.04603/) (2020)
- [Spatial Attention Point Network for Deep-learning-based Robust Autonomous Robot Motion Generation](https://mlsanity.com/paper/2103.01598/) (2021)

## Built on by (most cited first, 5 of 5)

- [Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors](https://mlsanity.com/paper/2410.17551/) (2024)
- [Cognitive Manipulation: Semi-supervised Visual Representation and Classroom-to-real Reinforcement Learning for Assembly in Semi-structured Environments](https://mlsanity.com/paper/2406.00364/) (2024)
- [Context-Based Meta Reinforcement Learning for Robust and Adaptable Peg-in-Hole Assembly Tasks](https://mlsanity.com/paper/2409.16208/) (2024)
- [A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Tasks](https://mlsanity.com/paper/2504.14820/) (2025)
- [Trajectory Entropy Reinforcement Learning for Robust Robot Motor Skill Learning](https://mlsanity.com/paper/2505.04193/) (2025)

## Abstract (arXiv)

Anchor-bolt insertion is a peg-in-hole task performed in the construction field for holes in concrete. Efforts have been made to automate this task, but the variable lighting and hole surface conditions, as well as the requirements for short setup and task execution time make the automation challenging. In this study, we introduce a vision and proprioceptive data-driven robot control model for this task that is robust to challenging lighting and hole surface conditions. This model consists of a spatial attention point network (SAP) and a deep reinforcement learning (DRL) policy that are trained jointly end-to-end to control the robot. The model is trained in an offline manner, with a sample-efficient framework designed to reduce training time and minimize the reality gap when transferring the model to the physical world. Through evaluations with an industrial robot performing the task in 12 unknown holes, starting from 16 different initial positions, and under three different lighting conditions (two with misleading shadows), we demonstrate that SAP can generate relevant attention points of the image even in challenging lighting conditions. We also show that the proposed model enables task execution with higher success rate and shorter task completion time than various baselines. Due to the proposed model's high effectiveness even in severe lighting, initial positions, and hole conditions, and the offline training framework's high sample-efficiency and short training time, this approach can be easily applied to construction.