# A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1810.01531 · HTML: https://mlsanity.com/paper/1810.01531/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2018.

## Built on by (most cited first, 30 of 36)

- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [XIRL: Cross-embodiment Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.03911/) (2021)
- [Offline Learning from Demonstrations and Unlabeled Experience](https://mlsanity.com/paper/2011.13885/) (2020)
- [Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model](https://mlsanity.com/paper/2412.13630/) (2024)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification](https://mlsanity.com/paper/2103.12656/) (2021)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study](https://mlsanity.com/paper/2103.11512/) (2021)
- [Learning Dynamics Models for Model Predictive Agents](https://mlsanity.com/paper/2109.14311/) (2021)
- [Scaling data-driven robotics with reward sketching and batch reinforcement learning](https://mlsanity.com/paper/1909.12200/) (2019)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [InsertionNet -- A Scalable Solution for Insertion](https://mlsanity.com/paper/2104.14223/) (2021)
- [Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation](https://mlsanity.com/paper/2112.00597/) (2021)
- [A Primer on Maximum Causal Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11409/) (2022)
- [Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning](https://mlsanity.com/paper/2503.01837/) (2025)
- [DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks](https://mlsanity.com/paper/2404.16779/) (2024)
- [Understanding Multi-Modal Perception Using Behavioral Cloning for Peg-In-a-Hole Insertion Tasks](https://mlsanity.com/paper/2007.11646/) (2020)
- [InsertionNet 2.0: Minimal Contact Multi-Step Insertion Using Multimodal Multiview Sensory Input](https://mlsanity.com/paper/2203.01153/) (2022)
- [ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/1910.12154/) (2019)
- [Minimizing Human Assistance: Augmenting a Single Demonstration for Deep Reinforcement Learning](https://mlsanity.com/paper/2209.11275/) (2022)
- [Real2Sim or Sim2Real: Robotics Visual Insertion using Deep Reinforcement Learning and Real2Sim Policy Adaptation](https://mlsanity.com/paper/2206.02679/) (2022)
- [A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning](https://mlsanity.com/paper/2108.03222/) (2021)
- [Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings](https://mlsanity.com/paper/2112.04910/) (2021)
- [Aligning an optical interferometer with beam divergence control and continuous action space](https://mlsanity.com/paper/2107.04457/) (2021)
- [Zero-Shot Transfer of Haptics-Based Object Insertion Policies](https://mlsanity.com/paper/2301.12587/) (2023)
- [Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments](https://mlsanity.com/paper/2303.03365/) (2023)
- [Neural Task Success Classifiers for Robotic Manipulation from Few Real Demonstrations](https://mlsanity.com/paper/2107.00722/) (2021)
- [Boosting Reinforcement Learning and Planning with Demonstrations: A Survey](https://mlsanity.com/paper/2303.13489/) (2023)

## Abstract (arXiv)

Insertion is a challenging haptic and visual control problem with significant practical value for manufacturing. Existing approaches in the model-based robotics community can be highly effective when task geometry is known, but are complex and cumbersome to implement, and must be tailored to each individual problem by a qualified engineer. Within the learning community there is a long history of insertion research, but existing approaches are typically either too sample-inefficient to run on real robots, or assume access to high-level object features, e.g. socket pose. In this paper we show that relatively minor modifications to an off-the-shelf Deep-RL algorithm (DDPG), combined with a small number of human demonstrations, allows the robot to quickly learn to solve these tasks efficiently and robustly. Our approach requires no modeling or simulation, no parameterized search or alignment behaviors, no vision system aside from raw images, and no reward shaping. We evaluate our approach on a narrow-clearance peg-insertion task and a deformable clip-insertion task, both of which include variability in the socket position. Our results show that these tasks can be solved reliably on the real robot in less than 10 minutes of interaction time, and that the resulting policies are robust to variance in the socket position and orientation.