# A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Tasks

arXiv: https://arxiv.org/abs/2504.14820 · HTML: https://mlsanity.com/paper/2504.14820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 31)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Generalization in Reinforcement Learning by Soft Data Augmentation](https://mlsanity.com/paper/2011.13389/) (2020)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation](https://mlsanity.com/paper/2107.00644/) (2021)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning](https://mlsanity.com/paper/2212.08860/) (2022)
- [TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning](https://mlsanity.com/paper/2306.13229/) (2023)
- [On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline](https://mlsanity.com/paper/2212.05749/) (2022)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning](https://mlsanity.com/paper/2407.15815/) (2024)
- [Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly](https://mlsanity.com/paper/1903.01066/) (2019)
- [Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning](https://mlsanity.com/paper/2209.09203/) (2022)
- [Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages](https://mlsanity.com/paper/2310.07418/) (2023)
- [TacSL: A Library for Visuotactile Sensor Simulation and Learning](https://mlsanity.com/paper/2408.06506/) (2024)
- [The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning](https://mlsanity.com/paper/2311.00924/) (2023)
- [AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries](https://mlsanity.com/paper/2407.08028/) (2024)
- [Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks](https://mlsanity.com/paper/2110.12618/) (2021)
- [Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks](https://mlsanity.com/paper/2102.04022/) (2021)
- [A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2405.17416/) (2024)
- [Learning Sequences of Manipulation Primitives for Robotic Assembly](https://mlsanity.com/paper/2011.00778/) (2020)
- [Visual Spatial Attention and Proprioceptive Data-Driven Reinforcement Learning for Robust Peg-in-Hole Task Under Variable Conditions](https://mlsanity.com/paper/2312.16438/) (2023)
- [Proactive Action Visual Residual Reinforcement Learning for Contact-Rich Tasks Using a Torque-Controlled Robot](https://mlsanity.com/paper/2010.13056/) (2020)
- [Hierarchical Primitive Composition: Simultaneous Activation of Skills with Inconsistent Action Dimensions in Multiple Hierarchies](https://mlsanity.com/paper/2110.01833/) (2021)

## Abstract (arXiv)

For peg-in-hole tasks, humans rely on binocular visual perception to locate the peg above the hole surface and then proceed with insertion. This paper draws insights from this behavior to enable agents to learn efficient assembly strategies through visual reinforcement learning. Hence, we propose a Separate Primitive Policy (S2P) to learn how to derive location and insertion actions simultaneously. S2P is compatible with model-free reinforcement learning algorithms. Ten insertion tasks featuring different polygons are developed as benchmarks for evaluations. Simulation experiments show that S2P can boost the sample efficiency and success rate even with force constraints. Real-world experiments are also performed to verify the feasibility of S2P. Ablations are finally given to discuss the generalizability of S2P and some factors that affect its performance.