# Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies

arXiv: https://arxiv.org/abs/2505.11858 · HTML: https://mlsanity.com/paper/2505.11858/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2025.

## Built on by (most cited first, 4 of 4)

- [Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies](https://mlsanity.com/paper/2507.06519/) (2025)
- [Control Synthesis with Reinforcement Learning: A Modeling Perspective](https://mlsanity.com/paper/2510.25063/) (2025)
- [EasyInsert: A Data-Efficient and Generalizable Insertion Policy](https://mlsanity.com/paper/2505.16187/) (2025)
- [Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System](https://mlsanity.com/paper/2606.22145/) (2026)

## Abstract (arXiv)

Object insertion under tight tolerances (< hspace-.02in 1mm) is an important but challenging assembly task as even small errors can result in undesirable contacts. Recent efforts focused on Reinforcement Learning (RL), which often depends on careful definition of dense reward functions. This work proposes an effective strategy for such tasks that integrates traditional model-based control with RL to achieve improved insertion accuracy. The policy is trained exclusively in simulation and is zero-shot transferred to the real system. It employs a potential field-based controller to acquire a model-based policy for inserting a plug into a socket given full observability in simulation. This policy is then integrated with residual RL, which is trained in simulation given only a sparse, goal-reaching reward. A curriculum scheme over observation noise and action magnitude is used for training the residual RL policy. Both policy components use as input the SE(3) poses of both the plug and the socket and return the plug's SE(3) pose transform, which is executed by a robotic arm using a controller. The integrated policy is deployed on the real system without further training or fine-tuning, given a visual SE(3) object tracker. The proposed solution and alternatives are evaluated across a variety of objects and conditions in simulation and reality. The proposed approach outperforms recent RL-based methods in this domain and prior efforts with hybrid policies. Ablations highlight the impact of each component of the approach.