# Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty

arXiv: https://arxiv.org/abs/2106.04306 · HTML: https://mlsanity.com/paper/2106.04306/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1907.13098/) (2019)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Deep Reinforcement Learning for High Precision Assembly Tasks](https://mlsanity.com/paper/1708.04033/) (2017)
- [Learning Contact-Rich Manipulation Skills with Guided Policy Search](https://mlsanity.com/paper/1501.05611/) (2015)
- [Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks](https://mlsanity.com/paper/2004.14404/) (2020)
- [One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors](https://mlsanity.com/paper/1509.06841/) (2015)
- [MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics](https://mlsanity.com/paper/1909.13111/) (2019)
- [A Framework for Robot Manipulation: Skill Formalism, Meta Learning and Adaptive Control](https://mlsanity.com/paper/1805.08576/) (2018)
- [Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments](https://mlsanity.com/paper/1909.10972/) (2019)
- [Fast robust peg-in-hole insertion with continuous visual servoing](https://mlsanity.com/paper/2011.06399/) (2020)
- [Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control](https://mlsanity.com/paper/1910.07972/) (2019)
- [Transferable Force-Torque Dynamics Model for Peg-in-hole Task](https://mlsanity.com/paper/1912.00260/) (2019)

## Built on by (most cited first, 10 of 10)

- [Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2111.13129/) (2021)
- [Seq2Seq Imitation Learning for Tactile Feedback-based Manipulation](https://mlsanity.com/paper/2303.02646/) (2023)
- [Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priors](https://mlsanity.com/paper/2406.19768/) (2024)
- [RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning](https://mlsanity.com/paper/2401.14858/) (2024)
- [Impedance Primitive-augmented Hierarchical Reinforcement Learning for Sequential Tasks](https://mlsanity.com/paper/2508.19607/) (2025)
- [ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies](https://mlsanity.com/paper/2606.09630/) (2026)
- [Cognitive Manipulation: Semi-supervised Visual Representation and Classroom-to-real Reinforcement Learning for Assembly in Semi-structured Environments](https://mlsanity.com/paper/2406.00364/) (2024)
- [MoRe-ERL: Learning Motion Residuals using Episodic Reinforcement Learning](https://mlsanity.com/paper/2508.01409/) (2025)
- [DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand](https://mlsanity.com/paper/2606.28323/) (2026)
- [Hybrid Residual Reinforcement Learning for Contact-Rich Robotic Book Insertion](https://mlsanity.com/paper/2609.19962/) (2026)

## Abstract (arXiv)

While classic control theory offers state of the art solutions in many problem scenarios, it is often desired to improve beyond the structure of such solutions and surpass their limitations. To this end, residual policy learning (RPL) offers a formulation to improve existing controllers with reinforcement learning (RL) by learning an additive "residual" to the output of a given controller. However, the applicability of such an approach highly depends on the structure of the controller. Often, internal feedback signals of the controller limit an RL algorithm to adequately change the policy and, hence, learn the task. We propose a new formulation that addresses these limitations by also modifying the feedback signals to the controller with an RL policy and show superior performance of our approach on a contact-rich peg-insertion task under position and orientation uncertainty. In addition, we use a recent Cartesian impedance control architecture as the control framework which can be available to us as a black-box while assuming no knowledge about its input/output structure, and show the difficulties of standard RPL. Furthermore, we introduce an adaptive curriculum for the given task to gradually increase the task difficulty in terms of position and orientation uncertainty. A video showing the results can be found at https://youtu.be/SAZm_Krze7U .