# Residual Reinforcement Learning for Robot Control

arXiv: https://arxiv.org/abs/1812.03201 · HTML: https://mlsanity.com/paper/1812.03201/

## Key facts

- Cited by 274 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2018.

## Built on by (most cited first, 30 of 274)

- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy](https://mlsanity.com/paper/2502.05450/) (2025)
- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills](https://mlsanity.com/paper/2502.01143/) (2025)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration](https://mlsanity.com/paper/2105.06411/) (2021)
- [From Imitation to Refinement -- Residual RL for Precise Assembly](https://mlsanity.com/paper/2407.16677/) (2024)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations](https://mlsanity.com/paper/2303.01497/) (2023)
- [Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model](https://mlsanity.com/paper/2412.13630/) (2024)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction](https://mlsanity.com/paper/2405.10315/) (2024)
- [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025)
- [Adapting Rapid Motor Adaptation for Bipedal Robots](https://mlsanity.com/paper/2205.15299/) (2022)
- [Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation](https://mlsanity.com/paper/2201.07779/) (2022)
- [RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning](https://mlsanity.com/paper/2412.09858/) (2024)
- [ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning](https://mlsanity.com/paper/2510.05070/) (2025)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [Continual Semantic Segmentation with Automatic Memory Sample Selection](https://mlsanity.com/paper/2304.05015/) (2023)
- [Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning](https://mlsanity.com/paper/2204.02863/) (2022)
- [Grounding Multimodal Large Language Models in Actions](https://mlsanity.com/paper/2406.07904/) (2024)
- [Residual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2106.08050/) (2021)
- [DiWA: Diffusion Policy Adaptation with World Models](https://mlsanity.com/paper/2508.03645/) (2025)
- [Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study](https://mlsanity.com/paper/2103.11512/) (2021)
- [Human-in-the-Loop Task and Motion Planning for Imitation Learning](https://mlsanity.com/paper/2310.16014/) (2023)
- [Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections](https://mlsanity.com/paper/2506.16685/) (2025)
- [Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation](https://mlsanity.com/paper/2008.07682/) (2020)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)

## Abstract (arXiv)

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.