# Residual Reinforcement Learning from Demonstrations arXiv: https://arxiv.org/abs/2106.08050 ยท HTML: https://mlsanity.com/paper/2106.08050/ ## Key facts - Cited by 31 papers in the map (data through 14 Oct 2026). - Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. - Builds on 25 of its 36 references that are in the map. - First posted on arXiv 15 Jun 2021. ## Placed in - Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/) ## Builds on (most cited first, 25 of 25) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020) - [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017) - [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018) - [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017) - [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016) - [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017) - [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018) - [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017) - [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016) - [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020) - [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018) - [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018) - [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018) - [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020) - [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019) - [Model-Based Offline Planning](https://mlsanity.com/paper/2008.05556/) (2020) - [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018) - [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019) - [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020) - [Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing](https://mlsanity.com/paper/1808.03246/) (2018) - [TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer](https://mlsanity.com/paper/1907.11200/) (2019) - [MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics](https://mlsanity.com/paper/1909.13111/) (2019) - [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://mlsanity.com/paper/1912.13503/) (2019) - [Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments](https://mlsanity.com/paper/1909.10972/) (2019) ## Built on by (most cited first, 30 of 31) - [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024) - [ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills](https://mlsanity.com/paper/2502.01143/) (2025) - [From Imitation to Refinement -- Residual RL for Precise Assembly](https://mlsanity.com/paper/2407.16677/) (2024) - [Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations](https://mlsanity.com/paper/2303.01497/) (2023) - [Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model](https://mlsanity.com/paper/2412.13630/) (2024) - [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025) - [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025) - [ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learning](https://mlsanity.com/paper/2510.05070/) (2025) - [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022) - [Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion](https://mlsanity.com/paper/2402.16075/) (2024) - [Touch begins where vision ends: Generalizable policies for contact-rich manipulation](https://mlsanity.com/paper/2506.13762/) (2025) - [Learning a Universal Human Prior for Dexterous Manipulation from Human Preference](https://mlsanity.com/paper/2304.04602/) (2023) - [Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping](https://mlsanity.com/paper/2410.02475/) (2024) - [RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation](https://mlsanity.com/paper/2602.01789/) (2026) - [Accelerating Residual Reinforcement Learning with Uncertainty Estimation](https://mlsanity.com/paper/2506.17564/) (2025) - [RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learning](https://mlsanity.com/paper/2401.14858/) (2024) - [Factorizing Diffusion Policies for Observation Modality Prioritization](https://mlsanity.com/paper/2509.16830/) (2025) - [TacCoRL: Integrating Tactile Feedback into VLA via Simulation](https://mlsanity.com/paper/2606.11743/) (2026) - [Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spaces](https://mlsanity.com/paper/2310.14788/) (2023) - [Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics](https://mlsanity.com/paper/2507.00611/) (2025) - [An Agency-Transferring Model-Free Policy Enhancement Technique](https://mlsanity.com/paper/2606.09825/) (2026) - [AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance](https://mlsanity.com/paper/2606.29209/) (2026) - [Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance](https://mlsanity.com/paper/2504.18766/) (2025) - [Learning-based Autonomous Oversteer Control and Collision Avoidance](https://mlsanity.com/paper/2505.15275/) (2025) - [Residual Reinforcement Learning for Waste-Container Lifting Using Large-Scale Cranes with Underactuated Tools](https://mlsanity.com/paper/2602.05895/) (2026) - [DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand](https://mlsanity.com/paper/2606.28323/) (2026) - [Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learning](https://mlsanity.com/paper/2607.11530/) (2026) - [C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video](https://mlsanity.com/paper/2608.07045/) (2026) - [Hybrid Residual Reinforcement Learning for Contact-Rich Robotic Book Insertion](https://mlsanity.com/paper/2609.19962/) (2026) - [Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation](https://mlsanity.com/paper/2609.30023/) (2026) ## Abstract (arXiv) Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Learning from images, proprioceptive inputs and a sparse task-completion reward relaxes the requirement of accessing full state features, such as object and target positions. In addition, replacing the base controller with a policy learned from demonstrations removes the dependency on a hand-engineered controller in favour of a dataset of demonstrations, which can be provided by non-experts. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning, and is capable of solving high-dimensional, sparse-reward tasks out of reach for RL from scratch.