mlsanityData to 7 Oct 2026

Residual Reinforcement Learning from Demonstrations

arXiv 2106.08050 ↗First posted 15 Jun 2021cs.LGcited by 27

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Learning from images, proprioceptive inputs and a sparse task-completion reward relaxes the requirement of accessing full state features, such as object and target positions. In addition, replacing the base controller with a policy learned from demonstrations removes the dependency on a hand-engineered controller in favour of a dataset of demonstrations, which can be provided by non-experts. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning, and is capable of solving high-dimensional, sparse-reward tasks out of reach for RL from scratch.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 36 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

25 of 36 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2020Critic Regularized Regressioncited by 269
  12. 2018Residual Reinforcement Learning for Robot Controlcited by 242
13 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  2. 2018Residual Policy Learningcited by 146
  3. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  4. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  5. 2020Model-Based Offline Planningcited by 126
  6. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  7. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
  8. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  9. 2018Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncingcited by 36
  10. 2019TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfercited by 21
  11. 2019MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamicscited by 18
  12. 2019Side-Tuning: A Baseline for Network Adaptation via Additive Side Networkscited by 16
  13. 2019Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environmentscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 5; 2025: 10; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  3. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  4. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  5. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  6. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  7. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  8. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 34
  9. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  10. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  11. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 13
  12. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
15 more
  1. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  2. 2024Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Graspingcited by 8
  3. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  4. 2026TacCoRL: Integrating Tactile Feedback into VLA via Simulationcited by 3
  5. 2023Specialized Deep Residual Policy Safe Reinforcement Learning-Based Controller for Complex and Continuous State-Action Spacescited by 2
  6. 2025Factorizing Diffusion Policies for Observation Modality Prioritizationcited by 2
  7. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  8. 2026An Agency-Transferring Model-Free Policy Enhancement Techniquecited by 1
  9. 2025Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidancecited by 0
  10. 2025Learning-based Autonomous Oversteer Control and Collision Avoidancecited by 0
  11. 2026DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Handcited by 0
  12. 2026AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidancecited by 0
  13. 2026Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learningcited by 0
  14. 2026C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Videocited by 0
  15. 2026VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
  2. 2026Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
  3. 2020Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation
  4. 2024Reward Machine Inference for Robotic Manipulation
  5. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
  6. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrations
  7. 2026Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models
  8. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward