mlsanityData to 7 Oct 2026

Residual Policy Learning

arXiv 1812.06298 ↗First posted 15 Dec 2018cs.RO cs.LGcited by 146

Not on the map. Why?

Abstract (arXiv)

We present Residual Policy Learning (RPL): a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch remains data-inefficient or intractable, but learning a residual on top of the initial controller can yield substantial improvements. We study RPL in six challenging MuJoCo tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration. For initial controllers, we consider both hand-designed policies and model-predictive controllers with known or learned transition models. By combining learning with control algorithms, RPL can perform long-horizon, sparse-reward tasks for which reinforcement learning alone fails. Moreover, we find that RPL consistently and substantially improves on the initial controllers. We argue that RPL is a promising approach for combining the complementary strengths of deep reinforcement learning and robotic control, pushing the boundaries of what either can achieve independently. Video and code at https://k-r-allen.github.io/residual-policy-learning/.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 12; 2023: 16; 2024: 16; 2025: 21; 2026: 60.

146 papers.

Sort:

By area:

  1. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  2. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  3. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  4. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  5. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  6. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  7. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  8. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  9. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  10. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  11. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  12. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 34
38 more
  1. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  2. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  3. 2021Residual Reinforcement Learning from Demonstrationscited by 27
  4. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  5. 2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 23
  6. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  7. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  8. 2022Robot Learning of Mobile Manipulation with Reachability Behavior Priorscited by 20
  9. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  10. 2021Injecting Knowledge in Data-driven Vehicle Trajectory Predictorscited by 18
  11. 2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 18
  12. 2021Composable Energy Policies for Reactive Motion Generation and Reinforcement Learningcited by 16
  13. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  14. 2021InsertionNet -- A Scalable Solution for Insertioncited by 15
  15. 2023AdaptNet: Policy Adaptation for Physics-Based Character Controlcited by 15
  16. 2022TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specificationcited by 13
  17. 2025Bootstrapped Model Predictive Controlcited by 13
  18. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 13
  19. 2022Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewardscited by 10
  20. 2024Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Surveycited by 10
  21. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  22. 2022Cascaded Compositional Residual Learning for Complex Interactive Behaviorscited by 9
  23. 2021Machine Learning for Mechanical Ventilation Controlcited by 8
  24. 2024PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrationscited by 8
  25. 2024Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Graspingcited by 8
  26. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  27. 2023Deep Model Predictive Optimizationcited by 7
  28. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 7
  29. 2021Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertaintycited by 6
  30. 2024Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systemscited by 6
  31. 2025SIME: Enhancing Policy Self-Improvement with Modal-level Explorationcited by 6
  32. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  33. 2021Solving the Real Robot Challenge using Deep Reinforcement Learningcited by 5
  34. 2023A Physics Enhanced Residual Learning (PERL) Framework for Vehicle Trajectory Predictioncited by 5
  35. 2024Generalizing Cooperative Eco-driving via Multi-residual Task Learningcited by 5
  36. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  37. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  38. 2021Grasp and Motion Planning for Dexterous Manipulation for the Real Robot Challengecited by 4

50 most cited of 146 All 146 (JSON)