mlsanityData to 7 Oct 2026

From Imitation to Refinement -- Residual RL for Precise Assembly

arXiv 2407.16677 ↗First posted 23 Jul 2024cs.RO cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks. However, we find that the ease of teaching comes at the cost of unreliable performance that saturates with increasing data for tasks requiring precision. The performance saturation can be attributed to two critical factors: (a) distribution shift resulting from the use of offline data and (b) the lack of closed-loop corrective control caused by action chucking (predicting a set of future actions executed open-loop) critical for BC performance. Our key insight is that by predicting action chunks, BC policies function more like trajectory "planners" than closed-loop controllers necessary for reliable execution. To address these challenges, we devise a simple yet effective method, ResiP (Residual for Precise Manipulation), that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities. ResiP augments a frozen, chunked BC model with a fully closed-loop residual policy trained with reinforcement learning (RL) that addresses distribution shifts and introduces closed-loop corrections over open-loop execution of action chunks predicted by the BC trajectory planner. Videos, code, and data: https://residual-assembly.github.io.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 81 of its 110 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

81 of 110 references.

References · 81

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  12. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
38 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  13. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  14. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  15. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  16. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  17. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  18. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  19. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  20. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  21. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  22. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  23. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  24. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  25. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  26. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  27. 2022Online Decision Transformercited by 194
  28. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  29. 2024Behavior Generation with Latent Actionscited by 159
  30. 2018Residual Policy Learningcited by 146
  31. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  32. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  33. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  34. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  35. 2022Rapid Locomotion via Reinforcement Learningcited by 130
  36. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  37. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  38. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 28; 2026: 50.

80 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  2. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  3. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  4. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  5. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  6. 2025ResMimic: From General Motion Tracking to Humanoid Whole-body Loco-Manipulation via Residual Learningcited by 34
  7. 2026Q-learning with Adjoint Matchingcited by 33
  8. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  9. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  10. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  11. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  12. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
38 more
  1. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 25
  2. 2025Inference-Time Enhancement of Generative Robot Policies via Predictive World Modelingcited by 22
  3. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  4. 2025Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfercited by 13
  5. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 12
  6. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 10
  7. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  8. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  9. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  10. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  11. 2024Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learningcited by 6
  12. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transportcited by 6
  13. 2025Self-Improving Loops for Visual Robotic Planningcited by 6
  14. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuningcited by 5
  15. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  16. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Drivingcited by 4
  17. 2025LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrationscited by 3
  18. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
  19. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  20. 2025Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfercited by 2
  21. 2025Distilling Realizable Students from Unrealizable Teacherscited by 2
  22. 2025Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approachcited by 2
  23. 2025Factorizing Diffusion Policies for Observation Modality Prioritizationcited by 2
  24. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  25. 2026FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulationcited by 2
  26. 2026PPGuide: Steering Diffusion Policies with Performance Predictive Guidancecited by 2
  27. 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learningcited by 2
  28. 2026HDFlow: Hierarchical Diffusion-Flow Planning for Long-horizon Taskscited by 2
  29. 2026Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policiescited by 2
  30. 2026ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policiescited by 2
  31. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  32. 2026Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Modelscited by 2
  33. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  34. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  35. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  36. 2026Refinement of Accelerated Demonstrations via Incremental Iterative Reference Learning Control for Fast Contact-Rich Imitation Learningcited by 1
  37. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  38. 2026BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Modelscited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
  2. 2024Stable-BC: Controlling Covariate Shift with Stable Behavior Cloning
  3. 2025Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking
  4. 2025Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control
  5. 2026Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
  6. 2024How Generalizable Is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation
  7. 2025Robust Online Residual Refinement via Koopman-Guided Dynamics Modeling
  8. 2026Is Online Interaction Necessary for Recovery? A Minimalist Approach to Robust Planning via Perturbation ↗