mlsanityData to 7 Oct 2026

Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrations

arXiv 2410.14957 ↗First posted 19 Oct 2024cs.RO cs.AI cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Offline-to-online reinforcement learning (O2O RL) aims to obtain a continually improving policy as it interacts with the environment, while ensuring the initial policy behaviour is satisficing. This satisficing behaviour is necessary for robotic manipulation where random exploration can be costly due to catastrophic failures and time. O2O RL is especially compelling when we can only obtain a scarce amount of (potentially suboptimal) demonstrationsunicodex2014a scenario where behavioural cloning (BC) is known to suffer from distribution shift. Previous works have outlined the challenges in applying O2O RL algorithms under the image-based environments. In this work, we propose a novel O2O RL algorithm that can learn in a real-life image-based robotic vacuum grasping task with a small number of demonstrations where BC fails majority of the time. The proposed algorithm replaces the target network in off-policy actor-critic algorithms with a regularization technique inspired by neural tangent kernel. We demonstrate that the proposed algorithm can reach above 90% success rate in under two hours of interaction time, with only 50 human demonstrations, while BC and existing commonly-used RL algorithms fail to achieve similar performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 63 of its 75 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

63 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Equivariant Offline Reinforcement Learning20 Jun
  2. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning27 Oct

Other references · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
38 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2020Reinforcement Learning with Augmented Datacited by 452
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  13. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  14. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  15. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  16. 2018Self-Imitation Learningcited by 135
  17. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  18. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  19. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  20. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  21. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  22. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  23. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 89
  24. 2024Foundation Policies with Hilbert Representationscited by 89
  25. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learningcited by 80
  26. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  27. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  28. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  29. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  30. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  31. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  32. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  33. 2020Safe Reinforcement Learning with Natural Language Constraintscited by 22
  34. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  35. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  36. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  37. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  38. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  2. 2025Accelerating Transformers in Online RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  2. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  3. 2023Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings
  4. 2026Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
  6. 2025Dexterous Grasping with Real-World Robotic Reinforcement Learning
  7. 2025RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
  8. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policies