Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
On the map: Offline RL and imitation learning from data → Real-world robot offline RL
Abstract (arXiv)
End-to-end learning robotic manipulation with high data efficiency is one of the key challenges in robotics. The latest methods that utilize human demonstration data and unsupervised representation learning has proven to be a promising direction to improve RL learning efficiency. The use of demonstration data also allows "warming-up" the RL policies using offline data with imitation learning or the recently emerged offline reinforcement learning algorithms. However, existing works often treat offline policy learning and online exploration as two separate processes, which are often accompanied by severe performance drop during the offline-to-online transition. Furthermore, many robotic manipulation tasks involve complex sub-task structures, which are very challenging to be solved in RL with sparse reward. In this work, we propose a unified offline-to-online RL framework that resolves the transition performance drop issue. Additionally, we introduce goal-aware state information to the RL agent, which can greatly reduce task complexity and accelerate policy learning. Combined with an advanced unsupervised representation learning module, our framework achieves great training efficiency and performance compared with the state-of-the-art methods in multiple robotic manipulation tasks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 45 references that are in the map. First posted on arXiv 21 Oct 2021.
Builds on
38 of 45 references.
References · 38
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2015Trust Region Policy Optimizationcited by 2,580
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2016OpenAI Gymcited by 1,736
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
26 more
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
- 2018Learning Dexterous In-Hand Manipulationcited by 687
- 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
- 2019Behavior Regularized Offline Reinforcement Learningcited by 662
- 2020MOPO: Model-based Offline Policy Optimizationcited by 636
- 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
- 2020Reinforcement Learning with Augmented Datacited by 468
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
- 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
- 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
- 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
- 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
- 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
- 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
- 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
- 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
- 2020Learning Dense Rewards for Contact-Rich Manipulation Taskscited by 19
- 2020Disentangled Planning and Control in Vision Based Robotics via Reward Machinescited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
- 2025RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
- 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
- 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
- 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
- 2023Offline Skill Generalization via Task and Motion Planning
- 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
- 2026Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition