mlsanityData to 7 Oct 2026

Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

arXiv 2107.00591 ↗First posted 1 Jul 2021cs.RO cs.LGcited by 211

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://github.com/shlee94/Off2OnRL.

Key facts. Cited by 211 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 46 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

36 of 46 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  3. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  4. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
13 more
  1. 2015Importance Weighted Autoencoderscited by 454
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  3. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  4. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  5. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  7. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  8. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  13. 2020Experience Replay with Likelihood-free Importance Weightscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 25; 2023: 48; 2024: 47; 2025: 49; 2026: 40.

211 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2024Diffusion Policy Policy Optimizationcited by 255
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2022Online Decision Transformercited by 194
  6. 2023Mitigating the Alignment Tax of RLHFcited by 181
  7. 2025Flow Q-Learningcited by 162
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  9. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  10. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  11. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  12. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
38 more
  1. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  2. 2025Reinforcement Learning with Action Chunkingcited by 97
  3. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  5. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  6. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  7. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 59
  8. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  9. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  10. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  11. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  12. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  13. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  14. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  15. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  16. 2023Finetuning Offline World Models in the Real Worldcited by 39
  17. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  18. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  19. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  20. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  21. 2026Q-learning with Adjoint Matchingcited by 33
  22. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  23. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  24. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  25. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  26. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  27. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutionscited by 26
  28. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  29. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  30. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  31. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  32. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  33. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  34. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  35. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  36. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  37. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  38. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20

50 most cited of 211 All 211 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021The Least Restriction for Offline Reinforcement Learning
  2. 2019An Optimistic Perspective on Offline Reinforcement Learning
  3. 2022Offline Reinforcement Learning at Multiple Frequencies
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  5. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling