mlsanityData to 14 Oct 2026

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

arXiv 2210.13846 ↗First posted 25 Oct 2022cs.LG cs.ROcited by 34

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior policy. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble of Q functions to further increase the sample efficiency of online fine-tuning by performing a large number of learning updates. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark. Code is available: urlhttps://github.com/zhaoyi11/adaptivebc.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 51 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

38 of 51 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016OpenAI Gymcited by 1,736
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2014How transferable are features in deep neural networks?cited by 1,225
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
14 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2016Deep Exploration via Bootstrapped DQNcited by 527
  5. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  7. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2020Model-Based Offline Planningcited by 129
  11. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  12. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  13. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  14. 2020Ready Policy One: World Building Through Active Learningcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 8; 2025: 8; 2026: 7.

34 papers.

Sort:

By area:

  1. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  2. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  3. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  4. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  5. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  6. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  7. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  8. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
  9. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  10. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  11. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  12. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
22 more
  1. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  2. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 7
  3. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  4. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  5. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  6. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  7. 2024Offline Behavior Distillationcited by 2
  8. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 2
  9. 2026ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learningcited by 2
  10. 2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemblecited by 1
  11. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  12. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspectivecited by 1
  13. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  14. 2025State Diversity Matters in Offline Behavior Distillationcited by 1
  15. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  16. 2026DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuningcited by 1
  17. 2025Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidancecited by 0
  18. 2025Fine-Tuning without Performance Degradationcited by 0
  19. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  20. 2026APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Compositioncited by 0
  21. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  22. 2026SymVD: Symmetric Vision Language Action Distillation for Robot Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  2. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  5. 2022Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learning
  6. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  7. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  8. 2025The Three Regimes of Offline-to-Online Reinforcement Learning