Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning (RL) enables training from fixed data without online interaction, but policies learned offline often struggle when deployed in dynamic environments due to distributional shift and unreliable value estimates on unseen state-action pairs. We introduce Behavior-Adaptive Q-Learning (BAQ), a framework designed to enable a smooth and reliable transition from offline to online RL. The key idea is to leverage an implicit behavioral model derived from offline data to provide a behavior-consistency signal during online fine-tuning. BAQ incorporates a dual-objective loss that (i) aligns the online policy toward the offline behavior when uncertainty is high, and (ii) gradually relaxes this constraint as more confident online experience is accumulated. This adaptive mechanism reduces error propagation from out-of-distribution estimates, stabilizes early online updates, and accelerates adaptation to new scenarios. Across standard benchmarks, BAQ consistently outperforms prior offline-to-online RL approaches, achieving faster recovery, improved robustness, and higher overall performance. Our results demonstrate that implicit behavior adaptation is a principled and practical solution for reliable real-world policy deployment.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 35 references that are in the map. First posted on arXiv 5 Nov 2025.
Builds on
25 of 35 references.
Closest ancestors · 19
Same direction, newest first
- 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning6 Jan
- 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning12 Dec
- 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
- 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
- 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
- 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
- 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
- 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
7 more
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
- 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
- 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 6
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
- 2018Behavioral Cloning from Observationcited by 492
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
- 2022Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scalecited by 13
- 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
- 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
- 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
- 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
- 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning
- 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
- 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
- 2021Offline RL Without Off-Policy Evaluation