From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline-to-online reinforcement learning (O2O RL) faces a central challenge between retaining offline conservatism and adapting to online feedback under distribution shift. This challenge arises because data behavior evolves during fine-tuning, rendering data origin a misleading basis for constraint handling and thereby leading to objective-data mismatch. We therefore propose Dynamic Alignment for RElaxation (DARE), a distribution-aware framework for sample-level constraint relaxation based on the behavioral consistency with a behavior model. To our knowledge, DARE is the first to condition constraint relaxation on behavioral consistency via a posterior-induced exchange mechanism, moving beyond a binary offline/online data distinction. Importantly, DARE requires only per-sample behavioral alignment, enabling instantiation on top of many offline algorithms with flexible choices of behavior models and fine-tuning objectives. We provide a theoretical analysis showing that behavior-based sample exchange consistently improves the distinction between offline-like and online-like subsets. Experiments on D4RL demonstrate that DARE consistently improves fine-tuning stability and achieves superior final performance over strong offline-to-online baselines. (The code is publicly available at urlhttps://github.com/lpzu/DARE.)
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 48 references that are in the map. First posted on arXiv 5 Nov 2025.
Builds on
38 of 48 references.
Closest ancestors · 20
Same direction, newest first
- 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption29 Sep
- 2025Reinforcement Learning with Action Chunking10 Jul
- 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL25 Dec
- 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
- 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
- 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning12 Dec
- 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
- 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
- 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
- 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
8 more
- 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
- 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 18
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
- 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
- 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
- 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 111
- 2023Synthetic Experience Replaycited by 101
6 more
- 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
- 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
- 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 25
- 2024Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Modelscited by 18
- 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
- 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
- 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL
- 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
- 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
- 2025A Clean Slate for Offline Reinforcement Learning
- 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
- 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
- 2025The Three Regimes of Offline-to-Online Reinforcement Learning