mlsanityData to 14 Oct 2026

ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization

arXiv 2605.14497 ↗First posted 14 May 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning harnesses the stability of offline pretraining and the flexibility of online fine-tuning. A key challenge lies in the non-stationary distribution shift between offline datasets and the evolving online policy. Common approaches often rely on static mixing ratios or heuristic-based replay strategies, which lack adaptability to different environments and varying training dynamics, resulting in suboptimal tradeoff between stability and asymptotic performance. In this work, we propose Reinforcement Learning with Optimized Adaptive Data-mixing (ROAD), a dynamic plug-and-play framework that automates the data replay process. We identify a fundamental objective misalignment in existing approaches. To tackle this, we formulate the data selection problem as a bi-level optimization process, interpreting the data mixing strategy as a meta-decision governing the policy performance (outer-level) during online fine-tuning, while the conventional Q-learning updates operate at the inner level. To make it tractable, we propose a practical algorithm using a multi-armed bandit mechanism. This is guided by a surrogate objective approximating the bi-level gradient, which simultaneously maintains offline priors and prevents value overestimation. Our empirical results demonstrate that this approach consistently outperforms existing data replay methods across various datasets, eliminating the need for manual, context-specific adjustments while achieving superior stability and asymptotic performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 40 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

33 of 40 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning11 Dec
  2. 2025The Three Regimes of Offline-to-Online Reinforcement Learning1 Oct
  3. 2025Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation9 Aug
  4. 2025Reinforcement Learning with Action Chunking10 Jul
  5. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning11 Jun
  6. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Data11 Feb
  7. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  8. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  9. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  10. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  11. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  12. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
7 more
  1. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  5. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  6. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  7. 2020Reinforcement Learning for Combinatorial Optimization: A Surveycited by 236
  8. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  9. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  10. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
  11. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  12. 2024Functional Bilevel Optimization for Machine Learningcited by 20
2 more
  1. 2024Combinatorial Multivariant Multi-Armed Bandits with Applications to Episodic Reinforcement Learning and Beyondcited by 7
  2. 2025SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-trainingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  2. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  3. 2026Offline-Online Reinforcement Learning for Linear Mixture MDPs
  4. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2023Bi-Level Offline Policy Optimization with Limited Exploration
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness