mlsanityData to 14 Oct 2026

Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning

arXiv 2512.10510 ↗First posted 11 Dec 2025cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-Online Reinforcement Learning (O2O RL) faces a critical dilemma in balancing the use of a fixed offline dataset with newly collected online experiences. Standard methods, often relying on a fixed data-mixing ratio, struggle to manage the trade-off between early learning stability and asymptotic performance. To overcome this, we introduce the Adaptive Replay Buffer (ARB), a novel approach that dynamically prioritizes data sampling based on a lightweight metric we call 'on-policyness'. Unlike prior methods that rely on complex learning procedures or fixed ratios, ARB is designed to be learning-free and simple to implement, seamlessly integrating into existing O2O RL algorithms. It assesses how closely collected trajectories align with the current policy's behavior and assigns a proportional sampling weight to each transition within that trajectory. This strategy effectively leverages offline data for initial stability while progressively focusing learning on the most relevant, high-rewarding online experiences. Our extensive experiments on D4RL benchmarks demonstrate that ARB consistently mitigates early performance degradation and significantly improves the final performance of various O2O RL algorithms, highlighting the importance of an adaptive, behavior-aware replay buffer design. Our code is publicly available at https://github.com/song970407/ARB.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 43 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

26 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
  3. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  5. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  6. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  9. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
1 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 13

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  7. 2020Revisiting Fundamentals of Experience Replaycited by 175
  8. 2022Jump-Start Reinforcement Learningcited by 112
  9. 2019Experience Replay Optimizationcited by 48
  10. 2023A Survey on Causal Reinforcement Learningcited by 45
  11. 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 18
  12. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
1 more
  1. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Replay-buffer engineering for noise-aware quantum circuit optimizationcited by 1
  2. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0
  3. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Offline-to-Online Reinforcement Learning
  2. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  3. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  4. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  5. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Data
  6. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  7. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL