mlsanityData to 7 Oct 2026

Active Advantage-Aligned Online Reinforcement Learning with Offline Data

arXiv 2502.07937 ↗First posted 11 Feb 2025cs.LG stat.MLcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Online reinforcement learning (RL) enhances policies through direct interactions with the environment, but faces challenges related to sample efficiency. In contrast, offline RL leverages extensive pre-collected data to learn policies, but often produces suboptimal results due to limited data coverage. Recent efforts integrate offline and online RL in order to harness the advantages of both approaches. However, effectively combining online and offline RL remains challenging due to issues that include catastrophic forgetting, lack of robustness to data quality and limited sample efficiency in data utilization. In an effort to address these challenges, we introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned (A3) sampling strategy that dynamically prioritizes data aligned with the policy's evolving needs from both online and offline sources, optimizing policy improvement. Moreover, we provide theoretical insights into the effectiveness of our active sampling strategy and conduct diverse empirical experiments and ablation studies, demonstrating that our method outperforms competing online RL techniques that leverage offline data.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 74 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

49 of 74 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
  2. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning30 May
  3. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective12 Mar
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  5. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  10. 2020Critic Regularized Regression26 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
24 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  4. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2018Selective Experience Replay for Lifelong Learningcited by 239
  7. 2020Revisiting Fundamentals of Experience Replaycited by 171
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  9. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  10. 2019When to use parametric models in reinforcement learning?cited by 118
  11. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  12. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  13. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  14. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  15. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  16. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  17. 2019Prioritized Sequence Experience Replaycited by 20
  18. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  19. 2022Actor Prioritized Experience Replaycited by 19
  20. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  21. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  22. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
  23. 2024Entropy-Reinforced Planning with Large Language Models for Drug Discoverycited by 6
  24. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  2. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  3. 2025FragmentGPT: A Unified GPT Model for Fragment Growing, Linking, and Merging in Molecular Designcited by 1
  4. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Datacited by 1
  5. 2026ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Offline-to-Online Reinforcement Learning
  2. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  3. 2022Leveraging Offline Data in Online Reinforcement Learning
  4. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  5. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  7. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
  8. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness