mlsanityData to 14 Oct 2026

SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

arXiv 2605.05863 ↗First posted 7 May 2026cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Incorporating prior data into online reinforcement learning accelerates training but typically forces a difficult trade-off between high computational costs and long, multi-stage training pipelines. While fixed-length stabilization phases are significantly more computationally efficient than static update schedules, they require task-dependent manual tuning, risking either the waste of prior knowledge or severe overfitting. To address this, we propose SOPE, an algorithm that uses an actor-aligned Off-Policy Policy Evaluation (OPE) signal as an automated early-stopping mechanism to dynamically control the length of offline training phases. By evaluating the critic on a held-out validation split under the current policy's action distribution, SOPE halts gradient updates exactly when out-of-distribution benefits saturate, eliminating the need for manual schedule tuning. Evaluated on 25 continuous control tasks from the Minari benchmark suite, SOPE improves baseline performance by up to 45.6% while reducing the required TFLOPs by up to 22x, thus balancing the tradeoff between sample and computational efficiency. These findings demonstrate that adaptive, evaluation-driven update schedules are more effective than relying on static, exhaustive update schedules.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 34 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

24 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Data11 Feb
  2. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning15 Jan

Other references · 22

  1. 2016Layer Normalizationcited by 5,669
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
10 more
  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  3. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  4. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  5. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  6. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  7. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  8. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  9. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  10. 2024Reinforcement Learning for High-Level Strategic Control in Tower Defense Gamescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ARC-RL: A Reinforcement Learning Playground Inspired by ARC Raiderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline RL Without Off-Policy Evaluation
  2. 2026Active Offline-to-Online Reinforcement Learning
  3. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  4. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  7. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL