mlsanityData to 14 Oct 2026

Evaluation-Time Policy Switching for Offline Reinforcement Learning

arXiv 2503.12222 ↗First posted 15 Mar 2025cs.LG cs.AI stat.MLcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) looks at learning how to optimally solve tasks using a fixed dataset of interactions from the environment. Many off-policy algorithms developed for online learning struggle in the offline setting as they tend to over-estimate the behaviour of out of distributions actions. Existing offline RL algorithms adapt off-policy algorithms, employing techniques such as constraining the policy or modifying the value function to achieve good performance on individual datasets but struggle to adapt to different tasks or datasets of different qualities without tuning hyper-parameters. We introduce a policy switching technique that dynamically combines the behaviour of a pure off-policy RL agent, for improving behaviour, and a behavioural cloning (BC) agent, for staying close to the data. We achieve this by using a combination of epistemic uncertainty, quantified by our RL model, and a metric for aleatoric uncertainty extracted from the dataset. We show empirically that our policy switching technique can outperform not only the individual algorithms used in the switching process but also compete with state-of-the-art methods on numerous benchmarks. Our use of epistemic uncertainty for policy switching also allows us to naturally extend our method to the domain of offline to online fine-tuning allowing our model to adapt quickly and safely from online data, either matching or exceeding the performance of current methods that typically require additional modification or hyper-parameter fine-tuning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 34 references that are in the map. First posted on arXiv 15 Mar 2025.

Builds on

22 of 34 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 12

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2017Deep Reinforcement Learning that Matterscited by 754
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  12. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Meta Policy Switching for Secure UAV Deconfliction in Adversarial Airspacecited by 0
  2. 2026Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shiftcited by 0
  3. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  4. 2026A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  5. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias