mlsanityData to 14 Oct 2026

To Switch or Not to Switch? Balanced Policy Switching in Offline Reinforcement Learning

arXiv 2407.01837 ↗First posted 1 Jul 2024stat.ML cs.IT cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) -- finding the optimal behaviour (also referred to as policy) maximizing the collected long-term cumulative reward -- is among the most influential approaches in machine learning with a large number of successful applications. In several decision problems, however, one faces the possibility of policy switching -- changing from the current policy to a new one -- which incurs a non-negligible cost, and in the decision one is limited to using historical data without the availability for further online interaction. Despite the inevitable importance of this offline learning scenario, to our best knowledge, very little effort has been made to tackle the key problem of balancing between the gain and the cost of switching in a flexible and principled way. Leveraging ideas from the area of optimal transport, we initialize the systematic study of policy switching in offline RL. We establish fundamental properties and design a Net Actor-Critic algorithm for the proposed novel switching formulation. Numerical experiments demonstrate the efficiency of our approach on multiple robot control benchmarks of the Gymnasium and traffic light control from SUMO-RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 52 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

22 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples11 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  9. 2019Batch Policy Learning under Constraintscited by 244
  10. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  11. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  12. 2017Device Placement Optimization with Reinforcement Learningcited by 86
6 more
  1. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  2. 2018Convergence and Concentration of Empirical Measures under Wasserstein Distance in Unbounded Functional Spacescited by 41
  3. 2021A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Costcited by 40
  4. 2022Sample-Efficient Reinforcement Learning with loglog(T) Switching Costcited by 32
  5. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  6. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  3. 2024Rethinking Optimal Transport in Offline Reinforcement Learning
  4. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2021Offline RL Without Off-Policy Evaluation
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions