mlsanityData to 14 Oct 2026

Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning

arXiv 2412.18946 ↗First posted 25 Dec 2024cs.LG cs.AIcited by 13

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning (OSRL) involves learning a decision-making policy to maximize rewards from a fixed batch of training data to satisfy pre-defined safety constraints. However, adapting to varying safety constraints during deployment without retraining remains an under-explored challenge. To address this challenge, we introduce constraint-adaptive policy switching (CAPS), a wrapper framework around existing offline RL algorithms. During training, CAPS uses offline data to learn multiple policies with a shared representation that optimize different reward and cost trade-offs. During testing, CAPS switches between those policies by selecting at each state the policy that maximizes future rewards among those that satisfy the current cost constraint. Our experiments on 38 tasks from the DSRL benchmark demonstrate that CAPS consistently outperforms existing methods, establishing a strong wrapper-based baseline for OSRL. The code is publicly available at https://github.com/yassineCh/CAPS.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 23 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

17 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  2. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints1 Jun

Other references · 15

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  9. 2017Deep Reinforcement Learning: An Overviewcited by 333
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  11. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  12. 2019Batch Policy Learning under Constraintscited by 244
3 more
  1. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  3. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 2
  2. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  3. 2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancycited by 1
  4. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  5. 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0
  6. 2025Meta Policy Switching for Secure UAV Deconfliction in Adversarial Airspacecited by 0
  7. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
  8. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchycited by 0
  9. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  10. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0
  11. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  12. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0
1 more
  1. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  2. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning
  3. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  4. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  5. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance