mlsanityData to 14 Oct 2026

Guardian: Decoupling Exploration from Safety in Reinforcement Learning

arXiv 2510.22859 ↗First posted 26 Oct 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Hybrid offline--online reinforcement learning (O2O RL) promises both sample efficiency and robust exploration, but suffers from instability due to distribution shift between offline and online data. We introduce RLPD-GX, a framework that decouples policy optimization from safety enforcement: a reward-seeking learner explores freely, while a projection-based guardian guarantees rule-consistent execution and safe value backups. This design preserves the exploratory value of online interactions without collapsing to conservative policies. To further stabilize training, we propose dynamic curricula that gradually extend temporal horizons and anneal offline--online data mixing. We prove convergence via a contraction property of the guarded Bellman operator, and empirically show state-of-the-art performance on Atari-100k, achieving a normalized mean score of 3.02 (+45% over prior hybrid methods) with stronger safety and stability. Beyond Atari, ablations demonstrate consistent gains across safety-critical and long-horizon tasks, underscoring the generality of our design. Extensive and comprehensive results highlight decoupled safety enforcement as a simple yet principled route to robust O2O RL, suggesting a broader paradigm for reconciling exploration and safety in reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 51 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

38 of 51 references.

References · 38

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  5. 2017Safe Reinforcement Learning via Shieldingcited by 449
  6. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  7. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2017Deep Reinforcement Learning: An Overviewcited by 333
  10. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  11. 2021Mastering Atari Games with Limited Datacited by 237
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
26 more
  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  2. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  3. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  4. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  5. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 89
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  7. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  8. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 59
  9. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 58
  10. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 42
  11. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 40
  12. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 37
  13. 2025DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-offcited by 34
  14. 2025OSC: Cognitive Orchestration through Dynamic Knowledge Alignment in Multi-Agent LLM Collaborationcited by 30
  15. 2024Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Maskingcited by 19
  16. 2025Learning Dynamics of VLM Finetuningcited by 17
  17. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 14
  18. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  19. 2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
  20. 2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
  21. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  22. 2024Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficientcited by 4
  23. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  24. 2025Advancing Safe Mechanical Ventilation Using Offline RL With Hybrid Actions and Clinically Aligned Rewardscited by 3
  25. 2024A Survey of Offline and Online Learning-Based Algorithms for Multirotor UAVscited by 2
  26. 2025Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correctioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  2. 2025Online Optimization for Offline Safe Reinforcement Learning
  3. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
  4. 2025Safe Reinforcement Learning with Minimal Supervision
  5. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  6. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  7. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage
  8. 2024Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding