mlsanityData to 14 Oct 2026

Safe-Support Q-Learning: Learning without Unsafe Exploration

arXiv 2604.25379 ↗First posted 28 Apr 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the policy induced from the trained Q-values and propose a parametric policy extraction method to approximate the optimal policy. Our approach provides a unified framework that can be adapted to different action spaces and types of behavior policies. Experimental results demonstrate that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than existing baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 46 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

29 of 46 references.

References · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2021Convex optimizationcited by 2,515
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
17 more
  1. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  4. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  5. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  6. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  7. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  8. 2020Conservative Safety Critics for Explorationcited by 113
  9. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  10. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  11. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  12. 2019Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamicscited by 47
  13. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  14. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 21
  15. 2021Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangiancited by 15
  16. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  17. 2023TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  2. 2026Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective
  3. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
  4. 2025CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
  5. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  6. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  7. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
  8. 2025Safety Representations for Safer Policy Learning