mlsanityData to 14 Oct 2026

Q-learning-based Model-free Safety Filter

arXiv 2411.19809 ↗First posted 29 Nov 2024cs.RO cs.AI cs.SYcited by 1

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Ensuring safety via safety filters in real-world robotics presents significant challenges, particularly when the system dynamics is complex or unavailable. To handle this issue, learning-based safety filters recently gained popularity, which can be classified as model-based and model-free methods. Existing model-based approaches requires various assumptions on system model (e.g., control-affine), which limits their application in complex systems, and existing model-free approaches need substantial modifications to standard RL algorithms and lack versatility. This paper proposes a simple, plugin-and-play, and effective model-free safety filter learning framework. We introduce a novel reward formulation and use Q-learning to learn Q-value functions to safeguard arbitrary task specific nominal policies via filtering out their potentially unsafe actions. The threshold used in the filtering process is supported by our theoretical analysis. Due to its model-free nature and simplicity, our framework can be seamlessly integrated with various RL algorithms. We validate the proposed approach through simulations on double integrator and Dubin's car systems and demonstrate its effectiveness in real-world experiments with a soft robotic limb.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 32 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

23 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Safety Optimized Reinforcement Learning via Multi-Objective Policy Optimization23 Feb
  2. 2024Learning safety critics via a non-contractive binary bellman operator23 Jan

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2019Model-Based Reinforcement Learning for Ataricited by 502
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  10. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  11. 2020Projection-Based Constrained Policy Optimizationcited by 203
  12. 2022Rapid Locomotion via Reinforcement Learningcited by 168
9 more
  1. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  2. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  3. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 73
  4. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  5. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 59
  6. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  7. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 27
  8. 2019Probabilistic model predictive safety certification for learning-based controlcited by 22
  9. 2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Verifiable Safety Q-Filters via Hamilton-Jacobi Reachability and Multiplicative Q-Networkscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026Verifiable Model-Free Safety Filters via Reinforcement Learning
  2. 2026Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deployments
  3. 2026Safe-Support Q-Learning: Learning without Unsafe Exploration
  4. 2024Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning
  5. 2025Learning State-Action Control Barrier Functions for Model-Free Control under Constraints
  6. 2026COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection
  7. 2024Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents
  8. 2025Provably Optimal Reinforcement Learning under Safety Filtering