mlsanityData to 7 Oct 2026

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

arXiv 2603.15136 ↗First posted 16 Mar 2026cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 48 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

36 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning8 Feb
  2. 2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functions11 Dec
  3. 2024SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systems8 Apr
  4. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  5. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  6. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraints1 Jun
  7. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  8. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 28

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
16 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2017Safe Reinforcement Learning via Shieldingcited by 427
  3. 2018Reward Constrained Policy Optimizationcited by 397
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  6. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 261
  7. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  8. 2025Flow Q-Learningcited by 162
  9. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  10. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  11. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  12. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
  13. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  14. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  15. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  16. 2025A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systemscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026CrossSafe: Towards Cross-Embodiment Latent Safety Filterscited by 0
  2. 2026Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge Viewcited by 0
  3. 2026A Unified Bellman Operator for Safety-Critical Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
  2. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  3. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  4. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  5. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  6. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  7. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning
  8. 2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy