mlsanityData to 14 Oct 2026

Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL

arXiv 2505.08179 ↗First posted 13 May 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning(OSRL) derives constraint-satisfying policies from pre-collected datasets, offers a promising avenue for deploying RL in safety-critical real-world domains such as robotics. However, the majority of existing approaches emphasize only short-term safety, neglecting long-horizon considerations. Consequently, they may violate safety constraints and fail to ensure sustained protection during online deployment. Moreover, the learned policies often struggle to handle states and actions that are not present or out-of-distribution(OOD) from the offline dataset, and exhibit limited sample efficiency. To address these challenges, we propose a novel framework Feasibility-Aware offline Safe Reinforcement Learning with CVAE-based Pessimism (FASP). First, we employ Hamilton-Jacobi (H-J) reachability analysis to generate reliable safety labels, which serve as supervisory signals for training both a conditional variational autoencoder (CVAE) and a safety classifier. This approach not only ensures high sampling efficiency but also provides rigorous long-horizon safety guarantees. Furthermore, we utilize pessimistic estimation methods to estimate the Q-value of reward and cost, which mitigates the extrapolation errors induces by OOD actions, and penalize unsafe actions to enabled the agent to proactively avoid high-risk behaviors. Moreover, we theoretically prove the validity of this pessimistic estimation. Extensive experiments on DSRL benchmarks demonstrate that FASP algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 52 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

30 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  3. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 27

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  6. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  7. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  8. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  9. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  10. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 201
  11. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  12. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
15 more
  1. 2019Active Domain Randomizationcited by 170
  2. 2018Learning Equations for Extrapolation and Controlcited by 156
  3. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  4. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  5. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  6. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  7. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  8. 2023Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learningcited by 14
  9. 2020Sampling-based Reachability Analysis: A Random Set Theory Approach with Adversarial Samplingcited by 11
  10. 2023On Safety and Liveness Filtering Using Hamilton-Jacobi Reachability Analysiscited by 11
  11. 2020Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learningcited by 9
  12. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  13. 2024Scalable Synthesis of Formally Verified Neural Value Function for Hamilton-Jacobi Reachability Analysiscited by 7
  14. 2025Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimationcited by 6
  15. 2024Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Drivingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  2. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learning
  3. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning
  4. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
  5. 2026Sampling-Based Safe Reinforcement Learning
  6. 2026CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
  7. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
  8. 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning