mlsanityData to 14 Oct 2026

Boundary-to-Region Supervision for Offline Safe Reinforcement Learning

arXiv 2509.25727 ↗First posted 30 Sep 2025cs.LG cs.AI cs.ROcited by 1

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe reinforcement learning aims to learn policies that satisfy predefined safety constraints from static datasets. Existing sequence-model-based methods condition action generation on symmetric input tokens for return-to-go and cost-to-go, neglecting their intrinsic asymmetry: return-to-go (RTG) serves as a flexible performance target, while cost-to-go (CTG) should represent a rigid safety boundary. This symmetric conditioning leads to unreliable constraint satisfaction, especially when encountering out-of-distribution cost trajectories. To address this, we propose Boundary-to-Region (B2R), a framework that enables asymmetric conditioning through cost signal realignment . B2R redefines CTG as a boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories while preserving reward structures. Combined with rotary positional embeddings , it enhances exploration within the safe region. Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks while achieving superior reward performance over baseline methods. This work highlights the limitations of symmetric token conditioning and establishes a new theoretical and practical approach for applying sequence models to safe RL. Our code is available at https://github.com/HuikangSu/B2R.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 41 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

33 of 41 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Offline Safe Reinforcement Learning Using Trajectory Classification19 Dec
  2. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning12 Dec
  3. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning11 Dec
  4. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning19 Jul
  5. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
  6. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration18 Sep
  7. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
  8. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
  9. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 24

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2019Learning Latent Plans from Playcited by 316
  10. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  11. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  12. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
12 more
  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  2. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  3. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  4. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  5. 2023Behavior Proximal Policy Optimizationcited by 43
  6. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  7. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  8. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  9. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 17
  10. 2022Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarioscited by 13
  11. 2025Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Modelscited by 13
  12. 2023Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penaltiescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
  2. 2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancy
  3. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  4. 2026Directional Constraints for Efficient Exploration in Safe Reinforcement Learning
  5. 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
  6. 2026Conditional Sequence Modeling for Safe Reinforcement Learning
  7. 2026Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning
  8. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning