# Boundary-to-Region Supervision for Offline Safe Reinforcement Learning

arXiv: https://arxiv.org/abs/2509.25727 · HTML: https://mlsanity.com/paper/2509.25727/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 30 of 33)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2106.10783/) (2021)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Constraints Penalized Q-learning for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2107.09003/) (2021)
- [COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2204.08957/) (2022)
- [Constrained Decision Transformer for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2302.07351/) (2023)
- [Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model](https://mlsanity.com/paper/2401.10700/) (2024)
- [Datasets and Benchmarks for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2306.09303/) (2023)
- [DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11492/) (2021)
- [Behavior Proximal Policy Optimization](https://mlsanity.com/paper/2302.11312/) (2023)
- [Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning](https://mlsanity.com/paper/2306.12755/) (2023)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Reinformer: Max-Return Sequence Modeling for Offline RL](https://mlsanity.com/paper/2405.08740/) (2024)
- [Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2311.10747/) (2023)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios](https://mlsanity.com/paper/2210.02300/) (2022)
- [Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Models](https://mlsanity.com/paper/2508.11582/) (2025)
- [OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2407.14653/) (2024)
- [Offline Safe Reinforcement Learning Using Trajectory Classification](https://mlsanity.com/paper/2412.15429/) (2024)

## Built on by (most cited first, 1 of 1)

- [Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2605.02777/) (2026)

## Abstract (arXiv)

Offline safe reinforcement learning aims to learn policies that satisfy predefined safety constraints from static datasets. Existing sequence-model-based methods condition action generation on symmetric input tokens for return-to-go and cost-to-go, neglecting their intrinsic asymmetry: return-to-go (RTG) serves as a flexible performance target, while cost-to-go (CTG) should represent a rigid safety boundary. This symmetric conditioning leads to unreliable constraint satisfaction, especially when encountering out-of-distribution cost trajectories. To address this, we propose Boundary-to-Region (B2R), a framework that enables asymmetric conditioning through cost signal realignment . B2R redefines CTG as a boundary constraint under a fixed safety budget, unifying the cost distribution of all feasible trajectories while preserving reward structures. Combined with rotary positional embeddings , it enhances exploration within the safe region. Experimental results show that B2R satisfies safety constraints in 35 out of 38 safety-critical tasks while achieving superior reward performance over baseline methods. This work highlights the limitations of symmetric token conditioning and establishes a new theoretical and practical approach for applying sequence models to safe RL. Our code is available at https://github.com/HuikangSu/B2R.