GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RL
On the map: Offline RL and imitation learning from data → Safe offline RL
Abstract (arXiv)
Offline Safe Reinforcement Learning (OSRL) aims to learn a policy to achieve high performance in sequential decision-making while satisfying constraints, using only pre-collected datasets. Recent works, inspired by the strong capabilities of Generative Models (GMs), reformulate decision-making in OSRL as a conditional generative process, where GMs generate desirable actions conditioned on predefined reward and cost values. However, GM-assisted methods face two major challenges in OSRL: (1) lacking the ability to "stitch" optimal transitions from suboptimal trajectories within the dataset, and (2) struggling to balance reward targets with cost targets, particularly when they are conflict. To address these issues, we propose Goal-Assisted Stitching (GAS), a novel algorithm designed to enhance stitching capabilities while effectively balancing reward maximization and constraint satisfaction. To enhance the stitching ability, GAS first augments and relabels the dataset at the transition level, enabling the construction of high-quality trajectories from suboptimal ones. GAS also introduces novel goal functions, which estimate the optimal achievable reward and cost goals from the dataset. These goal functions, trained using expectile regression on the relabeled and augmented dataset, allow GAS to accommodate a broader range of reward-cost return pairs and achieve a better tradeoff between reward maximization and constraint satisfaction compared to human-specified values. The estimated goals then guide policy training, ensuring robust performance under constrained settings. Furthermore, to improve training stability and efficiency, we reshape the dataset to achieve a more uniform reward-cost return distribution. Empirical results validate the effectiveness of GAS, demonstrating superior performance in balancing reward maximization and constraint satisfaction compared to existing methods.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 41 references that are in the map. First posted on arXiv 5 Feb 2026.
Builds on
32 of 41 references.
Closest ancestors · 6
Same direction, newest first
- 2024Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning25 Dec
- 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning19 Jul
- 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model19 Jan
- 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning15 Jun
- 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning14 Feb
- 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul
Other references · 26
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2019Behavior Regularized Offline Reinforcement Learningcited by 654
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
- 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
- 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
- 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
14 more
- 2022Online Decision Transformercited by 194
- 2021Delving into Deep Imbalanced Regressioncited by 157
- 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
- 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
- 2022Balanced MSE for Imbalanced Visual Regressioncited by 89
- 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
- 2023Elastic Decision Transformercited by 66
- 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
- 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
- 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
- 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
- 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
- 2022Learning Imbalanced Datasets with Maximum Margin Losscited by 9
- 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Online Optimization for Offline Safe Reinforcement Learning
- 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
- 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning
- 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
- 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
- 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
- 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data
- 2024OGBench: Benchmarking Offline Goal-Conditioned RL