mlsanityData to 7 Oct 2026

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

arXiv 2301.12203 ↗First posted 28 Jan 2023cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Offline safe RL is of great practical relevance for deploying agents in real-world applications. However, acquiring constraint-satisfying policies from the fixed dataset is non-trivial for conventional approaches. Even worse, the learned constraints are stationary and may become invalid when the online safety requirement changes. In this paper, we present a novel offline safe RL approach referred to as SaFormer, which tackles the above issues via conditional sequence modeling. In contrast to existing sequence models, we propose cost-related tokens to restrict the action space and a posterior safety verification to enforce the constraint explicitly. Specifically, SaFormer performs a two-stage auto-regression conditioned by the maximum remaining cost to generate feasible candidates. It then filters out unsafe attempts and executes the optimal action with the highest expected return. Extensive experiments demonstrate the efficacy of SaFormer featuring (1) competitive returns with tightened constraint satisfaction; (2) adaptability to the in-range cost values of the offline data without retraining; (3) generalizability for constraints beyond the current dataset.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 33 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

20 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2018Reward Constrained Policy Optimizationcited by 397
  11. 2020Critic Regularized Regressioncited by 269
  12. 2019Batch Policy Learning under Constraintscited by 242
7 more
  1. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  2. 2020Projection-Based Constrained Policy Optimizationcited by 195
  3. 2022Online Decision Transformercited by 194
  4. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  7. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 3; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  4. 2023Graph Decision Transformercited by 20
  5. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 12
  6. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  7. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  8. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RLcited by 8
  9. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  10. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
  11. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  12. 2023Resilient Constrained Reinforcement Learningcited by 1
4 more
  1. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  2. 2025Decision Flow Policy Optimizationcited by 0
  3. 2026MoMo: Conditioned Contrastive Representation Learning for Preference-Modulated Planningcited by 0
  4. 2026CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  2. 2025Online Optimization for Offline Safe Reinforcement Learning
  3. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
  4. 2024Offline Safe Reinforcement Learning Using Trajectory Classification
  5. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  6. 2024FOSP: Fine-tuning Offline Safe Policy through World Models
  7. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  8. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning