Data to 7 Oct 2026

Constrained Decision Transformer for Offline Safe Reinforcement Learning

arXiv 2302.07351 ↗First posted 14 Feb 2023cs.LG cs.AI cs.ROcited by 65

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Safe reinforcement learning (RL) trains a constraint satisfaction policy by interacting with the environment. We aim to tackle a more challenging problem: learning a safe policy from an offline dataset. We study the offline safe RL problem from a novel multi-objective optimization perspective and propose the ε-reducible concept to characterize problem difficulties. The inherent trade-offs between safety and task performance inspire us to propose the constrained decision transformer (CDT) approach, which can dynamically adjust the trade-offs during deployment. Extensive experiments show the advantages of the proposed method in learning an adaptive, safe, robust, and high-reward policy. CDT outperforms its variants and strong offline safe RL baselines by a large margin with the same hyperparameters across all tasks, while keeping the zero-shot adaptation capability to different constraint thresholds, making our approach more suitable for real-world RL under constraints. The code is available at https://github.com/liuzuxin/OSRL.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 67 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

50 of 67 references.

Closest ancestors · 1

Same direction, newest first

2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning19 Jul

Other references · 49

2017Proximal Policy Optimization Algorithmscited by 14,441
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Constrained Policy Optimizationcited by 928
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833

37 more

2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2018Reward Constrained Policy Optimizationcited by 397
2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
2020Critic Regularized Regressioncited by 269
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2019Batch Policy Learning under Constraintscited by 242
2018Safe Exploration in Continuous Action Spacescited by 237
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2020Projection-Based Constrained Policy Optimizationcited by 195
2022Online Decision Transformercited by 194
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2022SAAC: Safe Reinforcement Learning as an Adversarial Game of Actor-Criticscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 16; 2025: 18; 2026: 19.

65 papers.

Sort:

By area:

2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
2025Generative Auto-Bidding with Value-Guided Explorationscited by 23
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17

38 more

2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
2025Skill Expansion and Composition in Parameter Spacecited by 13
2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 7
2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
2023C-MCTS: Safe Planning with Monte Carlo Tree Searchcited by 6
2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
2025Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Alignercited by 5
2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
2025Provably-Safe Neural Network Training Using Hybrid Zonotope Reachability Analysiscited by 3
2025From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Controlcited by 3
2025Generative Large-Scale Pre-trained Models for Automated Ad Bidding Optimizationcited by 3
2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 3
2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
2025Safe In-Context Reinforcement Learningcited by 2
2025Model-Based Diffusion Sampling for Predictive Control in Offline Decision Makingcited by 2
2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
2023Resilient Constrained Reinforcement Learningcited by 1
2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 1
2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1
2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
2026In-Context Reinforcement Learning From Suboptimal Historical Datacited by 1
2026TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learningcited by 1
2024Transformer-based Stagewise Decomposition for Large-Scale Multistage Stochastic Optimizationcited by 0
2025GNN-DT: Graph Neural Network Enhanced Decision Transformer for Efficient Optimization in Dynamic Environmentscited by 0
2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewardscited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way