Data to 14 Oct 2026

Datasets and Benchmarks for Offline Safe Reinforcement Learning

arXiv 2306.09303 ↗First posted 15 Jun 2023cs.LG cs.AI cs.ROcited by 59

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

This paper presents a comprehensive benchmarking suite tailored to offline safe reinforcement learning (RL) challenges, aiming to foster progress in the development and evaluation of safe learning algorithms in both the training and deployment phases. Our benchmark suite contains three packages: 1) expertly crafted safe policies, 2) D4RL-styled datasets along with environment wrappers, and 3) high-quality offline safe RL baseline implementations. We feature a methodical data collection pipeline powered by advanced safe RL algorithms, which facilitates the generation of diverse datasets across 38 popular safe RL tasks, from robot control to autonomous driving. We further introduce an array of data post-processing filters, capable of modifying each dataset's diversity, thereby simulating various data collection conditions. Additionally, we provide elegant and extensible implementations of prevalent offline safe RL algorithms to accelerate research in this area. Through extensive experiments with over 50000 CPU and 800 GPU hours of computations, we evaluate and compare the performance of these baseline algorithms on the collected datasets, offering insights into their strengths, limitations, and potential areas of improvement. Our benchmarking framework serves as a valuable resource for researchers and practitioners, facilitating the development of more robust and reliable offline safe RL solutions in safety-critical applications. The benchmark website is available at urlwww.offline-saferl.org.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 75 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

53 of 75 references.

Closest ancestors · 3

Same direction, newest first

Other references · 50

2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Continuous control with deep reinforcement learningcited by 3,407
2015Trust Region Policy Optimizationcited by 2,580
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2016Concrete Problems in AI Safetycited by 1,642
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2017Constrained Policy Optimizationcited by 983

38 more

2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2019Behavior Regularized Offline Reinforcement Learningcited by 662
2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
2018Safe Exploration in Continuous Action Spacescited by 247
2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
2019Batch Policy Learning under Constraintscited by 244
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
2018Programmatically Interpretable Reinforcement Learningcited by 193
2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
2021Offline RL Without Off-Policy Evaluationcited by 151
2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
2018A Kernel Theory of Modern Data Augmentationcited by 106
2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
2016Fairness in Reinforcement Learningcited by 95
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
2022Constrained Policy Optimization via Bayesian World Modelscited by 57
2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 39
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
2022Reinforcement Learning with Stepwise Fairness Constraintscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 19; 2026: 18.

59 papers.

Sort:

By area:

2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 17
2025Skill Expansion and Composition in Parameter Spacecited by 14
2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 10
2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learningcited by 7
2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 6
2026Safe Exploration via Policy Priorscited by 6

38 more

2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
2025Learning Conservative Neural Control Barrier Functions from Offline Datacited by 5
2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
2024Adaptive Primal-Dual Method for Safe Reinforcement Learningcited by 4
2025Safety Representations for Safer Policy Learningcited by 4
2025Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Drivingcited by 4
2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 4
2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learningcited by 3
2025From Uncertain to Safe: Conformal Adaptation of Diffusion Models for Safe PDE Controlcited by 3
2025Model-Based Diffusion Sampling for Predictive Control in Offline Decision Makingcited by 3
2025A2Perf: Real-World Autonomous Agents Benchmarkcited by 2
2025A Provable Approach for End-to-End Safe Reinforcement Learningcited by 2
2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
2026A Recipe for Stable Offline Multi-agent Reinforcement Learningcited by 2
2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
2024Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraintscited by 1
2024Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learningcited by 1
2024A Safe Self-evolution Algorithm for Autonomous Driving Based on Data-Driven Risk Quantification Modelcited by 1
2024Safe PDE Boundary Control with Neural Operatorscited by 1
2025Offline Constrained Reinforcement Learning under Partial Data Coveragecited by 1
2025Semi-gradient DICE for Offline Constrained Reinforcement Learningcited by 1
2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
2024A Safety-Oriented Self-Learning Algorithm for Autonomous Driving: Evolution Starting from a Basic Modelcited by 0
2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
2025TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labelscited by 0
2025Skill-based Safe Reinforcement Learning with Risk Planningcited by 0
2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RLcited by 0
2025PyTupli: A Scalable Infrastructure for Collaborative Offline Reinforcement Learning Projectscited by 0
2025Prompting Decision Transformers for Zero-Shot Reach-Avoid Policiescited by 0
2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
2026LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchycited by 0
2026Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learningcited by 0
2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way