Data to 7 Oct 2026

Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation

arXiv 2202.06558 ↗First posted 14 Feb 2022cs.LG cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Satisfying safety constraints almost surely (or with probability one) can be critical for the deployment of Reinforcement Learning (RL) in real-life applications. For example, plane landing and take-off should ideally occur with probability one. We address the problem by introducing Safety Augmented (Saute) Markov Decision Processes (MDPs), where the safety constraints are eliminated by augmenting them into the state-space and reshaping the objective. We show that Saute MDP satisfies the Bellman equation and moves us closer to solving Safe RL with constraints satisfied almost surely. We argue that Saute MDP allows viewing the Safe RL problem from a different perspective enabling new features. For instance, our approach has a plug-and-play nature, i.e., any RL algorithm can be "Sauteed". Additionally, state augmentation allows for policy generalization across safety constraints. We finally show that Saute RL algorithms can outperform their state-of-the-art counterparts when constraint satisfaction is of high importance.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 33 of its 60 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

33 of 60 references.

References · 33

2017Proximal Policy Optimization Algorithmscited by 14,441
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Trust Region Policy Optimizationcited by 2,503
2016OpenAI Gymcited by 1,684
2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
2017Constrained Policy Optimizationcited by 928
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278

21 more

2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
2018Safe Exploration in Continuous Action Spacescited by 237
2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
2020Projection-Based Constrained Policy Optimizationcited by 195
2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
2018Learning-based Model Predictive Control for Safe Explorationcited by 116
2020Conservative Safety Critics for Explorationcited by 107
2014Optimizing the CVaR via Samplingcited by 98
2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
2018Simple random search provides a competitive approach to reinforcement learningcited by 75
2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
2018Safely Learning to Control the Constrained Linear Quadratic Regulatorcited by 31
2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 28
2018Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigationcited by 15
2021DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Interventioncited by 4
2020SafePILCO: a software tool for safe and data-efficient policy synthesiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 16; 2024: 11; 2025: 10; 2026: 20.

63 papers.

Sort:

By area:

2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 27
2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26

38 more

2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17
2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 15
2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
2023Compositional Policy Learning in Stochastic Control Systems with Formal Guaranteescited by 12
2024Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectoriescited by 9
2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
2024SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learningcited by 8
2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
2026Safe Exploration via Policy Priorscited by 6
2022Effects of Safety State Augmentation on Safe Explorationcited by 5
2023ROSARL: Reward-Only Safe Reinforcement Learningcited by 5
2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
2025Safety Representations for Safer Policy Learningcited by 4
2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 4
2023Learning Adaptive Safety for Multi-Agent Systemscited by 3
2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
2022Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraintscited by 2
2023Distributional constrained reinforcement learning for supply chain optimizationcited by 2
2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
2022Finding Safe Zones of policies Markov Decision Processescited by 1
2023A Multiplicative Value Function for Safe and Efficient Reinforcement Learningcited by 1
2023Anytime-Competitive Reinforcement Learning with Policy Priorcited by 1
2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learningcited by 1
2026Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Explorationcited by 1
2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
2026CRAX: Fast Safe Reinforcement Learning Benchmarkingcited by 1
2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancycited by 1
2026Vision--Language Signals in Constrained RL: Safety Gains Without Anticipationcited by 1
2023Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimizationcited by 0
2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Modelscited by 0

50 most cited of 63 All 63 (JSON)