# Safety-Constrained Reinforcement Learning for MDPs

arXiv: https://arxiv.org/abs/1510.05880 · HTML: https://mlsanity.com/paper/1510.05880/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2015.

## Built on by (most cited first, 30 of 33)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Safe Reinforcement Learning via Shielding under Partial Observability](https://mlsanity.com/paper/2204.00755/) (2022)
- [Safe Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2203.02638/) (2022)
- [COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking](https://mlsanity.com/paper/2209.07133/) (2022)
- [Reinforcement Learning with Almost Sure Constraints](https://mlsanity.com/paper/2112.05198/) (2021)
- [Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes (Technical Report)](https://mlsanity.com/paper/2112.09655/) (2021)
- [Thresholded Lexicographic Ordered Multiobjective Reinforcement Learning](https://mlsanity.com/paper/2408.13493/) (2024)
- [Certifiably Robust Policies for Uncertain Parametric Environments](https://mlsanity.com/paper/2408.03093/) (2024)
- [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023)
- [Shielding in Resource-Constrained Goal POMDPs](https://mlsanity.com/paper/2211.15349/) (2022)
- [Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties](https://mlsanity.com/paper/2301.11592/) (2023)
- [Personalized Reinforcement Learning with a Budget of Policies](https://mlsanity.com/paper/2401.06514/) (2024)
- [The Impact of the Geometric Properties of the Constraint Set in Safe Optimization with Bandit Feedback](https://mlsanity.com/paper/2305.00889/) (2023)
- [Learning to Act Safely with Limited Exposure and Almost Sure Certainty](https://mlsanity.com/paper/2105.08748/) (2021)
- [Shielded Deep Reinforcement Learning for Complex Spacecraft Tasking](https://mlsanity.com/paper/2403.05693/) (2024)
- [Federated Reinforcement Learning with Constraint Heterogeneity](https://mlsanity.com/paper/2405.03236/) (2024)
- [Reinforcement Learning for Sustainable Energy: A Survey](https://mlsanity.com/paper/2407.18597/) (2024)
- [Compositional shield synthesis for safe reinforcement learning in partial observability](https://mlsanity.com/paper/2509.12085/) (2025)
- [Robust Probabilistic Shielding for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2605.10293/) (2026)
- [Shields to Guarantee Probabilistic Safety in MDPs](https://mlsanity.com/paper/2605.10888/) (2026)
- [Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning](https://mlsanity.com/paper/2405.19414/) (2024)
- [Robust Parameter Learning for Uncertain MDPs](https://mlsanity.com/paper/2605.01339/) (2026)
- [Safe Learning of Uncertain Environments](https://mlsanity.com/paper/2103.01413/) (2021)
- [Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees](https://mlsanity.com/paper/2303.12558/) (2023)
- [Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects](https://mlsanity.com/paper/2304.03081/) (2023)
- [Safe Online Convex Optimization with Multi-Point Feedback](https://mlsanity.com/paper/2407.11471/) (2024)
- [A Safety-Oriented Self-Learning Algorithm for Autonomous Driving: Evolution Starting from a Basic Model](https://mlsanity.com/paper/2408.12190/) (2024)
- [Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand](https://mlsanity.com/paper/2503.05276/) (2025)

## Abstract (arXiv)

We consider controller synthesis for stochastic and partially unknown environments in which safety is essential. Specifically, we abstract the problem as a Markov decision process in which the expected performance is measured using a cost function that is unknown prior to run-time exploration of the state space. Standard learning approaches synthesize cost-optimal strategies without guaranteeing safety properties. To remedy this, we first compute safe, permissive strategies. Then, exploration is constrained to these strategies and thereby meets the imposed safety requirements. Exploiting an iterative learning procedure, the resulting policy is safety-constrained and optimal. We show correctness and completeness of the method and discuss the use of several heuristics to increase its scalability. Finally, we demonstrate the applicability by means of a prototype implementation.