mlsanityData to 7 Oct 2026

Safe Policy Improvement in Constrained Markov Decision Processes

arXiv 2210.11259 ↗First posted 20 Oct 2022cs.LG cs.AI cs.FLcited by 0

Not on the map. Why?

Abstract (arXiv)

The automatic synthesis of a policy through reinforcement learning (RL) from a given set of formal requirements depends on the construction of a reward signal and consists of the iterative application of many policy-improvement steps. The synthesis algorithm has to balance target, safety, and comfort requirements in a single objective and to guarantee that the policy improvement does not increase the number of safety-requirements violations, especially for safety-critical applications. In this work, we present a solution to the synthesis problem by solving its two main challenges: reward-shaping from a set of formal requirements and safe policy update. For the former, we propose an automatic reward-shaping procedure, defining a scalar reward signal compliant with the task specification. For the latter, we introduce an algorithm ensuring that the policy is improved in a safe fashion with high-confidence guarantees. We also discuss the adoption of a model-based RL algorithm to efficiently use the collected data and train a model-free agent on the predicted trajectories, where the safety violation does not have the same impact as in the real world. Finally, we demonstrate in standard control benchmarks that the resulting learning procedure is effective and robust even under heavy perturbations of the hyperparameters.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

34 of 66 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2018World Modelscited by 1,444
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  10. 2017Deep Reinforcement Learning that Matterscited by 727
  11. 2017Safe Reinforcement Learning via Shieldingcited by 427
  12. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
22 more
  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  2. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  3. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  4. 2018Safe Exploration in Continuous Action Spacescited by 237
  5. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  6. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  7. 2016Reinforcement Learning With Temporal Logic Rewardscited by 114
  8. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  9. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 97
  10. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 96
  11. 2021Compositional Reinforcement Learning from Logical Specificationscited by 65
  12. 2014Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraintscited by 53
  13. 2019Liability, Ethics, and Culture-Aware Behavior Specification using Rulebookscited by 41
  14. 2020Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Taskscited by 28
  15. 2019Neural Simplex Architecturecited by 27
  16. 2021Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racingcited by 22
  17. 2017A Policy Search Method For Temporal Logic Specified Reinforcement Learning Taskscited by 19
  18. 2020RTAMT: Online Robustness Monitors from STLcited by 19
  19. 2021Parallel and Multi-Objective Falsification with Scenic and VerifAIcited by 8
  20. 2021Hierarchical Potential-based Reward Shaping from Task Specificationscited by 7
  21. 2015Temporal Logic as Filteringcited by 4
  22. 2015Robust Satisfaction of Temporal Logic Specifications via Reinforcement Learningcited by 2

Built on by

Not cited yet.