mlsanityData to 14 Oct 2026

Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees

arXiv 2312.01456 ↗First posted 3 Dec 2023cs.LG cs.SY eess.SYcited by 14

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SpectRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided by learning neural network policies together with reach-avoid supermartingales (RASM) for the graph's sub-tasks and then composing them into a global policy. We also derive a tighter lower bound compared to previous work on the probability of reach-avoidance implied by a RASM, which is required to find a compositional policy with an acceptable probabilistic threshold for complex tasks with multiple edge policies. We implement a prototype of our approach and evaluate it on a Stochastic Nine Rooms environment.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 3 Dec 2023.

Builds on

43 of 69 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2017Safe Reinforcement Learning via Shieldingcited by 449
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  7. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  9. 2018Safe Exploration in Continuous Action Spacescited by 247
  10. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 151
  11. 2020Neural Lyapunov Controlcited by 142
  12. 2018Learning-based Model Predictive Control for Safe Explorationcited by 120
31 more
  1. 2016Reinforcement Learning With Temporal Logic Rewardscited by 117
  2. 2018The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systemscited by 88
  3. 2019Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guaranteescited by 77
  4. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 75
  5. 2019Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learningcited by 75
  6. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  7. 2020A Composable Specification Language for Reinforcement Learning Taskscited by 69
  8. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  9. 2021Compositional Reinforcement Learning from Logical Specificationscited by 67
  10. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 61
  11. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  12. 2019Safe Exploration for Interactive Machine Learningcited by 51
  13. 2020Neurosymbolic Reinforcement Learning with Formally Verified Explorationcited by 49
  14. 2020Learning Certified Control using Contraction Metriccited by 49
  15. 2021Stabilizing Neural Control Using Self-Learned Almost Lyapunov Criticscited by 35
  16. 2021Stability Verification in Stochastic Control Systems via Neural Network Supermartingalescited by 26
  17. 2019Robust Regression for Safe Exploration in Controlcited by 24
  18. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 23
  19. 2022Safety Certification for Stochastic Systems via Neural Barrier Functionscited by 22
  20. 2021On the (In)Tractability of Reinforcement Learning for LTL Objectivescited by 18
  21. 2019A Barrier Function Approach to Finite-Time Stochastic System Verification and Controlcited by 14
  22. 2022Safety Guarantees for Neural Network Dynamic Systems via Stochastic Barrier Functionscited by 14
  23. 2019StocHy: automated verification and synthesis of stochastic processescited by 13
  24. 2021Verifiable and Compositional Reinforcement Learning Systemscited by 13
  25. 2021Sampling-Based Robust Control of Autonomous Systems with Non-Gaussian Noisecited by 13
  26. 2014FAUST2: Formal Abstractions of Uncountable-STate STochastic processescited by 12
  27. 2022Probabilities Are Not Enough: Formal Controller Synthesis for Stochastic Dynamical Models with Epistemic Uncertaintycited by 11
  28. 2016Stochastic Invariants for Probabilistic Terminationcited by 8
  29. 2020AMYTISS: Parallelized Automated Controller Synthesis for Large-Scale Stochastic Systemscited by 8
  30. 2022Learning Provably Stabilizing Neural Controllers for Discrete-Time Stochastic Systemscited by 5
  31. 2022Verified Compositions of Neural Network Controllers for Temporal Logic Control Objectivescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  2. 2024Neural Model Checkingcited by 6
  3. 2024Neural Continuous-Time Supermartingale Certificatescited by 5
  4. 2024Policy Verification in Stochastic Dynamical Systems Using Logarithmic Neural Certificatescited by 4
  5. 2024Composing Reinforcement Learning Policies, with Formal Guaranteescited by 2
  6. 2024Certified Policy Verification and Synthesis for MDPs under Distributional Reach-avoidance Propertiescited by 2
  7. 2025VeRecycle: Reclaiming Guarantees from Probabilistic Certificates for Stochastic Dynamical Systems after Changecited by 0
  8. 2025Automating the Refinement of Reinforcement Learning Specificationscited by 0
  9. 2025Neural Proofs for Sound Verification and Control of Complex Systemscited by 0
  10. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learningcited by 0
  11. 2026Value Functions as Supermartingale Certificatescited by 0
  12. 2026Supermartingale Certificates for Parametric MDPscited by 0
2 more
  1. 2026Certifying Almost-Sure Reachability in Unknown MDPscited by 0
  2. 2026Runtime Monitoring of Supermartingale Certificatescited by 0