mlsanityData to 7 Oct 2026

Joint Differentiable Optimization and Verification for Certified Reinforcement Learning

arXiv 2201.12243 ↗First posted 28 Jan 2022cs.LG cs.SY eess.SYcited by 11

Not on the map. Why?

Abstract (arXiv)

In model-based reinforcement learning for safety-critical control systems, it is important to formally certify system properties (e.g., safety, stability) under the learned controller. However, as existing methods typically apply formal verification after the controller has been learned, it is sometimes difficult to obtain any certificate, even after many iterations between learning and verification. To address this challenge, we propose a framework that jointly conducts reinforcement learning and formal verification by formulating and solving a novel bilevel optimization problem, which is differentiable by the gradients from the value function and certificates. Experiments on a variety of examples demonstrate the significant advantages of our framework over the model-based stochastic value gradient (SVG) method and the model-free proximal policy optimization (PPO) method in finding feasible controllers with barrier functions and Lyapunov functions that ensure system safety and stability.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

34 of 64 references.

References · 34

  1. 2019Differentiable Convex Optimization Layerscited by 458
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  3. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  4. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  5. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  6. 2020Neural Lyapunov Controlcited by 142
  7. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  8. 2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 91
  9. 2020Learning Control Barrier Functions from Expert Demonstrationscited by 88
  10. 2021Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificatescited by 71
  11. 2021Lyapunov-stable neural-network controlcited by 66
  12. 2018Reinforcement Learning for UAV Attitude Controlcited by 40
22 more
  1. 2021POLAR: A Polynomial Arithmetic Framework for Verifying Neural-Network Controlled Systemscited by 38
  2. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  3. 2020Neural Certificates for Safe Control Policiescited by 35
  4. 2019ReachNN: Reachability Analysis of Neural-Network Controlled Systemscited by 26
  5. 2021Safe Pontryagin Differentiable Programmingcited by 25
  6. 2020Learning Hybrid Control Barrier Functions from Datacited by 24
  7. 2022Learning Safe, Generalizable Perception-based Hybrid Control with Certificatescited by 24
  8. 2020Neural Contraction Metrics for Robust Estimation and Control: A Convex Optimization Approachcited by 22
  9. 2020Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAIcited by 21
  10. 2020One for Many: Transfer Learning for Building HVAC Controlcited by 16
  11. 2021Learning Robust Hybrid Control Barrier Functions for Uncertain Systemscited by 14
  12. 2018Reachability Analysis and Safety Verification for Neural Network Control Systemscited by 12
  13. 2022Physics-Aware Safety-Assured Design of Hierarchical Neural Network based Plannercited by 9
  14. 2020Energy-Efficient Control Adaptation with Safety Guarantees for Learning-Enabled Cyber-Physical Systemscited by 7
  15. 2020Learning Lyapunov Functions for Piecewise Affine Systems with Neural Network Controllerscited by 7
  16. 2022Safety-driven Interactive Planning for Neural Network-based Lane Changingcited by 6
  17. 2014Abstraction of Elementary Hybrid Systems by Variable Transformationcited by 4
  18. 2021Verification in the Loop: Correct-by-Construction Control Learning with Reach-avoid Guaranteescited by 4
  19. 2013Exponential-Condition-Based Barrier Certificate Generation for Safety Verification of Hybrid Systemscited by 3
  20. 2020Automatic Repair of Convex Optimization Problemscited by 3
  21. 2021Cocktail: Learn a Better Neural Network Controller from Multiple Experts via Adaptive Mixing and Robust Distillationcited by 3
  22. 2020Learning control for polynomial systems using sum of squares relaxationscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 5; 2025: 2.

11 papers.

  1. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  2. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  3. 2024Lyapunov-stable Neural Control for State and Output Feedback: A Novel Formulationcited by 23
  4. 2022Efficient Global Robustness Certification of Neural Networks via Interleaving Twin-Network Encodingcited by 15
  5. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delayscited by 10
  6. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  7. 2024Variational Delayed Policy Optimizationcited by 9
  8. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  9. 2025Learning Verifiable Control Policies Using Relaxed Verificationcited by 1
  10. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  11. 2024Inverse Delayed Reinforcement Learningcited by 0