mlsanityData to 7 Oct 2026

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

arXiv 2102.08539 ↗First posted 17 Feb 2021cs.LG cs.AI cs.SYcited by 12

Not on the map. Why?

Abstract (arXiv)

Safety is essential for reinforcement learning (RL) applied in real-world tasks like autonomous driving. Chance constraints which guarantee the satisfaction of state constraints at a high probability are suitable to represent the requirements in real-world environment with uncertainty. Existing chance constrained RL methods like the penalty method and the Lagrangian method either exhibit periodic oscillations or cannot satisfy the constraints. In this paper, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. Taking a control perspective, we first interpret the penalty method and the Lagrangian method as proportional feedback and integral feedback control, respectively. Then, a proportional-integral Lagrangian method is proposed to steady learning process while improving safety. To prevent integral overshooting and reduce conservatism, we introduce the integral separation technique inspired by PID control. Finally, an analytical gradient of the chance constraint is utilized for model-based policy optimization. The effectiveness of SPIL is demonstrated by a narrow car-following task. Experiments indicate that compared with previous methods, SPIL improves the performance while guaranteeing safety, with a steady learning process.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 10 of its 32 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

10 of 32 references.

References · 10

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2018Reward Constrained Policy Optimizationcited by 397
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  7. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  8. 2020Projection-Based Constrained Policy Optimizationcited by 195
  9. 2019Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learningcited by 10
  10. 2020Model-Based Actor-Critic with Chance Constraint for Stochastic Systemcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 2; 2023: 2; 2024: 4; 2025: 1.

12 papers.

  1. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  2. 2024DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planningcited by 14
  3. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  4. 2021Safe Chance Constrained Reinforcement Learning for Batch Process Controlcited by 10
  5. 2022Risk Verification of Stochastic Systems with Neural Network Controllerscited by 6
  6. 2021Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learningcited by 4
  7. 2022Policy Gradients for Probabilistic Constrained Reinforcement Learningcited by 4
  8. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learningcited by 2
  9. 2023A Multiplicative Value Function for Safe and Efficient Reinforcement Learningcited by 1
  10. 2023A Policy Optimization Method Towards Optimal-time Stabilitycited by 0
  11. 2024Positive-Unlabeled Constraint Learning for Inferring Nonlinear Continuous Constraints Functions from Expert Demonstrationscited by 0
  12. 2025Predictive Lagrangian Optimization for Constrained Reinforcement Learningcited by 0