mlsanityData to 7 Oct 2026

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods

arXiv 2007.03964 ↗First posted 8 Jul 2020math.OC cs.AI cs.LGcited by 302

Not on the map. Why?

Cited by 24 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Lagrangian methods are widely used algorithms for constrained optimization problems, but their learning dynamics exhibit oscillations and overshoot which, when applied to safe reinforcement learning, leads to constraint-violating behavior during agent training. We address this shortcoming by proposing a novel Lagrange multiplier update method that utilizes derivatives of the constraint function. We take a controls perspective, wherein the traditional Lagrange multiplier update behaves as integral control; our terms introduce proportional and derivative control, achieving favorable learning dynamics through damping and predictive measures. We apply our PID Lagrangian methods in deep RL, setting a new state of the art in Safety Gym, a safe RL benchmark. Lastly, we introduce a new method to ease controller tuning by providing invariance to the relative numerical scales of reward and cost. Our extensive experiments demonstrate improved performance and hyperparameter robustness, while our algorithms remain nearly as simple to derive and implement as the traditional Lagrangian approach.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 31; 2023: 53; 2024: 63; 2025: 52; 2026: 83.

302 papers.

Sort:

By area:

  1. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  3. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  5. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
  6. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  7. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  8. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  9. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  10. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  11. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  12. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
38 more
  1. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  3. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  4. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  5. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  6. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  7. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
  8. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  9. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  10. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  11. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  12. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
  13. 2022Verifying Learning-Based Robotic Navigation Systemscited by 33
  14. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  15. 2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 31
  16. 2024HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Drivingcited by 31
  17. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  18. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  19. 2021Safe Driving via Expert Guided Policy Optimizationcited by 29
  20. 2022Safe Reinforcement Learning for Legged Locomotioncited by 29
  21. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  22. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  23. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
  24. 2023Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithmscited by 27
  25. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  26. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  27. 2021Density Constrained Reinforcement Learningcited by 25
  28. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 25
  29. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  30. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
  31. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  32. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  33. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  34. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  35. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  36. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  37. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 16
  38. 2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 15

50 most cited of 302 All 302 (JSON)