mlsanityData to 14 Oct 2026

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs

arXiv 2302.01275 ↗First posted 2 Feb 2023cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

In recent years, Reinforcement Learning (RL) has been applied to real-world problems with increasing success. Such applications often require to put constraints on the agent's behavior. Existing algorithms for constrained RL (CRL) rely on gradient descent-ascent, but this approach comes with a caveat. While these algorithms are guaranteed to converge on average, they do not guarantee last-iterate convergence, i.e., the current policy of the agent may never converge to the optimal solution. In practice, it is often observed that the policy alternates between satisfying the constraints and maximizing the reward, rarely accomplishing both objectives simultaneously. Here, we address this problem by introducing Reinforcement Learning with Optimistic Ascent-Descent (ReLOAD), a principled CRL method with guaranteed last-iterate convergence. We demonstrate its empirical effectiveness on a wide variety of CRL problems including discrete MDPs and continuous control. In the process we establish a benchmark of challenging CRL problems.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 57 of its 94 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

57 of 94 references.

References · 57

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  8. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
38 more
  1. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  2. 2019Challenges of Real-World Reinforcement Learningcited by 291
  3. 2019A Theory of Regularized Markov Decision Processescited by 286
  4. 2018Safe Exploration in Continuous Action Spacescited by 247
  5. 2017Training GANs with Optimismcited by 217
  6. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  7. 2018Meta-Gradient Reinforcement Learningcited by 159
  8. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  9. 2020Exploration-Exploitation in Constrained MDPscited by 149
  10. 2018A Variational Inequality Perspective on Generative Adversarial Networkscited by 144
  11. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
  12. 2019Behaviour Suite for Reinforcement Learningcited by 132
  13. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  14. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  15. 2020Mirror Descent Policy Optimizationcited by 106
  16. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 101
  17. 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
  18. 2019On the convergence of single-call stochastic extra-gradient methodscited by 89
  19. 2021Reward is enough for convex MDPscited by 84
  20. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  21. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  22. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  23. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  24. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  25. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 52
  26. 2021Bootstrapped Meta-Learningcited by 45
  27. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  28. 2018A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivitycited by 36
  29. 2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 35
  30. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  31. 2020Logistic Q-Learningcited by 34
  32. 2022Accelerated Single-Call Methods for Constrained Min-Max Optimizationcited by 32
  33. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  34. 2021Online Apprenticeship Learningcited by 29
  35. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 29
  36. 2019Last-iterate convergence rates for min-max optimizationcited by 25
  37. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  38. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 9; 2025: 2; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  2. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  3. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  4. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  5. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  6. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  7. 2025Alignment of large language models with constrained learningcited by 8
  8. 2023Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processescited by 7
  9. 2024A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guaranteescited by 6
  10. 2024Uniform Last-Iterate Guarantee for Bandits and Reinforcement Learningcited by 5
  11. 2024Operator Splitting for Convex Constrained Markov Decision Processescited by 4
  12. 2023A State Representation for Diminishing Rewardscited by 3
8 more
  1. 2024Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learningcited by 2
  2. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  3. 2025Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguisecited by 1
  4. 2026Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Marginscited by 1
  5. 2023Acceleration in Policy Optimizationcited by 0
  6. 2023Model-Free, Regret-Optimal Best Policy Identification in Online CMDPscited by 0
  7. 2023Handling Cost and Constraints with Off-Policy Deep Reinforcement Learningcited by 0
  8. 2024Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPscited by 0