mlsanityData to 7 Oct 2026

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

arXiv 2105.12545 ↗First posted 26 May 2021cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose a successive convex approximation based off-policy optimization (SCAOPO) algorithm to solve the general constrained reinforcement learning problem, which is formulated as a constrained Markov decision process (CMDP) in the context of average cost. The SCAOPO is based on solving a sequence of convex objective/feasibility optimization problems obtained by replacing the objective and constraint functions in the original problems with convex surrogate functions. At each iteration, the convex surrogate problem can be efficiently solved by Lagrange dual method even the policy is parameterized by a high-dimensional function. Moreover, the SCAOPO enables to reuse old experiences from previous updates, thereby significantly reducing the implementation cost when deployed in the real-world engineering systems that need to online learn the environment. In spite of the time-varying state distribution and the stochastic bias incurred by the off-policy learning, the SCAOPO with a feasible initial point can still provably converge to a Karush-Kuhn-Tucker (KKT) point of the original problem almost surely.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

17 of 33 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2018Learning Dexterous In-Hand Manipulationcited by 643
  4. 2018Reward Constrained Policy Optimizationcited by 397
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  7. 2018Safe Exploration in Continuous Action Spacescited by 237
  8. 2020Projection-Based Constrained Policy Optimizationcited by 195
  9. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  10. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  11. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 129
  12. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
5 more
  1. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  2. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  3. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  4. 2019Artificial Intelligence-Enabled Intelligent 6G Networkscited by 27
  5. 2019Randomized Channel Sparsifying Hybrid Precoding for FDD Massive MIMO Systemscited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 3.

5 papers.

  1. 2023Asynchronous Multi-Model Dynamic Federated Learning over Wireless Networks: Theory, Modeling, and Optimizationcited by 3
  2. 2025A Hybrid Reinforcement Learning Framework for Hard Latency Constrained Resource Schedulingcited by 1
  3. 2023A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergencecited by 0
  4. 2025Context-aware Constrained Reinforcement Learning Based Energy-Efficient Power Scheduling for Non-stationary XR Data Trafficcited by 0
  5. 2025Joint User Priority and Power Scheduling for QoS-Aware WMMSE Precoding: A Constrained-Actor Attentive-Critic Approachcited by 0