mlsanityData to 7 Oct 2026

ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints

arXiv 2302.00808 ↗First posted 2 Feb 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) for constrained MDPs (CMDPs) is an increasingly important problem for various applications. Often, the average criterion is more suitable than the discounted criterion. Yet, RL for average-CMDPs (ACMDPs) remains a challenging problem. Algorithms designed for discounted constrained RL problems often do not perform well for the average CMDP setting. In this paper, we introduce a new policy optimization with function approximation algorithm for constrained MDPs with the average criterion. The Average-Constrained Policy Optimization (ACPO) algorithm is inspired by trust region-based policy optimization algorithms. We develop basic sensitivity theory for average CMDPs, and then use the corresponding bounds in the design of the algorithm. We provide theoretical guarantees on its performance, and through extensive experimental work in various challenging OpenAI Gym environments, show its superior empirical performance when compared to other state-of-the-art algorithms adapted for the ACMDPs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 47 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

23 of 47 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Reward Constrained Policy Optimizationcited by 397
  9. 2020Projection-Based Constrained Policy Optimizationcited by 195
  10. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  11. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  12. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
11 more
  1. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  2. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  3. 2020Learning Infinite-horizon Average-reward MDPs with Linear Function Approximationcited by 48
  4. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
  5. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  6. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  7. 2022Learning Infinite-Horizon Average-Reward Markov Decision Processes with Constraintscited by 29
  8. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  9. 2017Dykstra's Algorithm, ADMM, and Coordinate Descent: Connections, Insights, and Extensionscited by 12
  10. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  11. 2019A Convolutional Neural Network Approach Towards Self-Driving Carscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  2. 2024e-COP : Episodic Constrained Optimization of Policiescited by 2
  3. 2026Learning to Harvest Without Collapse in a Regenerative Commons: A Lagrangian Frameworkcited by 0
  4. 2026Learning Infinite-Horizon Average-Reward CMDPs via State Augmentationcited by 0