mlsanityData to 14 Oct 2026

Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities

arXiv 2305.17568 ↗First posted 27 May 2023cs.LG math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are described by it general utilities, i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-making goals such as risk, exploration, or imitations. The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose a primal-dual method utilizing shadow reward and kappa-hop neighbor truncation under a form of correlation decay property, where kappa is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of Oleft(T-2/3right). In the sample-based setting, we demonstrate that, with high probability, our algorithm requires widetildeOleft(ε-3.5right) samples to achieve an ε-FOSP with an approximation error of O(φ02kappa), where φ0∈ (0,1). Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 47 of its 90 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

47 of 90 references.

References · 47

  1. 2021Convex optimizationcited by 2,515
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  4. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  5. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  6. 2019Challenges of Real-World Reinforcement Learningcited by 291
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  9. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 220
  10. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  11. 2019Efficient Exploration via State Marginal Matchingcited by 188
  12. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
35 more
  1. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  2. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  3. 2018Graph Convolutional Reinforcement Learningcited by 152
  4. 2020Exploration-Exploitation in Constrained MDPscited by 149
  5. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  6. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  7. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  8. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 95
  9. 2021Reward is enough for convex MDPscited by 84
  10. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  11. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  12. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  13. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  14. 2019Reinforcement Learning with Convex Constraintscited by 60
  15. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  16. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  17. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  18. 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 53
  19. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  20. 2021ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Marketscited by 36
  21. 2021Multi-Agent Constrained Policy Optimisationcited by 35
  22. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 34
  23. 2020Multi-Agent Reinforcement Learning in Stochastic Networked Systemscited by 32
  24. 2019A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Boundcited by 31
  25. 2019Scalable Reinforcement Learning for Multi-Agent Networked Systemscited by 24
  26. 2020Single-Timescale Stochastic Nonconvex-Concave Optimization for Smooth Nonlinear TD Learningcited by 24
  27. 2022Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learningcited by 20
  28. 2020Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithmscited by 19
  29. 2017Safety-Aware Apprenticeship Learningcited by 12
  30. 2020Multi-Agent Meta-Reinforcement Learning for Self-Powered and Sustainable Edge Computing Systemscited by 9
  31. 2018Concentration bounds for two time scale stochastic approximationcited by 8
  32. 2021MARL with General Utilities via Decentralized Shadow Reward Actor-Criticcited by 5
  33. 2022Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)cited by 5
  34. 2021Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learningcited by 4
  35. 2023Scalable Multi-Agent Reinforcement Learning with General Utilitiescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 6.

15 papers.

Sort:

By area:

  1. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 32
  2. 2024On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learningcited by 6
  3. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibriumcited by 6
  4. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  5. 2024Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learningcited by 3
  6. 2024SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDPcited by 3
  7. 2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
  8. 2025Proactive Constrained Policy Optimization with Preemptive Penaltycited by 2
  9. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 1
  10. 2026Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Explorationcited by 1
  11. 2026Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibriacited by 1
  12. 2026Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamicscited by 1
3 more
  1. 2026Robust General Utility for Reinforcement Learningcited by 1
  2. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functionscited by 0
  3. 2026Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedbackcited by 0