mlsanityData to 14 Oct 2026

Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Games

arXiv 2503.22779 ↗First posted 28 Mar 2025cs.MA cs.GT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study a long-run mean-variance team stochastic game (MV-TSG), where each agent shares a common mean-variance objective for the system and takes actions independently to maximize it. MV-TSG has two main challenges. First, the variance metric is neither additive nor Markovian in a dynamic setting. Second, simultaneous policy updates of all agents lead to a non-stationary environment for each individual agent. Both challenges make dynamic programming inapplicable. In this paper, we study MV-TSGs from the perspective of sensitivity-based optimization. The performance difference and performance derivative formulas for joint policies are derived, which provide optimization information for MV-TSGs. We prove the existence of a deterministic Nash policy for this problem. Subsequently, we propose a Mean-Variance Multi-Agent Policy Iteration (MV-MAPI) algorithm with a sequential update scheme, where individual agent policies are updated one by one in a given order. We prove that the MV-MAPI algorithm converges to a first-order stationary point of the objective function. By analyzing the local geometry of stationary points, we derive specific conditions for stationary points to be (local) Nash equilibria, and further, strict local optima. To solve large-scale MV-TSGs in scenarios with unknown environmental parameters, we extend the idea of trust region methods to MV-MAPI and develop a multi-agent reinforcement learning algorithm named Mean-Variance Multi-Agent Trust Region Policy Optimization (MV-MATRPO). We derive a performance lower bound for each update of joint policies. Finally, numerical experiments on energy management in multiple microgrid systems are conducted.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

30 of 58 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  5. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  6. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  7. 2024Reinforcement learningcited by 247
  8. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  9. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  10. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 93
  11. 2023Heterogeneous-Agent Reinforcement Learningcited by 69
  12. 2015Decentralized Q-Learning for Stochastic Teams and Gamescited by 35
18 more
  1. 2021Gradient play in stochastic games: stationary points, convergence, and sample complexitycited by 34
  2. 2013On the Complexity of Solving Markov Decision Problemscited by 31
  3. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  4. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 30
  5. 2019A Cooperative Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Networkcited by 22
  6. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 20
  7. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 19
  8. 2022A Game-Theoretic Framework for Managing Risk in Multi-Agent Systemscited by 14
  9. 2019Decentralized Learning for Optimality in Stochastic Dynamic Teams and Games with Local Control and Global State Informationcited by 13
  10. 2020Risk-Sensitive Markov Decision Processes with Combined Metrics of Mean and Variancecited by 9
  11. 2023Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Financecited by 7
  12. 2016Stochastic Games for Smart Grid Energy Management with Prospect Prosumerscited by 6
  13. 2017Asynchronous Schemes for Stochastic and Misspecified Potential Games and Nonconvex Optimizationcited by 5
  14. 2024Provable Policy Gradient Methods for Average-Reward Markov Potential Gamescited by 5
  15. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 4
  16. 2016A Generalized Fundamental Matrix for Computing Fundamental Quantities of Markov Systemscited by 3
  17. 2016Zero-sum Risk-Sensitive Stochastic Gamescited by 2
  18. 2023Global Algorithms for Mean-Variance Optimization in Markov Decision Processescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Guided Policy Search for Non-Cooperative Dynamic Games
  2. 2024Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective
  3. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
  4. 2024Equilibrium Selection for Multi-agent Reinforcement Learning: A Unified Framework
  5. 2024A Scalable Game Theoretic Approach for Coordination of Multiple Dynamic Systems
  6. 2025Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games
  7. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games
  8. 2024Convergence of Decentralized Actor-Critic Algorithm in General-sum Markov Games